> ## Documentation Index
> Fetch the complete documentation index at: https://docs.llmtag.org/llms.txt
> Use this file to discover all available pages before exploring further.

# AI Agent Implementation Guide

> How AI agents and crawlers can implement LLMTAG protocol compliance

## AI Agent Compliance

This guide is for AI companies, researchers, and developers who want to implement LLMTAG protocol compliance in their AI agents and crawlers.

<Card title="Become LLMTAG Compliant" icon="shield-check" horizontal>
  **Respect Publisher Policies** • **Build Trust** • **Industry Leadership** • **Ethical AI Practices**
</Card>

## Why Implement LLMTAG Compliance?

### Benefits for AI Companies

<Columns cols={2}>
  <Card title="Legal Clarity" icon="scale-balanced">
    Clear, machine-readable policies reduce legal uncertainty and compliance risks.
  </Card>

  <Card title="Ethical Compliance" icon="heart">
    Respect publisher preferences and build trust with content creators.
  </Card>

  <Card title="Implementation Simplicity" icon="code">
    Standardized format makes compliance straightforward to implement.
  </Card>

  <Card title="Industry Leadership" icon="trophy">
    Be part of establishing ethical AI practices from the ground up.
  </Card>
</Columns>

### Benefits for the AI Ecosystem

<Columns cols={2}>
  <Card title="Sustainable AI" icon="leaf">
    Create a sustainable relationship between AI and content creation.
  </Card>

  <Card title="Trust Building" icon="handshake">
    Build trust between AI companies and content creators.
  </Card>

  <Card title="Innovation Protection" icon="lightbulb">
    Protect content creators while enabling AI innovation.
  </Card>

  <Card title="Global Standard" icon="globe">
    Establish a universal protocol that works across all platforms.
  </Card>
</Columns>

## Implementation Requirements

### Core Compliance Requirements

AI agents that claim compliance with the LLMTAG protocol must:

<Checklist>
  <CheckboxItem>**Check for llmtag.txt** before processing any content</CheckboxItem>
  <CheckboxItem>**Parse the file correctly** according to the specification</CheckboxItem>
  <CheckboxItem>**Respect all applicable directives** based on agent identity and content path</CheckboxItem>
  <CheckboxItem>**Handle errors gracefully** by applying default policies when files are inaccessible</CheckboxItem>
  <CheckboxItem>**Log compliance actions** for audit purposes</CheckboxItem>
  <CheckboxItem>**Provide transparency** about compliance practices</CheckboxItem>
</Checklist>

### Discovery Mechanism

AI agents should automatically check for `llmtag.txt` by making a GET request to:

```
https://[domain]/llmtag.txt
```

<Info>
  The discovery mechanism follows the same pattern as `robots.txt`, making it familiar and discoverable for both humans and automated systems.
</Info>

## Implementation Guide

### Step 1: Discovery

<Steps>
  <Step title="Check for llmtag.txt">
    Before processing any content from a domain, make a GET request to `https://domain.com/llmtag.txt`.
  </Step>

  <Step title="Handle HTTP Responses">
    Process different HTTP response codes appropriately:

    * **200 OK**: Parse the file content
    * **404 Not Found**: Apply default policies
    * **403 Forbidden**: Apply default policies
    * **500 Server Error**: Apply default policies
  </Step>

  <Step title="Cache Results">
    Cache the parsed policies to avoid repeated requests for the same domain.
  </Step>
</Steps>

### Step 2: Parsing

<Steps>
  <Step title="Validate File Format">
    Ensure the file starts with `spec_version: 3.0` or is otherwise valid.
  </Step>

  <Step title="Parse Directives">
    Parse all directives according to the specification.
  </Step>

  <Step title="Handle Scope Blocks">
    Process User-agent and Path blocks to determine applicable policies.
  </Step>

  <Step title="Apply Inheritance">
    Apply directive inheritance from global to specific scopes.
  </Step>
</Steps>

### Step 3: Policy Application

<Steps>
  <Step title="Identify Agent Scope">
    Determine which User-agent blocks apply to your agent.
  </Step>

  <Step title="Identify Path Scope">
    Determine which Path blocks apply to the content being accessed.
  </Step>

  <Step title="Apply Policies">
    Apply the most specific applicable policies.
  </Step>

  <Step title="Log Actions">
    Log all compliance actions for audit and transparency purposes.
  </Step>
</Steps>

## Code Examples

### Python Implementation

```python theme={null}
import requests
import re
from typing import Dict, List, Optional

class LLMTAGParser:
    def __init__(self):
        self.cache = {}
    
    def get_policies(self, domain: str, user_agent: str, path: str) -> Dict:
        """Get LLMTAG policies for a domain, user agent, and path."""
        
        # Check cache first
        cache_key = f"{domain}:{user_agent}:{path}"
        if cache_key in self.cache:
            return self.cache[cache_key]
        
        # Fetch llmtag.txt
        try:
            response = requests.get(f"https://{domain}/llmtag.txt", timeout=10)
            if response.status_code != 200:
                return self._get_default_policies()
            
            content = response.text
            policies = self._parse_llmtag(content, user_agent, path)
            
            # Cache the result
            self.cache[cache_key] = policies
            return policies
            
        except Exception as e:
            print(f"Error fetching llmtag.txt for {domain}: {e}")
            return self._get_default_policies()
    
    def _parse_llmtag(self, content: str, user_agent: str, path: str) -> Dict:
        """Parse llmtag.txt content and return applicable policies."""
        
        lines = content.strip().split('\n')
        policies = self._get_default_policies()
        
        current_scope = None
        current_agent = None
        current_path = None
        
        for line in lines:
            line = line.strip()
            if not line or line.startswith('#'):
                continue
            
            if line.startswith('spec_version:'):
                # Validate specification version
                version = line.split(':', 1)[1].strip()
                if version != '3.0':
                    print(f"Unsupported LLMTAG version: {version}")
                    return self._get_default_policies()
            
            elif line.startswith('User-agent:'):
                current_agent = line.split(':', 1)[1].strip()
                current_scope = 'agent'
            
            elif line.startswith('Path:'):
                current_path = line.split(':', 1)[1].strip()
                current_scope = 'path'
            
            elif line.startswith('ai_training_data:'):
                value = line.split(':', 1)[1].strip()
                if self._applies_to_current_scope(user_agent, path, current_agent, current_path):
                    policies['ai_training_data'] = value
            
            elif line.startswith('ai_use:'):
                value = line.split(':', 1)[1].strip()
                if self._applies_to_current_scope(user_agent, path, current_agent, current_path):
                    policies['ai_use'] = [v.strip() for v in value.split(',')]
        
        return policies
    
    def _applies_to_current_scope(self, user_agent: str, path: str, 
                                current_agent: str, current_path: str) -> bool:
        """Check if current scope applies to the given user agent and path."""
        
        if current_agent and current_agent.lower() not in user_agent.lower():
            return False
        
        if current_path and not path.startswith(current_path):
            return False
        
        return True
    
    def _get_default_policies(self) -> Dict:
        """Return default policies when no llmtag.txt is found."""
        return {
            'ai_training_data': 'allow',
            'ai_use': ['search_indexing']
        }

# Usage example
parser = LLMTAGParser()
policies = parser.get_policies('example.com', 'MyAI-Bot/1.0', '/blog/post-1')

if policies['ai_training_data'] == 'disallow':
    print("AI training not allowed for this content")
else:
    print("AI training allowed")

if 'generative_synthesis' not in policies['ai_use']:
    print("Generative synthesis not allowed")
else:
    print("Generative synthesis allowed")
```

### JavaScript Implementation

```javascript theme={null}
class LLMTAGParser {
    constructor() {
        this.cache = new Map();
    }
    
    async getPolicies(domain, userAgent, path) {
        // Check cache first
        const cacheKey = `${domain}:${userAgent}:${path}`;
        if (this.cache.has(cacheKey)) {
            return this.cache.get(cacheKey);
        }
        
        try {
            // Fetch llmtag.txt
            const response = await fetch(`https://${domain}/llmtag.txt`, {
                method: 'GET',
                timeout: 10000
            });
            
            if (!response.ok) {
                return this.getDefaultPolicies();
            }
            
            const content = await response.text();
            const policies = this.parseLLMTAG(content, userAgent, path);
            
            // Cache the result
            this.cache.set(cacheKey, policies);
            return policies;
            
        } catch (error) {
            console.error(`Error fetching llmtag.txt for ${domain}:`, error);
            return this.getDefaultPolicies();
        }
    }
    
    parseLLMTAG(content, userAgent, path) {
        const lines = content.trim().split('\n');
        const policies = this.getDefaultPolicies();
        
        let currentScope = null;
        let currentAgent = null;
        let currentPath = null;
        
        for (const line of lines) {
            const trimmedLine = line.trim();
            if (!trimmedLine || trimmedLine.startsWith('#')) {
                continue;
            }
            
            if (trimmedLine.startsWith('spec_version:')) {
                const version = trimmedLine.split(':')[1].trim();
                if (version !== '3.0') {
                    console.warn(`Unsupported LLMTAG version: ${version}`);
                    return this.getDefaultPolicies();
                }
            }
            
            else if (trimmedLine.startsWith('User-agent:')) {
                currentAgent = trimmedLine.split(':')[1].trim();
                currentScope = 'agent';
            }
            
            else if (trimmedLine.startsWith('Path:')) {
                currentPath = trimmedLine.split(':')[1].trim();
                currentScope = 'path';
            }
            
            else if (trimmedLine.startsWith('ai_training_data:')) {
                const value = trimmedLine.split(':')[1].trim();
                if (this.appliesToCurrentScope(userAgent, path, currentAgent, currentPath)) {
                    policies.ai_training_data = value;
                }
            }
            
            else if (trimmedLine.startsWith('ai_use:')) {
                const value = trimmedLine.split(':')[1].trim();
                if (this.appliesToCurrentScope(userAgent, path, currentAgent, currentPath)) {
                    policies.ai_use = value.split(',').map(v => v.trim());
                }
            }
        }
        
        return policies;
    }
    
    appliesToCurrentScope(userAgent, path, currentAgent, currentPath) {
        if (currentAgent && !userAgent.toLowerCase().includes(currentAgent.toLowerCase())) {
            return false;
        }
        
        if (currentPath && !path.startsWith(currentPath)) {
            return false;
        }
        
        return true;
    }
    
    getDefaultPolicies() {
        return {
            ai_training_data: 'allow',
            ai_use: ['search_indexing']
        };
    }
}

// Usage example
const parser = new LLMTAGParser();
const policies = await parser.getPolicies('example.com', 'MyAI-Bot/1.0', '/blog/post-1');

if (policies.ai_training_data === 'disallow') {
    console.log('AI training not allowed for this content');
} else {
    console.log('AI training allowed');
}

if (!policies.ai_use.includes('generative_synthesis')) {
    console.log('Generative synthesis not allowed');
} else {
    console.log('Generative synthesis allowed');
}
```

## Compliance Testing

### Testing Checklist

<Steps>
  <Step title="Test Discovery">
    Verify that your agent correctly discovers and fetches `llmtag.txt` files.
  </Step>

  <Step title="Test Parsing">
    Test parsing with various `llmtag.txt` file formats and edge cases.
  </Step>

  <Step title="Test Policy Application">
    Verify that policies are correctly applied based on user agent and path.
  </Step>

  <Step title="Test Error Handling">
    Ensure graceful handling of inaccessible or malformed files.
  </Step>

  <Step title="Test Caching">
    Verify that caching works correctly and doesn't cause stale policy issues.
  </Step>
</Steps>

### Test Cases

<Columns cols={2}>
  <Card title="Basic Compliance" icon="check">
    **Test:** Simple llmtag.txt with global policies
    **Expected:** Policies applied correctly
  </Card>

  <Card title="Agent-Specific Rules" icon="user-robot">
    **Test:** User-agent blocks with specific policies
    **Expected:** Correct policies for matching agents
  </Card>

  <Card title="Path-Based Rules" icon="folder-tree">
    **Test:** Path blocks with different policies
    **Expected:** Correct policies for matching paths
  </Card>

  <Card title="Error Handling" icon="exclamation-triangle">
    **Test:** 404, 403, 500 responses
    **Expected:** Default policies applied
  </Card>
</Columns>

## Best Practices

### Implementation Best Practices

<Columns cols={2}>
  <Card title="Respect Policies" icon="shield-check">
    Actually follow the policies you claim to support, not just check the files.
  </Card>

  <Card title="Be Transparent" icon="eye">
    Provide clear information about how you handle LLMTAG policies and compliance.
  </Card>

  <Card title="Implement Early" icon="rocket">
    Start implementing LLMTAG compliance now to build trust with content creators.
  </Card>

  <Card title="Provide Audit Trails" icon="list-check">
    Maintain logs of your compliance actions for transparency and accountability.
  </Card>
</Columns>

### Performance Considerations

<Note>
  Follow these tips to optimize your LLMTAG implementation:
</Note>

* **Cache policies** to avoid repeated requests
* **Use appropriate timeouts** for HTTP requests
* **Handle errors gracefully** without breaking functionality
* **Monitor performance** and optimize as needed

## Community and Support

### Getting Help

<CardGroup cols={2}>
  <Card title="Technical Support" icon="code" href="/resources/community">
    Get help with implementation questions and technical issues.
  </Card>

  <Card title="Compliance Guidance" icon="shield-check" href="mailto:compliance@llmtag.org">
    Contact us for guidance on compliance implementation.
  </Card>

  <Card title="Testing Tools" icon="flask" href="https://github.com/llmtag/testing-tools">
    Use our testing tools to verify your implementation.
  </Card>

  <Card title="Community Examples" icon="github" href="https://github.com/llmtag/examples">
    See implementation examples from other AI companies.
  </Card>
</CardGroup>

### Certification Program

<Card title="LLMTAG Compliance Certification" icon="certificate" href="mailto:certification@llmtag.org" horizontal>
  **Get Certified** • **Show Compliance** • **Build Trust** • **Industry Recognition**
</Card>

## Legal and Ethical Considerations

### Legal Framework

<Info>
  LLMTAG is a technical standard that communicates preferences, not legal requirements. However, respecting these preferences can help with legal compliance and ethical AI practices.
</Info>

### Ethical AI Practices

<Checklist>
  <CheckboxItem>**Respect Creator Rights**: Honor the preferences of content creators</CheckboxItem>
  <CheckboxItem>**Transparent Practices**: Be open about how you use content</CheckboxItem>
  <CheckboxItem>**Consent-Based Usage**: Use content only as permitted by creators</CheckboxItem>
  <CheckboxItem>**Accountable Actions**: Maintain records of compliance decisions</CheckboxItem>
</Checklist>