AI Crawler Access & robots.txt: Technical Guide to GPTBot, ClaudeBot, and PerplexityBot Indexation

Executive summary

TL;DR: AI answer engines can only cite pages their crawlers are allowed to fetch. Allow the search and retrieval bots (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude's crawlers, Googlebot, Bingbot, Applebot) in robots.txt, decide separately whether to allow model-training bots (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot), keep private paths disallowed, and confirm your CDN or firewall is not silently blocking them.

• robots.txt is governed by IETF RFC 9309; the longest matching rule wins.
• Training access and real-time retrieval access are controlled by different user agents.
• Edge firewalls often block AI bots even when robots.txt allows them, so test with real requests.

The two-tier crawling pipeline

AI systems reach your content in two distinct ways:

1. Bulk pre-training ingestion. Crawlers such as GPTBot, ClaudeBot, Google-Extended and CCBot collect large volumes of public text that may be used to train future models. Blocking them affects what a model "knows" long term, not whether you appear in today's answers.
2. Real-time retrieval (RAG). When a user asks a question, search-oriented agents such as OAI-SearchBot, ChatGPT-User and PerplexityBot fetch live pages, extract passages and cite them. Blocking these removes you from cited answers directly.

Retrieval is where visibility is won. As Aggarwal et al. write in the Princeton GEO study, generative engines "synthesize information from multiple sources" and adding citations, quotations and statistics can boost source visibility "by up to 40%" in generative responses. None of that applies if the retrieval crawler cannot read the page.

The 2026/2027 AI crawler directory

User agent	Operator	Purpose	Effect of blocking
OAI-SearchBot	OpenAI	Indexes pages for ChatGPT search results	Not shown or cited in ChatGPT search
ChatGPT-User	OpenAI	Fetches pages on demand for user requests	ChatGPT cannot open your pages live
GPTBot	OpenAI	Collects content for model training	Excluded from future training data
ClaudeBot	Anthropic	Collects content for model training	Excluded from Claude training data
anthropic-ai	Anthropic	Legacy Anthropic token still honoured by some sites	Kept for backwards compatibility
PerplexityBot	Perplexity	Indexes pages for Perplexity answers	Not surfaced or cited in Perplexity
Google-Extended	Google	Control token for Gemini training and grounding	Content not used for Gemini models; Search unaffected
Googlebot	Google	Google Search and AI Overviews index	Removed from Search and AI Overviews
Applebot-Extended	Apple	Control token for Apple model training	Not used for Apple Intelligence training
Applebot	Apple	Siri and Spotlight search index	Not shown in Siri/Spotlight results
CCBot	Common Crawl	Open web corpus used by many model builders	Excluded from Common Crawl datasets
Bingbot	Microsoft	Bing index powering Copilot and other engines	Removed from Bing and Copilot answers

Operators update their lists; check each vendor's official crawler documentation before making changes.

How RFC 9309 evaluates robots.txt

• Group selection: a crawler obeys the group whose User-agent line best matches its product token; if none match, it uses the * group. Groups are not combined.
• Longest match wins: RFC 9309 Section 2.2.2 states that "the most specific match found MUST be used. The most specific match is the match that has the most octets."
• Tie-breaker: "If an allow rule and a disallow rule are equivalent, then the allow rule SHOULD be used."
• Case sensitivity: Section 2.2.1 says crawlers "MUST use case-insensitive matching to find the group that matches the product token". Paths, however, are case-sensitive: /Admin and /admin are different rules.
• Special characters: * matches any sequence and $ anchors the end of a path.

Production-ready robots.txt templates

Architecture A: open generative ingestion

Maximises both citations and training presence.

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /dashboard/
Disallow: /api/
Disallow: /*?session=

Sitemap: https://YOUR-DOMAIN.COM/sitemap.xml

Architecture B: selective search ingestion

Allows real-time retrieval and search, but opts out of model training.

User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: PerplexityBot
User-agent: Googlebot
User-agent: Bingbot
User-agent: Applebot
Allow: /
Disallow: /admin/
Disallow: /api/

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/

Sitemap: https://YOUR-DOMAIN.COM/sitemap.xml

Copy and edit all placeholder text in CAPITALS in the code blocks before implementing.

CDN and edge firewall troubleshooting

A permissive robots.txt means nothing if the edge returns 403. Cloudflare's managed "Block AI bots" / AI Scrapers and Crawlers setting and bot-fight modes can block AI agents by default. Review Security → Bots, and allow verified bots you want to reach your site.

Run synthetic checks for each user agent:

curl -I -A "Mozilla/5.0 (compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)" https://YOUR-DOMAIN.COM/
curl -I -A "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" https://YOUR-DOMAIN.COM/
curl -I -A "Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" https://YOUR-DOMAIN.COM/
curl -s https://YOUR-DOMAIN.COM/robots.txt

Expect 200. A 403, 429 or challenge page means the firewall is blocking. User-agent spoofing tests only the rule path; real bots are verified by IP ranges the vendors publish, so also check firewall logs.

Common antipatterns to avoid

• A leftover Disallow: / under User-agent: * from a staging site.
• Blocking OAI-SearchBot or PerplexityBot while trying only to opt out of training.
• Expecting a bot to merge rules from its named group and the * group.
• Serving robots.txt with a 5xx error; RFC 9309 tells crawlers to treat it as fully disallowed.
• Rendering key content only with client-side JavaScript that AI fetchers do not execute.
• Relying on robots.txt to hide private data; use authentication instead.

Scholarly and standards citations

1. Koster, M., Illyes, G., Zeller, H., Sassman, L. (2022). RFC 9309: Robots Exclusion Protocol. IETF.
2. Aggarwal, P. et al. (2023). GEO: Generative Engine Optimization. Princeton University / arXiv:2311.09735.
3. OpenAI. Overview of OpenAI crawlers.
4. Google. Google crawlers and fetchers.
5. Perplexity. Perplexity crawlers.
6. Apple. About Applebot.