Which AI crawlers should your website allow?

This article covers the access stage of the getting cited by AI pillar: before any question of content quality or authority, the platform's crawler has to be able to fetch your page. What happens after access, when engines select which fetched pages to cite, is covered in how AI engines choose sources.

Which crawler does ChatGPT Search use?

ChatGPT search inclusion depends on OAI-SearchBot, not GPTBot. OpenAI's publisher guidance states that OAI-SearchBot is used to surface and link to websites in ChatGPT's search results, and OpenAI publishes its IP ranges and user agent so you can verify real requests.

GPTBot is a different crawler, associated with collecting content that may be used for model training. Blocking GPTBot expresses a training preference; it is not the switch that removes you from ChatGPT search. Many sites blocked GPTBot in 2023 to opt out of training and assumed that decision had no search cost, and under the current split that assumption is correct only if OAI-SearchBot remains allowed.

There is also a user-directed agent, ChatGPT-User, which fetches a page when a person asks ChatGPT to open a specific link. Treat it like a visitor acting through a browser, not a crawler building an index.

Which crawler controls Google AI Overviews and AI Mode?

Ordinary Googlebot controls eligibility. Google's AI features documentation says pages that appear as supporting links in AI Overviews and AI Mode must be indexed and eligible to be shown with a snippet in Search, using the same crawl and indexing pipeline as everything else.

Google-Extended is not the control for AI Overviews or AI Mode. It is a separate token governing other Google AI uses of your content. If you disallow Google-Extended while allowing Googlebot, you remain eligible for AI Overviews; if you disallow Googlebot, you disappear from Search and its AI features together.

Google also states there is no special schema type or AI-specific requirement for these features; its guidance for generative AI experiences is essentially standard indexing and content quality advice.

Which crawler does Perplexity use?

Perplexity's crawler documentation recommends allowing PerplexityBot, its crawler for building the index behind its search results. Perplexity also documents Perplexity-User, a user-directed fetcher that retrieves a page when a person's query needs it; the docs note it is not used to build the index.

Anthropic documents its crawlers similarly: ClaudeBot crawls the web for data that may improve models, and separate agents handle user-directed fetching and search-related retrieval. Follow Anthropic's current documentation for each agent's purpose rather than assuming ClaudeBot is a citation crawler.

What is the difference between search, training and user-directed crawlers?

Each vendor now runs up to three kinds of agent, and robots.txt lets you treat them separately:

The practical rule: decide your training position on its own merits, but keep search and user-directed agents allowed if you want to appear in AI answers.

How should robots.txt be configured?

If you want AI search visibility and accept training use, the simplest correct configuration is your normal allow-all with your private paths disallowed:

User-agent: *
Allow: /
Disallow: /api/
Disallow: /admin

If you want AI search visibility while opting out of training, name the training crawlers specifically and leave everything else open:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /

Never copy a "block all AI" robots.txt template without reading it: most of those templates disallow the search crawlers too, which removes you from the answers your buyers read.

Can a CDN or WAF block an allowed crawler?

Yes, and this is the most common silent failure. Your robots.txt can say allow while the request never reaches your server, because a layer in front of it rejects the crawler:

• Bot-management rules at CDNs can challenge or block verified crawlers by default or by category toggle.
• JavaScript challenges and CAPTCHAs return an interstitial page instead of your content; a crawler cannot solve them.
• Geo or rate rules can 403 the IP ranges crawlers arrive from.
• Authentication walls and cookie gates serve the crawler a login page rather than the article.

robots.txt is the invitation; the CDN decides who gets through the door. Check both.

How do you verify that a crawler can access a page?

Verify at three layers, in order:

1. robots.txt. Fetch yourdomain.com/robots.txt and check each search crawler's group resolves to allow for the page's path.
2. HTTP response. Check your CDN or server logs for requests from the crawler's published user agent and confirm they receive 200 with real HTML, not 403, 429 or a challenge page. For OpenAI, validate the requester against the published OAI-SearchBot IP list rather than trusting the user-agent string alone.
3. Index presence. For Google, use Search Console's URL Inspection to confirm the page is indexed and snippet-eligible. For other engines, ask the platform about the page's topic and see whether the page can appear at all.

If any layer fails, fix that layer before touching content. A perfectly written page behind a 403 does not exist as far as the answer engine is concerned. Access checks also protect your identity signals: profiles and schema only help if the pages carrying them are fetchable, which is why entity diagnosis starts after access is confirmed.

What CiteAgentic's technical audit reports

Product example. CiteAgentic's technical audit checks this access chain for your site: robots.txt rules per crawler, HTTP status served to honest crawler requests, rendering that hides content from non-browser readers, and missing or conflicting index directives. Each failure becomes a specific finding with the fix instruction, and after you deploy the change a re-scan verifies the same check now passes. It reports access as eligibility only; no access fix is ever presented as a promised citation.