Okara
Start Free
Back to Blog
By Fatima Rizwan · Published June 24, 2026 · 9 min read

robots.txt for AI Crawlers: Search, Training, and User-Triggered Bots

Understand which bots support search, training, and user-requested retrieval, with practical robots.txt examples for major AI and search platforms

There is no single switch for "AI crawlers." Providers use different bots for search indexing, model training, and fetches triggered by a user. Blocking one can affect training without affecting search. Blocking another can remove a site from a search product. Some user-triggered agents may not treat robots.txt like an automatic crawler does.

A useful robots.txt file starts with policy: which automated uses do you want to permit? The examples below are choices, not universal requirements. Check the linked first-party documentation before deploying them because bot names, products, and controls can change.

What robots.txt does

A robots.txt file gives compliant crawlers instructions about which paths they may fetch. Put it at the root of each host, such as https://www.example.com/robots.txt. A rule on the main domain does not automatically cover every subdomain.

Robots.txt is not access control. A blocked URL can still be discovered, and a bot can ignore or misinterpret the file. Keep private material behind authentication rather than relying on Disallow.

An explicit Allow: / is often unnecessary because a bot is normally allowed when no matching disallow rule exists. Explicit groups can still make a policy easier to audit. Test the finished file, especially when it already contains wildcard or path-specific rules.

Separate search, training, and user-triggered retrieval

These uses have different consequences:

  • Search crawlers build or support an index used to return links and answers.
  • Training crawlers collect material that may be used to develop foundation models.
  • User-triggered agents fetch a page because a person asked a product to visit or use it.

The categories are not interchangeable. A provider may reuse a crawl internally, but its published controls can still let site owners express different preferences.

Google's controls

Googlebot crawls for Google Search, including Search features such as AI Overviews and AI Mode. Google says a page must be indexed and eligible to appear in Search with a snippet before it can appear as a supporting link in those AI features.

There is no separate robots.txt crawler for AI Overviews. If you block Googlebot, you also block normal Google Search crawling for that content. To limit how page content appears across Search, Google points publishers to nosnippet, max-snippet, data-nosnippet, and noindex. Those controls have broader Search effects, so review them before use.

Google-Extended is not an AI Overview control

Google-Extended is a product token, not a distinct HTTP user-agent string. It lets publishers manage whether content Google crawls may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI services covered by Google's documentation.

Google states that Google-Extended does not affect inclusion or ranking in Google Search. Allowing it is not required for AI Overviews or AI Mode. Blocking it does not opt a page out of those Search features.

OpenAI's controls

OpenAI documents three relevant agents:

  • OAI-SearchBot is its automatic search crawler. OpenAI says sites that opt out will not be shown in ChatGPT search answers, apart from possible navigational links.
  • GPTBot crawls content that may be used to train OpenAI's generative AI foundation models.
  • ChatGPT-User supports certain user actions in ChatGPT and custom GPTs. It does not crawl automatically and does not determine Search inclusion. OpenAI says robots.txt rules may not apply because a user initiates these requests.

The search and training settings are independent. A site can allow OAI-SearchBot and disallow GPTBot.

Perplexity's controls

Perplexity documents two agents:

  • PerplexityBot is an automatic crawler used to surface and link websites in Perplexity search results. Perplexity says it is not used to collect content for foundation-model training.
  • Perplexity-User fetches pages in response to user actions. Perplexity says it is not an automatic crawler or a training crawler and that, because a user requested the fetch, it generally ignores robots.txt.

Perplexity's public crawler page does not list a separate training crawler. Do not label PerplexityBot as one or invent a training control the company has not documented.

Anthropic's controls

Anthropic documents three bots with separate purposes:

  • Claude-SearchBot navigates the web to improve search result quality.
  • ClaudeBot collects web content that could contribute to model training.
  • Claude-User retrieves content in response to a user's request.

ClaudeBot is training-related. It is not a generic Claude citation crawler. Sites that want Anthropic search access but do not want training access can make different choices for Claude-SearchBot and ClaudeBot.

Anthropic says its bots honor robots.txt. This differs from the guidance OpenAI and Perplexity publish for their user-triggered agents, so do not assume one provider's behavior applies to another.

Microsoft's controls

Bingbot crawls pages for the Bing index. Microsoft's webmaster guidance connects Bing crawling and indexing with both traditional search and AI-powered experiences such as Copilot.

Microsoft's cited public webmaster material does not define a set of separate training and user-triggered robots.txt agents comparable to OpenAI's or Anthropic's. Treat Bingbot as a search-indexing decision. Blocking it can affect Bing Search as well as Microsoft experiences that rely on the Bing index.

Microsoft also recommends accurate XML sitemaps and supports IndexNow for notifying participating search engines about changed URLs. These can improve discovery and freshness, but they do not guarantee inclusion in a search result or generated answer.

Policy example: allow search, decline documented training uses

This example permits the named automatic search crawlers while declining the documented training-related controls from OpenAI, Anthropic, and Google.

User-agent: Googlebot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Sitemap: https://www.example.com/sitemap.xml

This is not a complete opt-out from all possible training. It covers only the named controls and providers shown. It also does not stop user-triggered retrieval where a provider says robots.txt may not apply.

Replace the sitemap URL and merge these groups carefully with existing rules. A second group for the same agent can produce results you did not intend, depending on how a crawler combines matching groups.

Policy example: allow documented search and training crawlers

Some publishers prefer broad crawl access. They might use an explicit file like this:

User-agent: Googlebot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

Again, this is a policy choice. It does not improve rankings by itself or guarantee that any product will cite the site.

Policy example: block selected non-Google search crawlers

A publisher that does not want automatic retrieval for ChatGPT search, Perplexity, or Claude search could use:

User-agent: OAI-SearchBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Claude-SearchBot
Disallow: /

That choice may reduce or remove the site's eligibility in those providers' search results, according to their documentation. It says nothing about training unless the training agents have their own rules.

Google needs a different decision. There is no robots.txt rule that keeps ordinary Google Search crawling while blocking only AI Overviews or AI Mode. Blocking Googlebot affects Google Search broadly. Blocking Google-Extended does not block those Search features.

The same caution applies to Bing: blocking Bingbot is a Bing indexing decision, not a narrowly documented Copilot-only switch.

Check the whole delivery path

An accurate robots.txt file is only part of crawler access. A CDN, web application firewall, hosting rule, or bot-management product can still reject requests.

For providers that publish IP ranges, compare both the user-agent and the source IP or verified hostname. User-agent strings are easy to spoof. Use the current first-party IP files rather than copying an old list into a permanent firewall rule.

Review:

  1. the robots.txt file returned on every relevant host;
  2. HTTP status codes for the pages and robots.txt itself;
  3. CDN and firewall bot rules;
  4. server logs for the named agents;
  5. provider-published IP ranges or verification methods;
  6. noindex and snippet controls on pages meant for search;
  7. sitemap URLs and canonical tags.

Crawler access only creates eligibility. It does not guarantee indexing, ranking, retrieval, or citation.

Robots.txt is a public request to compliant crawlers. It does not grant a license, settle copyright questions, remove content already collected, or prevent a person from copying a public page.

If material must stay private, require authentication. If a provider offers account-level, contractual, or removal controls for a separate concern, handle that through the provider's documented process rather than assuming a robots.txt rule solves it.

Where Okara fits

Disclosure: Okara is our product.

Okara can audit named crawler rules, sitemap references, and common CDN conflicts. It cannot guarantee that a crawler will visit a page or that an AI product will cite it. A person should still approve the underlying search and training policy.

Frequently asked questions

Do I need to allow every AI crawler?

No. Decide separately whether you want each provider's search, training, and user-triggered access. The right policy depends on the site's goals and content rights.

Is ClaudeBot the crawler for Claude search citations?

No. Anthropic describes ClaudeBot as collecting content that may contribute to model training. It documents Claude-SearchBot for search and Claude-User for user-requested retrieval.

Does Google-Extended control AI Overviews?

No. Google says Google-Extended does not affect Google Search inclusion or ranking. Googlebot and normal Search and snippet controls apply to AI Overviews and AI Mode.

Can I appear in ChatGPT search while blocking OpenAI training?

OpenAI documents that choice. Allow OAI-SearchBot for Search and disallow GPTBot for training. ChatGPT-User is separate and may not follow robots.txt for user-triggered requests.

Is PerplexityBot a training crawler?

Perplexity says it is not used to crawl content for foundation-model training. It is the automatic crawler for surfacing and linking sites in Perplexity search results.

Does allowing a crawler guarantee citations?

No. It only removes one possible access barrier. A provider still decides whether to crawl, index, retrieve, or cite the page.

Should I add an llms.txt file?

An llms.txt file is not a replacement for robots.txt. The first-party crawler controls discussed here use robots.txt, and Google says no new AI text file is required for AI Overviews or AI Mode.

Sources