Decide who trains
on your work.
Set a policy per crawler and copy the result straight into robots.txt. Separate the bots that cite you from the bots that train on you.
OAI-SearchBotOpenAI · search indexingGPTBotOpenAI · model trainingChatGPT-UserOpenAI · on-demand fetchClaude-SearchBotAnthropic · search indexingClaudeBotAnthropic · model trainingClaude-UserAnthropic · on-demand fetchPerplexityBotPerplexity · search indexingPerplexity-UserPerplexity · on-demand fetchGoogle-ExtendedGoogle · Gemini training controlBingbotMicrosoft · Copilot search indexApplebot-ExtendedApple · Apple Intelligence controlMeta-ExternalAgentMeta · model trainingAmazonbotAmazon · model trainingBytespiderByteDance · model trainingCCBotCommon Crawl · training datasetChoosing a stance
There is no universally correct AI crawler policy, but there are three coherent ones, and most sites should pick one deliberately rather than drift into a fourth by accident.
Open. Allow everything. Appropriate when discovery is worth more to you than control — documentation, marketing sites, anything where being the cited source is the goal. This is the default if you have no robots.txt at all.
Cite but do not train. Allow the search and on-demand fetchers, decline the training crawlers. This is the position most publishers actually want: stay eligible for citation in AI answers, which carry attribution and referral traffic, while declining uncompensated use of your archive as training data.
Closed. Block everything. Coherent when your content is behind a paywall or is the product itself. Understand the cost: you will not appear in AI answers, and that surface is where a growing share of research now begins.
Which bots do what
The policy only makes sense if you know what each user-agent is for. Training crawlers build model weights and return nothing to you. Search crawlers index pages so an engine can retrieve and cite them. On-demand fetchers retrieve a page because a user asked a question about it right now — blocking those is the most direct way to make yourself uncitable.
| User-agent | Operator | What it does |
|---|---|---|
OAI-SearchBot | OpenAI | search indexing |
GPTBot | OpenAI | model training |
ChatGPT-User | OpenAI | on-demand fetch |
Claude-SearchBot | Anthropic | search indexing |
ClaudeBot | Anthropic | model training |
Claude-User | Anthropic | on-demand fetch |
PerplexityBot | Perplexity | search indexing |
Perplexity-User | Perplexity | on-demand fetch |
Google-Extended | Gemini training control | |
Bingbot | Microsoft | Copilot search index |
Applebot-Extended | Apple | Apple Intelligence control |
Meta-ExternalAgent | Meta | model training |
Amazonbot | Amazon | model training |
Bytespider | ByteDance | model training |
CCBot | Common Crawl | training dataset |
Applying the policy
Paste the generated block into the robots.txt at your domain root, alongside your existing rules rather than instead of them. Overwriting the file is the most common way sites accidentally drop directives that Googlebot or Bingbot depended on.
Remember what robots.txt is: a voluntary protocol. The major AI companies publish their user-agents and honour it, but it is a stated preference and not an access control. Anything genuinely sensitive belongs behind authentication, not behind a Disallow line.
After publishing, confirm the file resolves the way you intended with the AI crawler access checker. Hand-written robots.txt files disagree with their authors' intentions more often than you would expect.
Frequently asked questions
Should I block AI crawlers?+
It depends on your business. If discovery and citation drive your traffic, blocking search and on-demand fetchers removes you from AI answers entirely. If your content is your product and you are not compensated for training use, declining the training crawlers while allowing the search ones is the common middle position.
What is the difference between Google-Extended and Googlebot?+
Googlebot crawls for Search. Google-Extended is a separate control that governs whether your content is used for Gemini model training and grounding. Blocking Google-Extended does not remove you from Google Search or from AI Overviews sourced via Search.
Will blocking Bytespider or CCBot hurt me?+
Neither sends meaningful referral traffic. CCBot feeds Common Crawl, a public dataset used by many model builders, so blocking it has broad downstream effects on training corpora. Bytespider is ByteDance's crawler and has a reputation for aggressive crawl rates.
Do I need a separate group per user-agent?+
Only where the policy differs. A crawler matches the most specific group naming it; if no group names it, it falls back to the wildcard group. Declaring each bot explicitly is more verbose but leaves no ambiguity about intent.
Where do I put the generated block?+
In the robots.txt file at the root of your domain. Add it alongside your existing rules — do not replace them, or you may drop directives that Googlebot and Bingbot depend on.