Which developer references
block AI crawlers?
Documentation and Q&A sites — the sources answer engines lean on hardest for technical queries. 159 domains measured; 14 block at least one of the 30 tracked AI crawlers.
14of 159 developer references block at least one AI crawler · median access 100%
The AI crawlers these sites block most
Bytespider is blocked most often here, by 10 of 159 sites. What blocking each one means is in the AI crawler directory.
| User-agent | Operator | Purpose | Sites blocking |
|---|---|---|---|
Bytespider | ByteDance | model training | 10 of 159 |
CCBot | Common Crawl | training dataset | 8 of 159 |
GPTBot | OpenAI | model training | 8 of 159 |
PetalBot | Huawei | search indexing | 8 of 159 |
ClaudeBot | Anthropic | model training | 7 of 159 |
Amazonbot | Amazon | model training | 6 of 159 |
cohere-ai | Cohere | model training | 6 of 159 |
Diffbot | Diffbot | structured-data extraction | 6 of 159 |
Every domain in this category
Sorted by the share of tracked AI crawlers permitted, lowest first. Each report shows all 30 verdicts and the robots.txt rule behind them.
Method and caveats are described on the full study page: one robots.txt fetch per domain, resolved against 30 published AI user-agents, reported as the voluntary policy it is.