Which news publishers
block AI crawlers?
The sites with the most at stake: their archives train models and their reporting feeds live answers. 182 domains measured; 137 block at least one of the 30 tracked AI crawlers.
137of 182 news publishers block at least one AI crawler · median access 63%
The AI crawlers these sites block most
CCBot is blocked most often here, by 116 of 182 sites. What blocking each one means is in the AI crawler directory.
| User-agent | Operator | Purpose | Sites blocking |
|---|---|---|---|
CCBot | Common Crawl | training dataset | 116 of 182 |
ClaudeBot | Anthropic | model training | 108 of 182 |
Bytespider | ByteDance | model training | 105 of 182 |
GPTBot | OpenAI | model training | 104 of 182 |
PerplexityBot | Perplexity | search indexing | 102 of 182 |
Applebot-Extended | Apple | Apple Intelligence control | 93 of 182 |
Meta-ExternalAgent | Meta | model training | 93 of 182 |
cohere-ai | Cohere | model training | 92 of 182 |
Every domain in this category
Sorted by the share of tracked AI crawlers permitted, lowest first. Each report shows all 30 verdicts and the robots.txt rule behind them.
Method and caveats are described on the full study page: one robots.txt fetch per domain, resolved against 30 published AI user-agents, reported as the voluntary policy it is.