Which technology media
block AI crawlers?
Outlets covering the AI industry, and deciding whether its crawlers may read them. 128 domains measured; 78 block at least one of the 36 tracked AI crawlers.
78of 128 technology media block at least one AI crawler · median access 94%
The AI crawlers these sites block most
Bytespider is blocked most often here, by 62 of 128 sites. What blocking each one means is in the AI crawler directory.
| User-agent | Operator | Purpose | Sites blocking |
|---|---|---|---|
Bytespider | ByteDance | model training | 62 of 128 |
Applebot-Extended | Apple | Apple Intelligence control | 55 of 128 |
Meta-ExternalAgent | Meta | model training | 55 of 128 |
Diffbot | Diffbot | structured-data extraction | 53 of 128 |
CCBot | Common Crawl | training dataset | 51 of 128 |
Amazonbot | Amazon | model training | 48 of 128 |
ClaudeBot | Anthropic | model training | 45 of 128 |
GPTBot | OpenAI | model training | 42 of 128 |
Every domain in this category
Sorted by the share of tracked AI crawlers permitted, lowest first. Each report shows all 36 verdicts and the robots.txt rule behind them.
Method and caveats are described on the full study page: one robots.txt fetch per domain, resolved against 36 published AI user-agents, reported as the voluntary policy it is.