AI crawler access:
technology media
Outlets covering the AI industry, and deciding whether its crawlers may read them. 23 domains measured; 17 block at least one of the fifteen tracked AI crawlers.
17of 23 technology media block at least one AI crawler · median access 73%
Every domain in this category
Sorted by the share of tracked AI crawlers permitted, lowest first. Each report shows all fifteen verdicts and the robots.txt rule behind them.
| Domain | Access | Allowed | Blocked | Not listed |
|---|---|---|---|---|
| lifehacker.com | 13% | 2 | 13 | 0 |
| mashable.com | 13% | 2 | 13 | 0 |
| zdnet.com | 13% | 2 | 13 | 0 |
| arstechnica.com | 27% | 4 | 11 | 0 |
| theverge.com | 27% | 4 | 11 | 0 |
| wired.com | 27% | 4 | 11 | 0 |
| hackernoon.com | 40% | 6 | 9 | 0 |
| theregister.com | 40% | 6 | 9 | 0 |
| howtogeek.com | 60% | 9 | 6 | 0 |
| makeuseof.com | 60% | 9 | 6 | 0 |
| xda-developers.com | 60% | 9 | 6 | 0 |
| venturebeat.com | 73% | 11 | 4 | 0 |
| androidcentral.com | 80% | 12 | 3 | 0 |
| bleepingcomputer.com | 80% | 12 | 3 | 0 |
| gizmodo.com | 80% | 12 | 3 | 0 |
| techradar.com | 80% | 12 | 3 | 0 |
| tomshardware.com | 80% | 12 | 3 | 0 |
| anandtech.com | 100% | 0 | 0 | 15 |
| digitaltrends.com | 100% | 15 | 0 | 0 |
| engadget.com | 100% | 15 | 0 | 0 |
| macrumors.com | 100% | 15 | 0 | 0 |
| slashdot.org | 100% | 15 | 0 | 0 |
| thenextweb.com | 100% | 15 | 0 | 0 |
Method and caveats are described on the full study page: one robots.txt fetch per domain, resolved against fifteen published AI user-agents, reported as the voluntary policy it is. Check any site yourself with the free crawler access checker.