AI crawler access:
developer references
Documentation and Q&A sites — the sources answer engines lean on hardest for technical queries. 34 domains measured; 2 block at least one of the fifteen tracked AI crawlers.
2of 34 developer references block at least one AI crawler · median access 100%
Every domain in this category
Sorted by the share of tracked AI crawlers permitted, lowest first. Each report shows all fifteen verdicts and the robots.txt rule behind them.
| Domain | Access | Allowed | Blocked | Not listed |
|---|---|---|---|---|
| tutorialspoint.com | 80% | 12 | 3 | 0 |
| geeksforgeeks.org | 87% | 13 | 2 | 0 |
| angular.io | 100% | 15 | 0 | 0 |
| codecademy.com | 100% | 15 | 0 | 0 |
| crates.io | 100% | 15 | 0 | 0 |
| css-tricks.com | 100% | 15 | 0 | 0 |
| dev.to | 100% | 15 | 0 | 0 |
| developer.mozilla.org | 100% | 15 | 0 | 0 |
| digitalocean.com | 100% | 15 | 0 | 0 |
| docker.com | 100% | 15 | 0 | 0 |
| freecodecamp.org | 100% | 15 | 0 | 0 |
| github.com | 100% | 15 | 0 | 0 |
| gitlab.com | 100% | 15 | 0 | 0 |
| golang.org | 100% | 15 | 0 | 0 |
| hashnode.com | 100% | 15 | 0 | 0 |
| kubernetes.io | 100% | 15 | 0 | 0 |
| leetcode.com | 100% | 15 | 0 | 0 |
| mongodb.com | 100% | 15 | 0 | 0 |
| mysql.com | 100% | 15 | 0 | 0 |
| nodejs.org | 100% | 15 | 0 | 0 |
| packagist.org | 100% | 15 | 0 | 0 |
| php.net | 100% | 15 | 0 | 0 |
| postgresql.org | 100% | 15 | 0 | 0 |
| pypi.org | 100% | 15 | 0 | 0 |
| python.org | 100% | 15 | 0 | 0 |
| reactjs.org | 100% | 15 | 0 | 0 |
| redis.io | 100% | 15 | 0 | 0 |
| rubygems.org | 100% | 15 | 0 | 0 |
| sitepoint.com | 100% | 15 | 0 | 0 |
| smashingmagazine.com | 100% | 15 | 0 | 0 |
| sqlite.org | 100% | 15 | 0 | 0 |
| svelte.dev | 100% | 15 | 0 | 0 |
| vuejs.org | 100% | 15 | 0 | 0 |
| w3schools.com | 100% | 15 | 0 | 0 |
Method and caveats are described on the full study page: one robots.txt fetch per domain, resolved against fifteen published AI user-agents, reported as the voluntary policy it is. Check any site yourself with the free crawler access checker.