IndexHaloAI crawler access:
developer references
Documentation and Q&A sites — the sources answer engines lean on hardest for technical queries. 34 domains measured; 3 block at least one of the 28 tracked AI crawlers.
3of 34 developer references block at least one AI crawler · median access 100%
Every domain in this category
Sorted by the share of tracked AI crawlers permitted, lowest first. Each report shows all 28 verdicts and the robots.txt rule behind them.
| Domain | Access | Allowed | Blocked | Not listed |
|---|---|---|---|---|
| geeksforgeeks.org | 89% | 25 | 3 | 0 |
| tutorialspoint.com | 89% | 25 | 3 | 0 |
| github.com | 96% | 27 | 1 | 0 |
| angular.io | 100% | 28 | 0 | 0 |
| codecademy.com | 100% | 28 | 0 | 0 |
| crates.io | 100% | 28 | 0 | 0 |
| css-tricks.com | 100% | 28 | 0 | 0 |
| dev.to | 100% | 28 | 0 | 0 |
| developer.mozilla.org | 100% | 28 | 0 | 0 |
| digitalocean.com | 100% | 28 | 0 | 0 |
| docker.com | 100% | 28 | 0 | 0 |
| freecodecamp.org | 100% | 28 | 0 | 0 |
| gitlab.com | 100% | 28 | 0 | 0 |
| golang.org | 100% | 28 | 0 | 0 |
| hashnode.com | 100% | 28 | 0 | 0 |
| kubernetes.io | 100% | 28 | 0 | 0 |
| leetcode.com | 100% | 28 | 0 | 0 |
| mongodb.com | 100% | 28 | 0 | 0 |
| mysql.com | 100% | 28 | 0 | 0 |
| nodejs.org | 100% | 28 | 0 | 0 |
| packagist.org | 100% | 28 | 0 | 0 |
| php.net | 100% | 28 | 0 | 0 |
| postgresql.org | 100% | 28 | 0 | 0 |
| pypi.org | 100% | 28 | 0 | 0 |
| python.org | 100% | 28 | 0 | 0 |
| reactjs.org | 100% | 28 | 0 | 0 |
| redis.io | 100% | 28 | 0 | 0 |
| rubygems.org | 100% | 28 | 0 | 0 |
| sitepoint.com | 100% | 28 | 0 | 0 |
| smashingmagazine.com | 100% | 28 | 0 | 0 |
| sqlite.org | 100% | 28 | 0 | 0 |
| svelte.dev | 100% | 28 | 0 | 0 |
| vuejs.org | 100% | 28 | 0 | 0 |
| w3schools.com | 100% | 28 | 0 | 0 |
Method and caveats are described on the full study page: one robots.txt fetch per domain, resolved against 28 published AI user-agents, reported as the voluntary policy it is. Check any site yourself with the free crawler access checker.