IndexHaloAI crawler access:
reference and research
Encyclopaedias, journals and universities — the citation bedrock of the web. 25 domains measured; 8 block at least one of the 28 tracked AI crawlers.
8of 25 reference and research block at least one AI crawler · median access 100%
Every domain in this category
Sorted by the share of tracked AI crawlers permitted, lowest first. Each report shows all 28 verdicts and the robots.txt rule behind them.
| Domain | Access | Allowed | Blocked | Not listed |
|---|---|---|---|---|
| sciencedirect.com | 18% | 5 | 23 | 0 |
| academia.edu | 32% | 9 | 19 | 0 |
| nature.com | 43% | 12 | 16 | 0 |
| sciencemag.org | 82% | 23 | 5 | 0 |
| dictionary.com | 89% | 25 | 3 | 0 |
| ssrn.com | 89% | 25 | 3 | 0 |
| jstor.org | 96% | 27 | 1 | 0 |
| pnas.org | 96% | 27 | 1 | 0 |
| acm.org | 100% | 28 | 0 | 0 |
| archive.org | 100% | 28 | 0 | 0 |
| arxiv.org | 100% | 28 | 0 | 0 |
| berkeley.edu | 100% | 28 | 0 | 0 |
| britannica.com | 100% | 28 | 0 | 0 |
| cam.ac.uk | 100% | 28 | 0 | 0 |
| cmu.edu | 100% | 28 | 0 | 0 |
| gutenberg.org | 100% | 28 | 0 | 0 |
| merriam-webster.com | 100% | 28 | 0 | 0 |
| mit.edu | 100% | 28 | 0 | 0 |
| plos.org | 100% | 28 | 0 | 0 |
| pubmed.ncbi.nlm.nih.gov | 100% | 28 | 0 | 0 |
| researchgate.net | 100% | 28 | 0 | 0 |
| springer.com | 100% | 28 | 0 | 0 |
| stanford.edu | 100% | 28 | 0 | 0 |
| wikipedia.org | 100% | 28 | 0 | 0 |
| wiktionary.org | 100% | 28 | 0 | 0 |
Method and caveats are described on the full study page: one robots.txt fetch per domain, resolved against 28 published AI user-agents, reported as the voluntary policy it is. Check any site yourself with the free crawler access checker.