IndexHaloAI crawler access:
government and institutions
Public bodies and international organisations. 18 domains measured; 1 block at least one of the 28 tracked AI crawlers.
1of 18 government and institutions block at least one AI crawler · median access 100%
Every domain in this category
Sorted by the share of tracked AI crawlers permitted, lowest first. Each report shows all 28 verdicts and the robots.txt rule behind them.
| Domain | Access | Allowed | Blocked | Not listed |
|---|---|---|---|---|
| who.int | 96% | 0 | 1 | 27 |
| australia.gov.au | 100% | 0 | 0 | 28 |
| belastingdienst.nl | 100% | 28 | 0 | 0 |
| bund.de | 100% | 28 | 0 | 0 |
| canada.ca | 100% | 28 | 0 | 0 |
| cdc.gov | 100% | 0 | 0 | 28 |
| europa.eu | 100% | 28 | 0 | 0 |
| fda.gov | 100% | 28 | 0 | 0 |
| gov.uk | 100% | 28 | 0 | 0 |
| imf.org | 100% | 28 | 0 | 0 |
| irs.gov | 100% | 28 | 0 | 0 |
| nih.gov | 100% | 28 | 0 | 0 |
| oecd.org | 100% | 28 | 0 | 0 |
| sec.gov | 100% | 28 | 0 | 0 |
| skatteverket.se | 100% | 28 | 0 | 0 |
| un.org | 100% | 28 | 0 | 0 |
| usa.gov | 100% | 28 | 0 | 0 |
| worldbank.org | 100% | 28 | 0 | 0 |
Method and caveats are described on the full study page: one robots.txt fetch per domain, resolved against 28 published AI user-agents, reported as the voluntary policy it is. Check any site yourself with the free crawler access checker.