IndexHaloIs the web letting
the answer engines in?
Across 375 public sites, the median site allows 100% of known AI crawlers, and 66% are fully open. This index reads live robots policies — the same measurement we publish per domain — and aggregates it by sector and by AI provider. Every figure is measured, carries its sample size, and is dated from the crawl itself.
Openness by sector
Median share of AI crawlers allowed, and the share of sites in the sector that are fully open. Ranked most-open first.
| Sector | Median access | Fully open | |
|---|---|---|---|
| Hosting and infrastructure | 100% | 100% · n=21 | |
| Government and institutions | 100% | 94% · n=18 | |
| SEO and marketing tools | 100% | 92% · n=25 | |
| Developer references | 100% | 91% · n=32 | |
| SaaS platforms | 100% | 89% · n=53 | |
| AI companies | 100% | 85% · n=26 | |
| E-commerce and retail | 100% | 74% · n=19 | |
| Reference and research | 100% | 68% · n=25 | |
| Travel | 100% | 63% · n=19 | |
| Finance and business media | 100% | 62% · n=29 | |
| Health publishers | 100% | 56% · n=16 | |
| Social and media platforms | 90% | 42% · n=26 | |
| Technology media | 74% | 23% · n=22 | |
| News publishers | 40% | 9% · n=44 |
Full figures as data
[
{
"sector": "Hosting and infrastructure",
"domains": 21,
"medianAccessPct": 100,
"meanAccessPct": 100,
"fullyOpenPct": 100
},
{
"sector": "Government and institutions",
"domains": 18,
"medianAccessPct": 100,
"meanAccessPct": 100,
"fullyOpenPct": 94
},
{
"sector": "SEO and marketing tools",
"domains": 25,
"medianAccessPct": 100,
"meanAccessPct": 100,
"fullyOpenPct": 92
},
{
"sector": "Developer references",
"domains": 32,
"medianAccessPct": 100,
"meanAccessPct": 99,
"fullyOpenPct": 91
},
{
"sector": "SaaS platforms",
"domains": 53,
"medianAccessPct": 100,
"meanAccessPct": 98,
"fullyOpenPct": 89
},
{
"sector": "AI companies",
"domains": 26,
"medianAccessPct": 100,
"meanAccessPct": 99,
"fullyOpenPct": 85
},
{
"sector": "E-commerce and retail",
"domains": 19,
"medianAccessPct": 100,
"meanAccessPct": 89,
"fullyOpenPct": 74
},
{
"sector": "Reference and research",
"domains": 25,
"medianAccessPct": 100,
"meanAccessPct": 90,
"fullyOpenPct": 68
},
{
"sector": "Travel",
"domains": 19,
"medianAccessPct": 100,
"meanAccessPct": 88,
"fullyOpenPct": 63
},
{
"sector": "Finance and business media",
"domains": 29,
"medianAccessPct": 100,
"meanAccessPct": 84,
"fullyOpenPct": 62
},
{
"sector": "Health publishers",
"domains": 16,
"medianAccessPct": 100,
"meanAccessPct": 87,
"fullyOpenPct": 56
},
{
"sector": "Social and media platforms",
"domains": 26,
"medianAccessPct": 90,
"meanAccessPct": 72,
"fullyOpenPct": 42
},
{
"sector": "Technology media",
"domains": 22,
"medianAccessPct": 74,
"meanAccessPct": 63,
"fullyOpenPct": 23
},
{
"sector": "News publishers",
"domains": 44,
"medianAccessPct": 40,
"meanAccessPct": 48,
"fullyOpenPct": 9
}
]Which AI providers get blocked most
Share of the 375 measured sites that block at least one crawler from each provider, with the 95% confidence interval.
| Provider | Sites blocking | 95% CI |
|---|---|---|
| Common Crawl | 24% | 20–29% · n=375 |
| ByteDance | 23% | 19–28% · n=375 |
| Anthropic | 22% | 18–26% · n=375 |
| OpenAI | 20% | 17–25% · n=375 |
| Apple | 20% | 16–24% · n=375 |
| Diffbot | 19% | 15–23% · n=375 |
| 18% | 15–23% · n=375 | |
| Meta | 18% | 14–22% · n=375 |
| Amazon | 18% | 14–22% · n=375 |
| Perplexity | 17% | 14–21% · n=375 |
| Cohere | 17% | 14–21% · n=375 |
| Timpi | 15% | 11–19% · n=375 |
| Webz.io | 15% | 12–19% · n=375 |
| You.com | 14% | 11–18% · n=375 |
| ImageSift | 13% | 10–17% · n=375 |
| Huawei | 12% | 9–16% · n=375 |
| Mistral AI | 11% | 8–14% · n=375 |
| DuckDuckGo | 11% | 8–14% · n=375 |
| Allen Institute for AI | 11% | 8–14% · n=375 |
| Microsoft | 1% | 0–2% · n=375 |
How this is measured
Each domain's live robots policy is parsed and every known AI crawler is classified allowed or blocked (the same measurement published per-domain at /ai-crawler-report/). Access % is the share of AI crawlers a domain allows. Category and provider figures carry their domain count n and a Wilson 95% confidence interval, treating the measured set as a sample of the broader web. Nothing is modelled and no customer data is used.
Evidence class: measured. Source: 375 public domains, last crawled 2026-09-28. No customer data is used, and nothing on this page is modelled or estimated.