Is the web letting
the answer engines in?
Across 2188 public sites, the median site allows 100% of known AI crawlers, and 74% are fully open. This index reads live robots policies — the same measurement we publish per domain — and aggregates it by sector and by AI provider. Every figure is measured, carries its sample size, and is dated from the crawl itself.
Openness by sector
Median share of AI crawlers allowed, and the share of sites in the sector that are fully open. Ranked most-open first.
| Sector | Median access | Fully open |
|---|---|---|
| SaaS platforms | 100% | 94% · n=257 |
| Hosting and infrastructure | 100% | 94% · n=121 |
| Developer references | 100% | 91% · n=233 |
| Government and institutions | 100% | 91% · n=117 |
| SEO and marketing tools | 100% | 90% · n=122 |
| AI companies | 100% | 89% · n=156 |
| Finance and business media | 100% | 85% · n=144 |
| Travel | 100% | 84% · n=117 |
| E-commerce and retail | 100% | 76% · n=124 |
| Reference and research | 100% | 69% · n=145 |
| Health publishers | 100% | 68% · n=114 |
| Social and media platforms | 99% | 50% · n=146 |
| Technology media | 94% | 41% · n=153 |
| News publishers | 72% | 28% · n=239 |
Full figures as data
[
{
"sector": "SaaS platforms",
"domains": 257,
"medianAccessPct": 100,
"meanAccessPct": 99,
"fullyOpenPct": 94
},
{
"sector": "Hosting and infrastructure",
"domains": 121,
"medianAccessPct": 100,
"meanAccessPct": 99,
"fullyOpenPct": 94
},
{
"sector": "Developer references",
"domains": 233,
"medianAccessPct": 100,
"meanAccessPct": 97,
"fullyOpenPct": 91
},
{
"sector": "Government and institutions",
"domains": 117,
"medianAccessPct": 100,
"meanAccessPct": 97,
"fullyOpenPct": 91
},
{
"sector": "SEO and marketing tools",
"domains": 122,
"medianAccessPct": 100,
"meanAccessPct": 99,
"fullyOpenPct": 90
},
{
"sector": "AI companies",
"domains": 156,
"medianAccessPct": 100,
"meanAccessPct": 97,
"fullyOpenPct": 89
},
{
"sector": "Finance and business media",
"domains": 144,
"medianAccessPct": 100,
"meanAccessPct": 95,
"fullyOpenPct": 85
},
{
"sector": "Travel",
"domains": 117,
"medianAccessPct": 100,
"meanAccessPct": 97,
"fullyOpenPct": 84
},
{
"sector": "E-commerce and retail",
"domains": 124,
"medianAccessPct": 100,
"meanAccessPct": 94,
"fullyOpenPct": 76
},
{
"sector": "Reference and research",
"domains": 145,
"medianAccessPct": 100,
"meanAccessPct": 92,
"fullyOpenPct": 69
},
{
"sector": "Health publishers",
"domains": 114,
"medianAccessPct": 100,
"meanAccessPct": 89,
"fullyOpenPct": 68
},
{
"sector": "Social and media platforms",
"domains": 146,
"medianAccessPct": 99,
"meanAccessPct": 79,
"fullyOpenPct": 50
},
{
"sector": "Technology media",
"domains": 153,
"medianAccessPct": 94,
"meanAccessPct": 80,
"fullyOpenPct": 41
},
{
"sector": "News publishers",
"domains": 239,
"medianAccessPct": 72,
"meanAccessPct": 68,
"fullyOpenPct": 28
}
]Which AI providers get blocked most
Share of the 2188 measured sites that block at least one crawler from each provider, with the 95% confidence interval.
| Provider | Sites blocking | 95% CI |
|---|---|---|
| Common Crawl | 19% | 17–20% · n=2188 |
| ByteDance | 18% | 17–20% · n=2188 |
| OpenAI | 17% | 15–18% · n=2188 |
| Anthropic | 16% | 15–18% · n=2188 |
| Meta | 16% | 14–17% · n=2188 |
| 15% | 13–16% · n=2188 | |
| Apple | 15% | 13–16% · n=2188 |
| Amazon | 15% | 14–17% · n=2188 |
| Diffbot | 14% | 13–16% · n=2188 |
| Cohere | 13% | 12–15% · n=2188 |
| Perplexity | 12% | 11–13% · n=2188 |
| Webz.io | 12% | 11–14% · n=2188 |
| You.com | 11% | 9–12% · n=2188 |
| Huawei | 11% | 10–12% · n=2188 |
| Timpi | 10% | 8–11% · n=2188 |
| Allen Institute for AI | 8% | 7–9% · n=2188 |
| ImageSift | 8% | 7–10% · n=2188 |
| Mistral AI | 7% | 6–8% · n=2188 |
| DuckDuckGo | 7% | 6–8% · n=2188 |
| ElevenLabs | 2% | 2–3% · n=2183 |
| Microsoft | 1% | 0–1% · n=2188 |
How this is measured
Each domain's live robots policy is parsed and every known AI crawler is classified allowed or blocked (the same measurement published per-domain at /ai-crawler-report/). Access % is the share of AI crawlers a domain allows. Category and provider figures carry their domain count n and a Wilson 95% confidence interval, treating the measured set as a sample of the broader web. Nothing is modelled and no customer data is used.
Evidence class: measured. Source: 2188 public domains, last crawled 2026-11-12. No customer data is used, and nothing on this page is modelled or estimated.
Cite this data
You are welcome to quote or republish these figures. Please credit IndexHalo with a link to this page (CC BY 4.0). All our datasets are on the data page.
CITATION
IndexHalo. "The AI Visibility Index." Measured 12 November 2026. https://www.indexhalo.com/ai-visibility-index/LINK HTML
<a href="https://www.indexhalo.com/ai-visibility-index/">The AI Visibility Index, IndexHalo</a>