Skip to content
THE AI VISIBILITY INDEX · MEASURED 2026-11-12

Is the web letting
the answer engines in?

Across 2188 public sites, the median site allows 100% of known AI crawlers, and 74% are fully open. This index reads live robots policies — the same measurement we publish per domain — and aggregates it by sector and by AI provider. Every figure is measured, carries its sample size, and is dated from the crawl itself.

Openness by sector

Median share of AI crawlers allowed, and the share of sites in the sector that are fully open. Ranked most-open first.

SectorMedian accessFully open
SaaS platforms100%94% · n=257
Hosting and infrastructure100%94% · n=121
Developer references100%91% · n=233
Government and institutions100%91% · n=117
SEO and marketing tools100%90% · n=122
AI companies100%89% · n=156
Finance and business media100%85% · n=144
Travel100%84% · n=117
E-commerce and retail100%76% · n=124
Reference and research100%69% · n=145
Health publishers100%68% · n=114
Social and media platforms99%50% · n=146
Technology media94%41% · n=153
News publishers72%28% · n=239
Full figures as data
[
  {
    "sector": "SaaS platforms",
    "domains": 257,
    "medianAccessPct": 100,
    "meanAccessPct": 99,
    "fullyOpenPct": 94
  },
  {
    "sector": "Hosting and infrastructure",
    "domains": 121,
    "medianAccessPct": 100,
    "meanAccessPct": 99,
    "fullyOpenPct": 94
  },
  {
    "sector": "Developer references",
    "domains": 233,
    "medianAccessPct": 100,
    "meanAccessPct": 97,
    "fullyOpenPct": 91
  },
  {
    "sector": "Government and institutions",
    "domains": 117,
    "medianAccessPct": 100,
    "meanAccessPct": 97,
    "fullyOpenPct": 91
  },
  {
    "sector": "SEO and marketing tools",
    "domains": 122,
    "medianAccessPct": 100,
    "meanAccessPct": 99,
    "fullyOpenPct": 90
  },
  {
    "sector": "AI companies",
    "domains": 156,
    "medianAccessPct": 100,
    "meanAccessPct": 97,
    "fullyOpenPct": 89
  },
  {
    "sector": "Finance and business media",
    "domains": 144,
    "medianAccessPct": 100,
    "meanAccessPct": 95,
    "fullyOpenPct": 85
  },
  {
    "sector": "Travel",
    "domains": 117,
    "medianAccessPct": 100,
    "meanAccessPct": 97,
    "fullyOpenPct": 84
  },
  {
    "sector": "E-commerce and retail",
    "domains": 124,
    "medianAccessPct": 100,
    "meanAccessPct": 94,
    "fullyOpenPct": 76
  },
  {
    "sector": "Reference and research",
    "domains": 145,
    "medianAccessPct": 100,
    "meanAccessPct": 92,
    "fullyOpenPct": 69
  },
  {
    "sector": "Health publishers",
    "domains": 114,
    "medianAccessPct": 100,
    "meanAccessPct": 89,
    "fullyOpenPct": 68
  },
  {
    "sector": "Social and media platforms",
    "domains": 146,
    "medianAccessPct": 99,
    "meanAccessPct": 79,
    "fullyOpenPct": 50
  },
  {
    "sector": "Technology media",
    "domains": 153,
    "medianAccessPct": 94,
    "meanAccessPct": 80,
    "fullyOpenPct": 41
  },
  {
    "sector": "News publishers",
    "domains": 239,
    "medianAccessPct": 72,
    "meanAccessPct": 68,
    "fullyOpenPct": 28
  }
]

Which AI providers get blocked most

Share of the 2188 measured sites that block at least one crawler from each provider, with the 95% confidence interval.

ProviderSites blocking95% CI
Common Crawl19%17–20% · n=2188
ByteDance18%17–20% · n=2188
OpenAI17%15–18% · n=2188
Anthropic16%15–18% · n=2188
Meta16%14–17% · n=2188
Google15%13–16% · n=2188
Apple15%13–16% · n=2188
Amazon15%14–17% · n=2188
Diffbot14%13–16% · n=2188
Cohere13%12–15% · n=2188
Perplexity12%11–13% · n=2188
Webz.io12%11–14% · n=2188
You.com11%9–12% · n=2188
Huawei11%10–12% · n=2188
Timpi10%8–11% · n=2188
Allen Institute for AI8%7–9% · n=2188
ImageSift8%7–10% · n=2188
Mistral AI7%6–8% · n=2188
DuckDuckGo7%6–8% · n=2188
ElevenLabs2%2–3% · n=2183
Microsoft1%0–1% · n=2188

How this is measured

Each domain's live robots policy is parsed and every known AI crawler is classified allowed or blocked (the same measurement published per-domain at /ai-crawler-report/). Access % is the share of AI crawlers a domain allows. Category and provider figures carry their domain count n and a Wilson 95% confidence interval, treating the measured set as a sample of the broader web. Nothing is modelled and no customer data is used.

Evidence class: measured. Source: 2188 public domains, last crawled 2026-11-12. No customer data is used, and nothing on this page is modelled or estimated.

Cite this data

You are welcome to quote or republish these figures. Please credit IndexHalo with a link to this page (CC BY 4.0). All our datasets are on the data page.

CITATION

IndexHalo. "The AI Visibility Index." Measured 12 November 2026. https://www.indexhalo.com/ai-visibility-index/

LINK HTML

<a href="https://www.indexhalo.com/ai-visibility-index/">The AI Visibility Index, IndexHalo</a>