AI search data,
free to reuse
The figures behind our public reports, as files you can download, chart and quote. Reuse them for anything, as long as you credit IndexHalo with a link.
Key findings
- 131 of 389 major websites block at least one AI crawler in robots.txt, and 246 allow all 30 we track.
- The most blocked AI crawler is CCBot (Common Crawl), blocked by 94 of 389 sites.
- Across 869 AI answers to buyer questions, the websites cited most were g2.com (179 answers), zapier.com (168) and pcmag.com (139).
- In 9 of 29 categories, one brand was named in every AI answer, including HubSpot, Shopify, Amplitude, Stripe, Zoom.
Datasets
| Dataset | Covers | Measured | Download |
|---|---|---|---|
| AI crawler access across major websites | 389 websites and 30 AI crawlers, from each site's live robots.txt. Refreshed weekly. | 3 November 2026 | CSV · JSON |
| AI crawler rule changes | Which of those websites started or stopped blocking which crawler, from weekly readings. Refreshed weekly. | 3 November 2026 | CSV · JSON |
| Which brands AI recommends | 29 categories and 869 AI answers from ChatGPT, Gemini, Google AI Mode, Claude and Perplexity. | 3 November 2026 | CSV · JSON |
| The websites AI cites | The sites those 869 answers linked to, overall and by category. | 3 November 2026 | CSV · JSON |
Licence and limits
The data is published under CC BY 4.0: use it freely, including commercially, with a credit and a link to this page.
The crawler data reports what each site states in its robots.txt, a voluntary protocol, and not what any crawler did. The AI recommendation data is a dated sample: AI answers vary from run to run, so small differences are not meaningful. It contains counts and cited websites, and no answer text.
Writing about these figures? Our press kit has company facts and how to reach us.
Cite this data
You are welcome to quote or republish these figures. Please credit IndexHalo with a link to this page (CC BY 4.0). All our datasets are on the data page.
IndexHalo. "IndexHalo open data on AI search." Measured 3 November 2026. https://www.indexhalo.com/data/
<a href="https://www.indexhalo.com/data/">IndexHalo open data on AI search, IndexHalo</a>