Who changed their
AI crawler rules?
Which of 389 major websites started or stopped blocking which AI crawler, from a weekly reading of each site's robots.txt since 17 August 2026.
Latest: November 2026
7 sites changed their rules: 12 new blocks and 1 removed. Newly blocked most often: Claude-SearchBot (2), Perplexity-User (2) and Amazonbot (1). Read the November 2026 report.
| Date first seen | Site | Started blocking | Stopped blocking |
|---|---|---|---|
| 3 November 2026 | bbc.co.uk | Claude-SearchBot, DuckAssistBot | None |
| 3 November 2026 | researchgate.net | Applebot-Extended | None |
| 2 November 2026 | cnbc.com | Claude-SearchBot, Perplexity-User | None |
| 2 November 2026 | fool.com | CCBot, ClaudeBot, GPTBot | None |
| 2 November 2026 | ikea.com | Amazonbot | None |
| 2 November 2026 | metacritic.com | Meta-ExternalAgent, Meta-ExternalFetcher | None |
| 2 November 2026 | newsweek.com | Perplexity-User | Amazonbot |
Every month
| Month | Sites changed | New blocks | Removed |
|---|---|---|---|
| November 2026 | 7 | 12 | 1 |
| October 2026 | 17 | 24 | 8 |
| September 2026 | 9 | 11 | 1 |
| August 2026 | 5 | 29 | 3 |
How we know
Every week we read the robots.txt file of each site in the AI crawler study and compare it with the last reading. The date shown is the first reading where we saw the change, so the site may have made it a few days earlier. We can't see why a site made a change, and a robots.txt rule is a stated preference, not proof of what any crawler did.
We hide anything we can't stand behind:
- A crawler is only compared when it appears in both readings, so adding one to our list never counts as a change.
- A new change is read twice before it is recorded. Changes up to 2 November 2026 were re-read against the site's live file on 3 November 2026 instead, and any we couldn't confirm are left out.
- If a change is undone at the next reading, or a crawler flips three or more times at one site, we treat it as a misread.
- On 19 October 2026 we corrected how we read path rules in robots.txt. The differences that caused at 4 sites came from our code, so they aren't listed.
Download the record as CSV or JSON, free to reuse with credit, or see what each crawler does.
Cite this data
You are welcome to quote or republish these figures. Please credit IndexHalo with a link to this page (CC BY 4.0). All our datasets are on the data page.
IndexHalo. "AI crawler rule changes across major websites." Measured 3 November 2026. https://www.indexhalo.com/ai-crawlers/changes/
<a href="https://www.indexhalo.com/ai-crawlers/changes/">AI crawler rule changes across major websites, IndexHalo</a>