AI crawler rule changes: September 2026

Which sites changed what

Other months

November 2026, October 2026, August 2026, or the full log.

How we know

Every week we read the robots.txt file of each site in the AI crawler study and compare it with the last reading. The date shown is the first reading where we saw the change, so the site may have made it a few days earlier. We can't see why a site made a change, and a robots.txt rule is a stated preference, not proof of what any crawler did.

We hide anything we can't stand behind:

• A crawler is only compared when it appears in both readings, so adding one to our list never counts as a change.
• A new change is read twice before it is recorded. Changes up to 2 November 2026 were re-read against the site's live file on 3 November 2026 instead, and any we couldn't confirm are left out.
• If a change is undone at the next reading, or a crawler flips three or more times at one site, we treat it as a misread.
• On 19 October 2026 we corrected how we read path rules in robots.txt. The differences that caused at 4 sites came from our code, so they aren't listed.

Download the record as CSV or JSON, free to reuse with credit, or see what each crawler does.

IndexHalo. "AI crawler rule changes: September 2026." Measured 28 September 2026. https://www.indexhalo.com/ai-crawlers/changes/2026-09/

<a href="https://www.indexhalo.com/ai-crawlers/changes/2026-09/">AI crawler rule changes: September 2026, IndexHalo</a>