Which websites
block AI crawlers?

Which crawlers get blocked most

Blocking is not evenly distributed. Training crawlers are declined far more often than the search and on-demand fetchers that produce citations — which is the distinction most publishers are actually trying to draw.

How this was measured

For each domain we requested /robots.txt once, over HTTPS, on the apex and then www. The file was parsed into groups and each of the 30 user-agents resolved against it: a direct group wins, otherwise the wildcard group applies, otherwise the crawler is recorded as not listed. Domains that returned nothing on either host are excluded rather than guessed at. A site we have read before keeps that reading, with its date, for up to 13 weeks, so 389 of the sites attempted have a readable policy.

What this does not measure: whether any crawler honoured the policy, whether the content is useful once fetched, or anything about pages behind authentication. robots.txt is a voluntary protocol and a stated preference.

Browse by category

Every domain measured

Sorted by the share of tracked AI crawlers permitted, lowest first.

Where does your site sit?

Review your current robots.txt policy and read the ChatGPT citation guide for the training-versus-search distinction. The AI crawler directory explains what each bot does, and the change log shows which sites changed their rules.