Who actually blocks
the AI crawlers?

Which crawlers get blocked most

Blocking is not evenly distributed. Training crawlers are declined far more often than the search and on-demand fetchers that produce citations — which is the distinction most publishers are actually trying to draw.

How this was measured

For each domain we requested /robots.txt once, over HTTPS, on the apex and then www. The file was parsed into groups and each of the fifteen user-agents resolved against it: a direct group wins, otherwise the wildcard group applies, otherwise the crawler is recorded as not listed. Domains that returned nothing on either host are excluded rather than guessed at — 87 of the sites attempted produced a readable policy.

What this does not measure: whether any crawler honoured the policy, whether the content is useful once fetched, or anything about pages behind authentication. robots.txt is a voluntary protocol and a stated preference.

Every domain measured

Sorted by the share of tracked AI crawlers permitted, lowest first.

Where does your site sit?

Run the same fifteen-crawler resolution against your own domain, free and without an account, with the AI crawler access checker. If the answer surprises you, the policy builder composes a corrected robots.txt block, and the ChatGPT citation guide covers the training-versus-search distinction that catches most publishers out.