Can AI crawlers
read your site?
Type a website or one of its pages. You'll see which of 30 AI crawlers its robots.txt lets in, what that means for ChatGPT, Gemini, Claude, Perplexity and Google's AI answers, and the lines to add if one is blocked by mistake.
What the check reads
Every AI company that crawls the web names its crawlers, and a site's robots.txt can treat each one in three ways: block or allow it in a group that names it, cover it with the catch-all * group, or not mention it at all. The checker reads the live file, works out which rule applies to each of the 30 AI crawlers in our AI crawler directory, and shows the line that decided it.
It then sums that up for the six AI sources buyers use: ChatGPT, Gemini, Claude, Perplexity, Google AI Mode and Google AI Overviews. It also fetches the page once, because Google's AI features only use pages Google may index, and a noindex tag closes them as firmly as a robots.txt rule.
Training crawlers and answer crawlers are different decisions
Most AI companies run more than one crawler. OpenAI's GPTBot collects content that may train its models, while OAI-SearchBot and ChatGPT-User fetch pages so ChatGPT can answer questions and link its sources. Anthropic splits ClaudeBot from Claude-SearchBot and Claude-User in the same way.
Blocking a training crawler is a business decision, and the checker doesn't judge it. Blocking an answer crawler takes the site out of that product's answers, which is rarely the intention. Of the 389 major websites we read every week, 76 (20%) block GPTBot, but only 27 (7%) block OAI-SearchBot. A single Disallow: / under the wrong User-agent line can do both by accident.
How the suggested fix works
When an answer crawler is blocked, the result gives you the robots.txt lines to change. A crawler blocked by the * group gets a group of its own that allows it. A crawler named in its own group follows only that group and stops reading the * group, so any Disallow lines that keep private folders private are copied into the new group for you.
A crawler that your file blocks by name is a different case: the fix tells you which User-agent line to remove from the group that blocks it. Training crawlers are left exactly as they are. Publish the change, then run the check again to confirm it reads the way you meant.
What a clean result doesn't tell you
robots.txt is a request, not a lock. A firewall, CDN or bot protection service can still refuse a crawler the file allows, so if your server logs show 403 errors for a crawler the checker calls allowed, look at those rules first. Each crawler's directory page explains how to let it through and how to verify that a request really comes from it.
Being readable is also only the first step. This check says who may read a site. It can't say what ChatGPT, Gemini, Claude, Perplexity or Google's AI answers say when a buyer asks about your category: only a search can. IndexHalo runs those searches and keeps every answer and source, on plans from $49 a month for 100 searches.
Keep an eye on it
Rules change without anyone announcing it: a CMS update, a new security plugin or a well-meant edit can close a site to AI search overnight. We read robots.txt files every week, and the rule change log shows which major sites started or stopped blocking which crawler. Leave your address and we'll tell you when a site's rules change.
Two more places to look: the AI crawler study shows what 389 major websites allow, and the free llms.txt validator and generator checks the file that tells AI agents which of your pages matter.
Questions people ask
Why does the checker say a crawler is allowed when my logs show it getting errors?+
robots.txt only states a preference. A firewall, CDN or bot protection service can still refuse a crawler that robots.txt allows, and the crawler then gets a 403 or similar error. Each crawler's page in the AI crawler directory explains how to let it through and how to verify a request really comes from it.
What does not listed mean in the results?+
No group in the robots.txt names that crawler and there is no * group either, so the file says nothing about it. Under the robots.txt standard a crawler with no matching rule may fetch the page, so not listed works like allowed, but it is a default rather than a decision.
Does blocking Google-Extended remove a site from AI Overviews?+
No. Google says Google-Extended does not affect a site's inclusion in Google Search, and AI Overviews and AI Mode are part of Search. It controls whether content may be used for Gemini. AI Overviews and AI Mode follow Googlebot and the page's indexing and snippet rules.
Why are training crawlers left out of the suggested fix?+
Blocking a training crawler such as GPTBot or ClaudeBot is a legitimate business decision, and it does not usually remove a site from AI search answers. The fix only lets in the crawlers that fetch pages to answer questions and link their sources, and leaves every other rule as it is.
Can the checker test a single page rather than the whole site?+
Yes. Type the page's full address and the checker applies the robots.txt rules to that exact path, so a page in a blocked folder shows as blocked even when the home page is open. It also reads that page once to see whether it asks search engines not to index it.