Can AI engines
actually reach you?
Paste a URL. We fetch the site's live robots.txt, resolve all fifteen AI crawlers against it, and show which are allowed, which are blocked, and which rule decided it.
How the check works
Every AI company that crawls the web publishes a user-agent string. Your robots.txt file either names that string in a group of its own, covers it with a wildcard group, or says nothing about it at all. Those three cases produce very different outcomes, and most site owners have never checked which one applies to them.
This tool fetches your robots.txt exactly as a crawler would, parses it into groups, and resolves each of the fifteen user-agents below against the rules. Where a crawler is named directly, the direct rule wins. Where it is not, the wildcard group applies. Where neither exists, the crawler is reported as not listed — it will decide for itself, and in practice most will crawl.
The distinction that catches people out is between training crawlers and search crawlers. Blocking GPTBot stops your content being used to train OpenAI's models. It does not stop ChatGPT citing you, because live citations are fetched by OAI-SearchBot and ChatGPT-User under separate rules. Publishers who block all three because they only meant to decline training remove themselves from AI answers entirely, and usually do not notice for months.
The fifteen crawlers we resolve
These are the user-agents that materially affect whether your content can appear in an AI-generated answer. Each row is checked independently against your file.
| User-agent | Operator | What it does |
|---|---|---|
OAI-SearchBot | OpenAI | search indexing |
GPTBot | OpenAI | model training |
ChatGPT-User | OpenAI | on-demand fetch |
Claude-SearchBot | Anthropic | search indexing |
ClaudeBot | Anthropic | model training |
Claude-User | Anthropic | on-demand fetch |
PerplexityBot | Perplexity | search indexing |
Perplexity-User | Perplexity | on-demand fetch |
Google-Extended | Gemini training control | |
Bingbot | Microsoft | Copilot search index |
Applebot-Extended | Apple | Apple Intelligence control |
Meta-ExternalAgent | Meta | model training |
Amazonbot | Amazon | model training |
Bytespider | ByteDance | model training |
CCBot | Common Crawl | training dataset |
What to do with the result
If your access percentage is high and that was deliberate, nothing needs to change. If crawlers are blocked that you meant to allow, the fix is a few lines in robots.txt — our AI crawler policy generator will compose them for you.
If a crawler shows as not listed, you have no stated policy. That is worth correcting even when the default behaviour is the one you want, because a stated policy is a decision you can point at later and an unstated one is not.
Access is necessary but not sufficient. A crawler that can reach a page still has to find content it can extract, attribute and quote. That is what the full evidence-readiness report measures.
Frequently asked questions
Does blocking GPTBot remove my content from ChatGPT?+
No. GPTBot governs crawling for model training. Answers that cite live sources are fetched by OAI-SearchBot and ChatGPT-User, which are separate user-agents with separate rules. Blocking GPTBot while allowing the other two keeps you citable in ChatGPT answers without contributing to training.
What does 'not listed' mean?+
No rule in robots.txt matches that user-agent, and there is no wildcard group either. The crawler decides for itself, and most default to crawling. Not listed is not the same as allowed — it means you have expressed no policy.
Is robots.txt legally binding on AI crawlers?+
No. robots.txt is a voluntary protocol. Well-behaved crawlers from OpenAI, Anthropic, Google, Microsoft and Perplexity publish their user-agents and honour it. It is a stated preference, not an access control, and it should never be your only protection for sensitive material.
Why do the training and search bots need different answers?+
They serve different purposes. Training crawlers build model weights; you get no attribution and no traffic. Search and on-demand crawlers fetch pages to answer a live question, and those answers carry citations and links. Most publishers want the second even if they decline the first.
My site has no robots.txt. Is that a problem?+
It means every crawler is permitted by default. That is a valid choice, but it is an accident rather than a decision for most sites. Publishing an explicit policy tells each crawler what you intend and gives you a record of it.