Which AI bots
actually hit you?
Paste up to 5 MB of nginx or Apache access log. We parse every AI crawler hit, show what each one fetched and how often, and flag where your robots.txt policy and observed behaviour disagree.
Stated policy versus observed behaviour
Every other tool on this page — and most tools in the category — reads your robots.txt and reports what it says. This one reads your server logs and reports what crawlers did. The two are not the same thing, and the gap between them is where the interesting decisions live.
A crawler you explicitly allow that has not visited in thirty days is not a policy problem — it is a discovery gap, and the fix is sitemaps and internal links to the pages that matter, not another line in robots.txt. A crawler you blocked that is still hitting your origin means the policy is not enforced at the edge, which is a different and more urgent problem. Neither shows up in a robots.txt check. Both show up here, because the verdicts cross-reference the policy you paste against the traffic you measured.
This is the free tip of a larger idea. The IndexHalo platform ingests server and CDN logs continuously and turns them into a stated-vs-observed ledger per crawler — who started fetching you, who stopped, and whether what your robots.txt declares matches what the bots actually do. The dataset behind the public AI Access Index is the stated half; your logs are the observed half. Together they are a moat with two layers, and no comparable tool has the second.
What the report contains
For each of the fifteen tracked AI crawlers, the analyzer reports the number of hits in your window, the count of unique visitor hashes, the pages it fetched most often, the client-error (4xx) and server-error (5xx) rates with their sample sizes, and the first and last time it was seen. Crawlers with no hits are listed too, with a verdict that depends on whether your robots.txt allows, blocks, or ignores them.
Unprofiled bot-like agents — anything whose user-agent contains bot, crawl, spider and the rest but is not one of the fifteen — surface as discovery leads rather than verdicts. You see what else is reaching you without us guessing what it is for. The window itself is measured from the timestamps in your paste, so the report never claims a freshness the logs do not support.
Privacy
Raw IP addresses are SHA-256-hashed in memory the moment a line is parsed, and only the hash is used to count unique visitors. The parsed request lines never leave the server, and nothing from your paste is persisted after the report is returned. The report carries aggregate counts and verdicts only — never a raw address, never a path tied to a visitor. This is a measurement tool, not a collection.
If you need this analysis continuously, on live logs, without pasting anything, that is the platform's Crawler Ledger — the same engine wired to ingest from your CDN or server on a schedule. The free tool here is the honest preview: the same parser, the same verdicts, the same evidence class, bounded only by what you paste.
Frequently asked questions
What log format does the analyzer accept?+
nginx and Apache combined log format — the default your web server writes unless you reconfigured it. Each line needs the remote host, a timestamp, the request line, the status code, and the user-agent. Lines that don't match are counted as unparsed and skipped, never silently treated as hits, so a misformatted paste reads as a low parse rate rather than a wrong answer.
Do you see or store my IP addresses?+
No. Raw IPs are SHA-256-hashed in memory the moment a line is parsed, and only the hash is used to count unique visitors. The parsed request lines never leave the server, and nothing from your paste is stored after the report is returned. The report itself carries only aggregate counts and verdicts — never a raw address.
How is this different from checking robots.txt?+
robots.txt states your policy — what you permit. This reads your logs — what crawlers actually did. The two frequently disagree: a bot you allow that never visits is a discovery gap, and a bot you blocked that still hits means the policy isn't enforced at the edge. The cross-reference flags both, which is the whole point of the tool.
Which AI crawlers does it recognise?+
The same fifteen the IndexHalo crawler checker resolves: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Amazonbot, Bytespider, CCBot and the rest. Any other bot-like agent in your logs surfaces as a discovery lead, never as a verdict, so you can see what else is reaching you without us guessing what it is for.
How much log can I paste, and how is the window measured?+
Up to five megabytes, or about a hundred thousand lines — enough for a busy week on most sites. Larger excerpts are rejected with a clear limit rather than truncated silently. The window is measured from the newest and oldest timestamps in your paste, so an old upload reads as an old window, never as fresh demand.
What does 'stale' mean for a crawler?+
A crawler whose last hit in your window was thirty or more days before the window's end. It is still in the dataset but demand has gone quiet — fresh, citable pages may restore it, or the engine may have moved on. The verdict tells you which, and distinguishes a stale crawler from one that is simply absent.