What crawler logs can prove
A server record can establish that a request reached your infrastructure at a stated time, requested a particular URL, declared a particular user agent, and received a recorded HTTP response. That is operationally useful. It lets a team identify intended crawlers receiving 403, 404, 429, or 5xx responses, compare requested pages with the site inventory, and measure when important content was last requested.
The wording matters. A user-agent string is declared by the requester and can be imitated. Unless a separate verification process confirms the source network, the defensible phrase is “request declaring OAI-SearchBot,” not “OpenAI definitely crawled this page.”
A crawler request is not proof that a model trained on, cited, recommended, or used the requested material in an answer.
The minimum useful evidence record
Keep the request time, HTTP method, canonical same-site URL or path, response status, response-byte count, declared user agent, and a transparent crawler classification. Those fields are enough to measure availability and coverage without turning a GEO report into a visitor-surveillance product.
IP addresses are often present in raw server logs, but most page-level crawler analysis does not need to retain them. A privacy-conscious workflow can discard IPs and the raw file before storing the normalized evidence. If a team later needs network verification, that should be a distinct, governed process with a documented legal basis and retention policy.
Classify purpose, not just vendor
One company may publish several crawler identities for different purposes. Search discovery, user-requested retrieval, general AI crawling, model-use controls, and public web archives should not be blended into one number. Separate identities allow the report to answer a practical question: did a discovery crawler request the page, did a user-triggered agent retrieve it, or did only a general search crawler appear?
Join logs to the complete site inventory
Log counts become useful when mapped to a current whole-site crawl. For every canonical page, record recognized requests, distinct crawler families, response errors, and last-seen time. Then surface high-value analysed pages with no recognized request in the reporting window.
An unseen page is not automatically blocked. The export may cover only one edge service, one hostname, or a short period. The report should therefore label absence as “not observed within this log coverage” and ask the team to validate the export boundary before changing robots rules, internal links, or infrastructure.
Turn evidence into verifiable actions
- Fix response failures. For every 4xx or 5xx crawler request, identify the intended public response and remove accidental authentication, firewall, rate-limit, or routing failures.
- Close coverage gaps. Confirm the log scope, then improve sitemap inclusion, canonical consistency, internal discovery, and intended crawler access for unseen priority pages.
- Reduce crawl waste. Identify repeated crawler requests to redirects, parameter variants, duplicates, and low-value surfaces; consolidate where appropriate without blocking legitimate discovery.
- Define the test. Require a later recognized request and successful 2xx response for every affected URL.
Use a real-time feed after the baseline
A historical export establishes the baseline. A secure event hook can then receive sanitized crawler events from an edge worker, server, or log pipeline. Rotate the secret, limit batch size and request rate, reject external URLs, and store only normalized fields. This supports faster verification after a release without asking the customer for any model-provider key.
What the professional report should show
- Recognized request volume, identities, purposes, unique pages, errors, coverage, and last-seen time.
- A crawler-by-crawler matrix with request, page, error, and recency counts.
- A page table joined to GEO score and current crawl inventory.
- The exact URLs with response failures and the latest observed status.
- Unseen priority pages, clearly qualified by the imported evidence period.
- Assigned actions with a later-event verification rule.
- A permanent disclosure that logs do not prove training, citation, or downstream use.