MEASURED 21 SEPTEMBER 2026

AI crawler access
for pcmag.com

pcmag.com blocks 7 of the 13 crawlers that can cite a source in an AI answer (OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Meta-ExternalFetcher, DuckAssistBot, YouBot). Content behind those rules is unlikely to appear as a cited source on those surfaces.

39%of 28 tracked AI crawlers can reach pcmag.com — 11 allowed, 17 blocked, 0 not listed

That is below the 100% median across the 379 sites in this dataset. Resolved from https://pcmag.com/robots.txt on 21 September 2026.

Every crawler, and the rule that decided it

A crawler named directly in its own group is governed by that group. One not named falls back to the wildcard group. Where neither exists the crawler is reported as not listed — no policy has been expressed, and in practice most crawlers will proceed.

User-agent	Operator	Purpose	Access	Decided by
`OAI-SearchBot`	OpenAI	search indexing	Blocked	direct rule
`GPTBot`	OpenAI	model training	Blocked	direct rule
`ChatGPT-User`	OpenAI	on-demand fetch	Blocked	direct rule
`Claude-SearchBot`	Anthropic	search indexing	Allowed	direct rule
`ClaudeBot`	Anthropic	model training	Blocked	direct rule
`Claude-User`	Anthropic	on-demand fetch	Allowed	direct rule
`PerplexityBot`	Perplexity	search indexing	Blocked	direct rule
`Perplexity-User`	Perplexity	on-demand fetch	Blocked	direct rule
`Google-Extended`	Google	Gemini training control	Allowed	wildcard policy
`Bingbot`	Microsoft	Copilot search index	Allowed	wildcard policy
`Applebot-Extended`	Apple	Apple Intelligence control	Blocked	direct rule
`Meta-ExternalAgent`	Meta	model training	Blocked	direct rule
`Amazonbot`	Amazon	model training	Blocked	direct rule
`Bytespider`	ByteDance	model training	Blocked	direct rule
`CCBot`	Common Crawl	training dataset	Blocked	direct rule
`GoogleOther`	Google	research and development fetch	Allowed	wildcard policy
`Google-CloudVertexBot`	Google	Vertex AI on-demand fetch	Allowed	wildcard policy
`Meta-ExternalFetcher`	Meta	on-demand fetch	Blocked	direct rule
`MistralAI-User`	Mistral AI	on-demand fetch	Allowed	direct rule
`cohere-ai`	Cohere	model training	Blocked	direct rule
`DuckAssistBot`	DuckDuckGo	on-demand fetch	Blocked	direct rule
`YouBot`	You.com	search indexing	Blocked	direct rule
`Timpibot`	Timpi	training dataset	Blocked	direct rule
`AI2Bot`	Allen Institute for AI	model training	Allowed	direct rule
`Diffbot`	Diffbot	structured-data extraction	Blocked	direct rule
`ImagesiftBot`	ImageSift	training dataset	Allowed	direct rule
`PetalBot`	Huawei	search indexing	Allowed	direct rule
`Omgilibot`	Webz.io	training dataset	Allowed	wildcard policy

What this does and does not tell you

It tells you what pcmag.com states in its robots.txt, which is the file every well-behaved AI crawler consults before fetching. It is a stated preference and a voluntary protocol — not an access control, and not evidence about what any crawler actually did.

It also says nothing about whether the content is useful to an answer engine once fetched. Access is the precondition; extractable passages, visible dates, named authorship and sourced claims decide whether a reachable page is actually cited. That second layer is what the free evidence-readiness report measures.

Finally, robots.txt changes. This page reflects pcmag.com as it was on 21 September 2026. To see the position right now, run it through the live crawler access checker.

Check your own site

The same 28 user-agents, resolved against your live robots.txt, in about five seconds and without an account: AI crawler access checker. If the result is not what you intended, the robots.txt policy builder composes a corrected block, and how to get cited by ChatGPT explains why blocking GPTBot alone does not remove you from ChatGPT answers.

Browse the full study of 379 measured domains, or the technology media category, to see how this compares.

Embed this result

A live badge for pcmag.com, updated whenever this report refreshes. Paste it into a README or site footer:

EMBED HTML

<a href="https://www.indexhalo.com/ai-crawler-report/pcmag.com/">
  <img src="https://www.indexhalo.com/badge/ai-crawler-access/pcmag.com.svg"
       alt="AI crawler access for pcmag.com: 39%" height="20">
</a>