Does theaustralian.com.au block ChatGPT?
theaustralian.com.au blocks 12 of the 16 crawlers that can cite a source in an AI answer (Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Meta-ExternalFetcher, MistralAI-User, DuckAssistBot, YouBot, PetalBot, ElevenlabsBot, Meta-WebIndexer, Amzn-User). Content behind those rules is unlikely to appear as a cited source on those surfaces.
That is below the 100% median across the 2188 sites in this dataset. Resolved from https://theaustralian.com.au/robots.txt on 12 November 2026.
Does theaustralian.com.au block each AI product?
| AI product | Position in robots.txt | Crawlers checked |
|---|---|---|
| ChatGPT | Allows | 3 |
| Claude | Blocks all | 3 |
| Perplexity | Blocks all | 2 |
| Gemini and Google AI | Blocks Google-Extended, Google-Agent and Google-GeminiNotebook; not GoogleOther and Google-CloudVertexBot | 5 |
| Microsoft Copilot | Allows | 1 |
| Apple Intelligence | Blocks all | 1 |
| Meta AI | Blocks all | 3 |
Has theaustralian.com.au changed its AI crawler rules?
We read this file every week. Between 17 August 2026 and 12 November 2026 we recorded no lasting change in which AI crawlers it blocks. See which sites did change their rules.
What theaustralian.com.au's robots.txt says about each AI crawler
A crawler named directly in its own group is governed by that group. One not named falls back to the wildcard group. Where neither exists the crawler is reported as not listed — no policy has been expressed, and in practice most crawlers will proceed.
| User-agent | Operator | Purpose | Access | Decided by |
|---|---|---|---|---|
OAI-SearchBot | OpenAI | search indexing | Allowed | direct rule |
GPTBot | OpenAI | model training | Allowed | direct rule |
ChatGPT-User | OpenAI | on-demand fetch | Allowed | direct rule |
Claude-SearchBot | Anthropic | search indexing | Blocked | wildcard policy |
ClaudeBot | Anthropic | model training | Blocked | wildcard policy |
Claude-User | Anthropic | on-demand fetch | Blocked | wildcard policy |
PerplexityBot | Perplexity | search indexing | Blocked | wildcard policy |
Perplexity-User | Perplexity | on-demand fetch | Blocked | wildcard policy |
Google-Extended | Gemini training control | Blocked | direct rule | |
Bingbot | Microsoft | Copilot search index | Allowed | direct rule |
Applebot-Extended | Apple | Apple Intelligence control | Blocked | wildcard policy |
Meta-ExternalAgent | Meta | model training | Blocked | wildcard policy |
Amazonbot | Amazon | model training | Blocked | wildcard policy |
Bytespider | ByteDance | model training | Blocked | wildcard policy |
CCBot | Common Crawl | training dataset | Blocked | wildcard policy |
GoogleOther | research and development fetch | Allowed | direct rule | |
Google-CloudVertexBot | Vertex AI on-demand fetch | Allowed | direct rule | |
Meta-ExternalFetcher | Meta | on-demand fetch | Blocked | wildcard policy |
MistralAI-User | Mistral AI | on-demand fetch | Blocked | wildcard policy |
cohere-ai | Cohere | model training | Blocked | wildcard policy |
DuckAssistBot | DuckDuckGo | on-demand fetch | Blocked | wildcard policy |
YouBot | You.com | search indexing | Blocked | wildcard policy |
Timpibot | Timpi | training dataset | Blocked | wildcard policy |
AI2Bot | Allen Institute for AI | model training | Blocked | wildcard policy |
Diffbot | Diffbot | structured-data extraction | Blocked | wildcard policy |
ImagesiftBot | ImageSift | training dataset | Blocked | wildcard policy |
PetalBot | Huawei | search indexing | Blocked | wildcard policy |
Omgilibot | Webz.io | training dataset | Blocked | wildcard policy |
ElevenlabsCrawler | ElevenLabs | model training | Blocked | wildcard policy |
ElevenlabsBot | ElevenLabs | on-demand fetch | Blocked | wildcard policy |
Meta-WebIndexer | Meta | Meta AI search index | Blocked | wildcard policy |
Amzn-User | Amazon | on-demand fetch | Blocked | wildcard policy |
Google-Agent | agents acting for users | Blocked | wildcard policy | |
Google-GeminiNotebook | user-added sources | Blocked | wildcard policy | |
Webzio | Webz.io | web data for AI | Blocked | wildcard policy |
Webzio-Extended | Webz.io | AI training tagging | Blocked | wildcard policy |
Does theaustralian.com.au allow AI training?
For each AI company with both kinds of crawler: whether this robots.txt lets its training crawlers in, and whether it lets in the crawlers behind its answers.
| AI company | Training | Answers |
|---|---|---|
| OpenAI | Allowed GPTBot | Allowed OAI-SearchBot, ChatGPT-User |
| Anthropic | Refused ClaudeBot | Refused Claude-SearchBot, Claude-User |
| Meta | Refused Meta-ExternalAgent | Refused Meta-ExternalFetcher, Meta-WebIndexer |
| Amazon | Refused Amazonbot | Refused Amzn-User |
| ElevenLabs | Refused ElevenlabsCrawler | Refused ElevenlabsBot |
Google is not in this table: its AI answers read what Googlebot may read, and its training control is Google-Extended, in the full list.
The shared AI-training signal
On 12 November 2026 this file had no group for __usage_ai-training, so it expresses no preference with the shared signal. A rule under User-agent: * is not read as one, and each company's own crawler rules above keep their meaning.
Google announced this signal on 30 October 2026 and plans to support it with other AI companies in 2027. It is a draft standard and no company has confirmed it reads it yet, so this is what the file says, not what happens to the content.
What this does and does not tell you
It tells you what theaustralian.com.au states in its robots.txt, which is the file every well-behaved AI crawler consults before fetching. It is a stated preference and a voluntary protocol — not an access control, and not evidence about what any crawler actually did.
It also says nothing about whether the content is useful to an answer engine once fetched. Access is the precondition; extractable passages, visible dates, named authorship and sourced claims decide whether a reachable page is actually cited. Sign up for a free AI visibility report to see the original answers and which pages they cite.
Finally, robots.txt changes. This page reflects theaustralian.com.au as it was on 12 November 2026. To see the current position, inspect the domain’s live robots.txt file and its matching rules.
Check your own site
Run the free AI crawler checker on your own site for the same verdict on every crawler, and the robots.txt lines to change if one is blocked by mistake. The ChatGPT citation guide explains the distinction between training and search access.
Browse the full study of 2188 measured domains, or the news publishers category, to see how this compares.
Embed this result
A live badge for theaustralian.com.au, updated whenever this report refreshes. Paste it into a README or site footer:
<a href="https://www.indexhalo.com/ai-crawler-report/theaustralian.com.au/">
<img src="https://www.indexhalo.com/badge/ai-crawler-access/theaustralian.com.au.svg"
alt="AI crawler access for theaustralian.com.au: 17%" height="20">
</a>