The simplest robots.txt
Most files come down to a few lines. Each group names a crawler and lists what it may not request, and a catch-all group marked with * covers every crawler without its own group.
Google supports four fields: user-agent, allow, disallow and sitemap. It ignores others, including crawl-delay.
Four patterns from big websites
We resolve 30 AI crawlers against each site's live file. The sites fall into three groups, and a fourth pattern cuts across them. The rules below illustrate each pattern. They aren't copied from these sites, whose own reports list every verdict.
1. Leave the door open (258 sites)
These sites block none of the AI crawlers we track, because they have no rule that applies or a rule that allows them. Examples include wikipedia.org, youtube.com and shopify.com.
2. Name the AI crawlers one by one (114 sites)
This is the most common way to block. The site stays open to crawlers in general and lists the ones it declines. Examples include nytimes.com, bbc.co.uk and cnn.com.
3. Block by default (17 sites)
The catch-all group refuses everything, and only crawlers given their own group get in. It is the strictest pattern, and any new AI crawler is blocked from the day it appears. Examples include reddit.com, pinterest.com and imdb.com.
4. Decline training, allow search (49 sites)
Some operators run separate crawlers for model training and for search. These sites block OpenAI's GPTBot without blocking its OAI-SearchBot, so they can still be fetched for search results. All of them also belong to group 2 or 3. Examples include fool.com, forbes.com and techcrunch.com.
Which AI crawlers get blocked most
Counted across the 389 sites in the latest reading. See the GPTBot and OAI-SearchBot pages for how OpenAI's two crawlers differ. The full list and how it changes are in the AI crawler study and the change log.
What ten big websites do
"Blocked by robots.txt": what it means and what to do
Search Console uses this label when a rule stops Googlebot from requesting a URL. Google can't read the page, so it can't judge the content. Start by opening /robots.txt on that host and finding the rule that matches the path.
- If the block is a mistake, remove or narrow the rule, then check the URL again with URL Inspection in Search Console.
- If you want the page out of search, robots.txt is the wrong tool. Use noindex and let Google crawl the page, because a blocked crawler never sees the noindex rule.
- If you see "Indexed, though blocked by robots.txt", Google found links to the page and indexed the URL without crawling it. Decide which of the two cases above you are in.
Five rules that catch people out
- A crawler follows only one group. Google uses the most specific group that matches the crawler and ignores the rest. A crawler with its own group doesn't also follow your
User-agent: * rules.
- The longest path wins, not the first line. For
/page, Allow: /p beats Disallow: / because it is more specific. When rules conflict at the same length, Google uses the least restrictive one.
- Each host needs its own file. A file on
www.example.com doesn't cover blog.example.com, or a different protocol or port.
- It isn't private. Anyone can read the file, so listing a secret path advertises it. RFC 9309 says the protocol is no substitute for real security.
- There's a size limit. Google reads the first 500 KiB and ignores anything after it.
What this data doesn't show
These are well-known sites we chose to track, not a random sample of the web. We read what each file says about the site's home page, so a site may treat other paths differently. A file also doesn't show whether a crawler obeyed it, or why a site made its choice. A firewall can refuse a crawler the file allows, which the AI crawler directory covers.
