What is Google-Extended? Google's AI crawler
Google-Extended is not a separate crawler. It is a robots.txt token publishers use to manage how content Google has already crawled may be used for Gemini and Vertex AI, and Google crawls with its existing user agents.
63 of the 389 major websites we measure block Google-Extended in robots.txt (16%), making it number 10 of the 30 AI crawlers we track for how often it is blocked. 82 of those sites name it directly in their robots.txt; the rest are covered by a wildcard rule.
Key facts
| Fact | Detail |
|---|---|
| Operator | |
| Used for | Gemini |
| Type | Control token. Not a crawler. A robots.txt rule that governs how already crawled content may be used. |
| robots.txt token | Google-Extended |
| Operator's documentation | Read it here |
What blocking Google-Extended does
Google says Google-Extended does not affect a site's inclusion in Google Search and is not a Google Search ranking signal.
How to allow or block Google-Extended
Add one of these groups to the robots.txt file at the root of your domain. It is a stated preference that well-behaved crawlers read, not an enforcement mechanism.
User-agent: Google-Extended Disallow: /
User-agent: Google-Extended Allow: /
Google-Extended user agent string
Google-Extended has no user agent string of its own, because it doesn't crawl. It only appears in robots.txt, so you won't find it in your server logs.
Which sites recently changed their rules for Google-Extended?
| Date first seen | Site | Change |
|---|---|---|
| 26 October 2026 | nerdwallet.com | Stopped blocking |
| 19 October 2026 | aliexpress.com | Started blocking |
| 19 October 2026 | cnn.com | Stopped blocking |
| 19 October 2026 | howtogeek.com | Started blocking |
| 19 October 2026 | makeuseof.com | Started blocking |
| 19 October 2026 | xda-developers.com | Started blocking |
| 12 October 2026 | pexels.com | Stopped blocking |
| 7 September 2026 | straitstimes.com | Started blocking |
See the full change log.
Who blocks Google-Extended, by sector
| Sector | Sites measured | Blocking | Share |
|---|---|---|---|
| News publishers | 44 | 27 | 61% |
| Technology media | 25 | 9 | 36% |
| Reference and research | 25 | 6 | 24% |
| E-commerce and retail | 19 | 4 | 21% |
| Social and media platforms | 28 | 5 | 18% |
| Finance and business media | 29 | 5 | 17% |
| Travel | 19 | 3 | 16% |
| Health publishers | 16 | 2 | 13% |
| AI companies | 28 | 1 | 4% |
| SaaS platforms | 54 | 1 | 2% |
| Developer references | 34 | 0 | 0% |
| Government and institutions | 19 | 0 | 0% |
| Hosting and infrastructure | 23 | 0 | 0% |
| SEO and marketing tools | 26 | 0 | 0% |
Sites blocking Google-Extended
abcnews.go.com, academia.edu, alibaba.com, aliexpress.com, amazon.com, arstechnica.com, asahi.com, barrons.com, bbc.co.uk, behance.net, bloomberg.com, buzzfeednews.com, cnbc.com, cntraveler.com, corriere.it, dw.com, economist.com, figma.com, flickr.com, fortune.com, france24.com, ft.com, globeandmail.com, hackernoon.com, howtogeek.com, huffpost.com, imdb.com, instacart.com, investopedia.com, jstor.org, lemonde.fr, makeuseof.com, marketwatch.com, nature.com, nbcnews.com, newyorker.com, nytimes.com, pinterest.com, reddit.com, reuters.com, sciencedirect.com, sciencemag.org, slate.com, smh.com.au, ssrn.com, straitstimes.com, strava.com, techcrunch.com, theage.com.au, theatlantic.com, theregister.com, thetimes.co.uk, theverge.com, together.ai, travelandleisure.com, tripadvisor.com, usatoday.com, verywellhealth.com, vox.com, wired.com, and 3 more in the full dataset.
Sites that allow Google-Extended by name
amplitude.com, axios.com, bestbuy.com, cloudflare.com, css-tricks.com, deel.com, delta.com, disneyplus.com, fastcompany.com, frase.io, hulu.com, inc.com, jasper.ai, monday.com, netflix.com, netlify.com, quora.com, redis.io, render.com, rottentomatoes.com, seranking.com, sitepoint.com, soundcloud.com, squarespace.com, stability.ai, telegraph.co.uk, ulta.com, venturebeat.com, vimeo.com.
Other Google crawlers
GoogleOther (research and development fetch), Google-CloudVertexBot (Vertex AI on-demand fetch).
See all 30 AI crawlers, the rule change log or the full study. Being reachable is only the first step: how AI search engines choose what to cite covers the rest.
Cite this data
You are welcome to quote or republish these figures. Please credit IndexHalo with a link to this page (CC BY 4.0). All our datasets are on the data page.
IndexHalo. "Google-Extended block rate across major websites." Measured 3 November 2026. https://www.indexhalo.com/ai-crawlers/google-extended/
<a href="https://www.indexhalo.com/ai-crawlers/google-extended/">Google-Extended block rate across major websites, IndexHalo</a>