What is Applebot-Extended? Apple's AI crawler
Applebot-Extended is a robots.txt control rather than a crawler of its own: it governs how content crawled by Applebot may be used to train Apple's foundation models.
81 of the 389 major websites we measure block Applebot-Extended in robots.txt (21%), making it number 4 of the 30 AI crawlers we track for how often it is blocked. 81 of those sites name it directly in their robots.txt; the rest are covered by a wildcard rule.
Key facts
| Fact | Detail |
|---|---|
| Operator | Apple |
| Used for | Apple Intelligence |
| Type | Control token. Not a crawler. A robots.txt rule that governs how already crawled content may be used. |
| robots.txt token | Applebot-Extended |
| Operator's documentation | Read it here |
What blocking Applebot-Extended does
Apple's documentation treats it as separate from Applebot, which can still crawl the site for Apple's search features when Applebot-Extended is disallowed.
How to allow or block Applebot-Extended
Add one of these groups to the robots.txt file at the root of your domain. It is a stated preference that well-behaved crawlers read, not an enforcement mechanism.
User-agent: Applebot-Extended Disallow: /
User-agent: Applebot-Extended Allow: /
Applebot-Extended user agent string
Applebot-Extended has no user agent string of its own, because it doesn't crawl. It only appears in robots.txt, so you won't find it in your server logs.
Which sites recently changed their rules for Applebot-Extended?
| Date first seen | Site | Change |
|---|---|---|
| 3 November 2026 | researchgate.net | Started blocking |
| 19 October 2026 | androidcentral.com | Started blocking |
| 19 October 2026 | techradar.com | Started blocking |
| 19 October 2026 | tomshardware.com | Started blocking |
| 28 September 2026 | airbnb.com | Started blocking |
| 31 August 2026 | fastcompany.com | Started blocking |
| 31 August 2026 | inc.com | Started blocking |
See the full change log.
Who blocks Applebot-Extended, by sector
| Sector | Sites measured | Blocking | Share |
|---|---|---|---|
| Technology media | 25 | 18 | 72% |
| News publishers | 44 | 30 | 68% |
| Social and media platforms | 28 | 9 | 32% |
| Health publishers | 16 | 4 | 25% |
| Finance and business media | 29 | 7 | 24% |
| Reference and research | 25 | 6 | 24% |
| Travel | 19 | 4 | 21% |
| E-commerce and retail | 19 | 2 | 11% |
| SaaS platforms | 54 | 1 | 2% |
| AI companies | 28 | 0 | 0% |
| Developer references | 34 | 0 | 0% |
| Government and institutions | 19 | 0 | 0% |
| Hosting and infrastructure | 23 | 0 | 0% |
| SEO and marketing tools | 26 | 0 | 0% |
Sites blocking Applebot-Extended
academia.edu, airbnb.com, androidcentral.com, apnews.com, arstechnica.com, asahi.com, barrons.com, bbc.co.uk, behance.net, bleepingcomputer.com, bloomberg.com, buzzfeednews.com, canva.com, cnbc.com, cnn.com, cntraveler.com, corriere.it, dictionary.com, drugs.com, dw.com, ebay.com, economist.com, fastcompany.com, flickr.com, forbes.com, ft.com, hackernoon.com, healthline.com, howtogeek.com, huffpost.com, imdb.com, inc.com, instacart.com, investopedia.com, lemonde.fr, lifehacker.com, makeuseof.com, marketwatch.com, mashable.com, medicalnewstoday.com, medium.com, nature.com, nbcnews.com, netflix.com, newsweek.com, newyorker.com, nytimes.com, pcmag.com, pexels.com, pinterest.com, quora.com, reddit.com, researchgate.net, reuters.com, sciencedirect.com, sciencemag.org, slate.com, smh.com.au, spiegel.de, techcrunch.com, and 21 more in the full dataset.
Sites that allow Applebot-Extended by name
amplitude.com, delta.com, disneyplus.com, frase.io, hulu.com, jasper.ai, render.com, soundcloud.com, squarespace.com, stability.ai.
See all 30 AI crawlers, the rule change log or the full study. Being reachable is only the first step: how AI search engines choose what to cite covers the rest.
Cite this data
You are welcome to quote or republish these figures. Please credit IndexHalo with a link to this page (CC BY 4.0). All our datasets are on the data page.
IndexHalo. "Applebot-Extended block rate across major websites." Measured 3 November 2026. https://www.indexhalo.com/ai-crawlers/applebot-extended/
<a href="https://www.indexhalo.com/ai-crawlers/applebot-extended/">Applebot-Extended block rate across major websites, IndexHalo</a>