The AI crawler
directory

Three kinds of crawler, three different decisions

Most people searching for this list want to know whether blocking a bot will remove them from AI answers. It depends on the kind. Blocking a training crawler such as GPTBot is a different decision from blocking a search crawler such as OAI-SearchBot, and a single rule can do both by accident.

Search index crawlers

Decides whether a site can appear in that product's AI search answers.

On-demand fetch crawlers

Visits a page because someone asked a question. Several operators say robots.txt may not apply.

Model training crawlers

Collects content that may be used to train AI models. Blocking it does not usually remove a site from AI search.

Control token crawlers

Not a crawler. A robots.txt rule that governs how already crawled content may be used.

Other crawlers

Neither clearly search nor training.

SEO tool crawlers

Collects web data for an SEO tool. Listed here because people look it up beside AI crawlers. It isn't counted in the AI crawler figures.

Which sites changed their rules?

We read these files every week, so the AI crawler rule change log shows who started or stopped blocking which crawler, and when.

Why an AI crawler gets a 403 or 405 error

robots.txt only states a preference. A firewall, CDN or bot protection service can still refuse a crawler it allows. A 403 means the server refused the request, which usually points to a firewall or bot rule. A 405 means the server refused the request's method, so check any rule that only permits certain methods. Each crawler's page explains how to whitelist it, and how to verify a request is really that crawler.

Does a specific site block AI crawlers?

Each site below has its own report, with a verdict for every crawler and the robots.txt rule behind it. For the patterns these sites use, see our robots.txt examples. Browse by the full study or by sector: news publishers, technology media, developer references, saas platforms, ai companies, seo and marketing tools, hosting and infrastructure, reference and research, e-commerce and retail, finance and business media, government and institutions, health publishers, travel, social and media platforms.

500px.com, abcnews.go.com, academia.edu, acm.org, adobe.com, afar.com, ahrefs.com, ai.meta.com, airbnb.com, airtable.com, alibaba.com, aliexpress.com, aljazeera.com, amazon.com, amplitude.com, anandtech.com, androidcentral.com, angular.io, anthropic.com, apnews.com, archive.org, arstechnica.com, arxiv.org, asahi.com, asana.com, atlassian.com, australia.gov.au, aws.amazon.com, axios.com, azure.microsoft.com, backlinko.com, bankofamerica.com, bankrate.com, barclays.co.uk, barrons.com, basecamp.com, bbc.co.uk, behance.net, belastingdienst.nl, berkeley.edu, bestbuy.com, binance.com, bleepingcomputer.com, bloomberg.com, booking.com, botify.com, box.com, brex.com, brightedge.com, britannica.com, britishairways.com, bund.de, businessinsider.com, buzzfeednews.com, calendly.com, calm.com, cam.ac.uk, canada.ca, canva.com, cbsnews.com, cdc.gov, character.ai, chase.com, chewy.com, clearscope.io, clevelandclinic.org, clickup.com, close.com, cloud.google.com, cloudflare.com, cmu.edu, cnbc.com, cnn.com, cntraveler.com, codecademy.com, codeium.com, cohere.com, coinbase.com, conductor.com, contentkingapp.com, copy.ai, corriere.it, costco.com, crates.io, creditkarma.com, css-tricks.com, cursor.com, datadog.com, deel.com, deepmind.google, deliveroo.co.uk, delta.com, descript.com, dev.to, developer.mozilla.org, deviantart.com, dictionary.com, digitalocean.com, digitaltrends.com, disneyplus.com, dnsimple.com, docker.com, docusign.com, dribbble.com, drift.com, dropbox.com, drugs.com, dw.com, ebay.com, economist.com, elevenlabs.io, elpais.com, engadget.com, etsy.com, europa.eu, expedia.com, expensify.com, fastcompany.com, fastly.com, fda.gov, figma.com, fireflies.ai, fitbit.com, flickr.com, fly.io, fool.com, forbes.com, fortune.com, france24.com, frase.io, freecodecamp.org, freshworks.com, ft.com, gamma.app, gandi.net, geeksforgeeks.org, giphy.com, github.com, gitlab.com, gizmodo.com, globeandmail.com, golang.org, goodreads.com, goodrx.com, gov.uk, groq.com, gutenberg.org, hackernoon.com, hashnode.com, hbomax.com, headspace.com, healthline.com, hetzner.com, homedepot.com, hopkinsmedicine.org, hostelworld.com, hover.com, howtogeek.com, hsbc.com, hubspot.com, huffpost.com, huggingface.co, hulu.com, ikea.com, imdb.com, imf.org, inc.com, independent.co.uk, instacart.com, intercom.com, investopedia.com, irishtimes.com, irs.gov, jasper.ai, jstor.org, justeattakeaway.com, kayak.com, kinsta.com, klaviyo.com, kraken.com, kubernetes.io, latimes.com, leetcode.com, lemonde.fr, lifehacker.com, linear.app, linkedin.com, linode.com, lonelyplanet.com, loom.com, lufthansa.com, lumar.io, macrumors.com, mailchimp.com, majestic.com, makeuseof.com, marketmuse.com, marketwatch.com, mashable.com, medicalnewstoday.com, medium.com, merriam-webster.com, metacritic.com, miro.com, mistral.ai, mit.edu, mixpanel.com, monday.com, mongodb.com, monzo.com, morningstar.com, moz.com, myfitnesspal.com, mysql.com, n26.com, namecheap.com, nature.com, nbcnews.com, neilpatel.com, nerdwallet.com, netflix.com, netlify.app, netlify.com, newrelic.com, newsweek.com, newyorker.com, nhs.uk, nih.gov, nike.com, nodejs.org, notion.so, nytimes.com, oecd.org, omio.com, oncrawl.com, openai.com, otter.ai, ouraring.com, ovhcloud.com, packagist.org, pagerduty.com, pantheon.io, paypal.com, pcmag.com, perplexity.ai, pexels.com, php.net, pinterest.com, pipedrive.com, plos.org, pnas.org, porkbun.com, postgresql.org, producthunt.com, pubmed.ncbi.nlm.nih.gov, pypi.org, python.org, quickbooks.intuit.com, quora.com, railway.com, ramp.com, reactjs.org, reddit.com, redis.io, render.com, replicate.com, researchgate.net, reuters.com, revolut.com, rippling.com, rottentomatoes.com, roughguides.com, rubygems.org, runwayml.com, ryanair.com, sage.com, salesforce.com, santander.com, sciencedirect.com, sciencemag.org, scmp.com, screamingfrog.co.uk, searchenginejournal.com, searchengineland.com, sec.gov, segment.com, semrush.com, sendgrid.com, seranking.com, seroundtable.com, serpstat.com, service-public.fr, shopify.com, similarweb.com, sistrix.com, sitepoint.com, skatteverket.se, sky.com, skyscanner.net, slack.com, slashdot.org, slate.com, smashingmagazine.com, smh.com.au, soundcloud.com, sourcegraph.com, spiegel.de, spotify.com, springer.com, spyfu.com, sqlite.org, squarespace.com, ssrn.com, stability.ai, stanford.edu, starlingbank.com, straitstimes.com, strava.com, stripe.com, substack.com, surferseo.com, surveymonkey.com, svelte.dev, synthesia.io, tabnine.com, target.com, techcrunch.com, techradar.com, teladoc.com, telegraph.co.uk, theage.com.au, theatlantic.com, theguardian.com, thenextweb.com, theregister.com, thetimes.co.uk, theverge.com, time.com, together.ai, tomshardware.com, trainline.com, travelandleisure.com, trello.com, tripadvisor.com, tutorialspoint.com, twilio.com, twitch.tv, typeform.com, ulta.com, un.org, unsplash.com, usa.gov, usatoday.com, venturebeat.com, vercel.com, verywellhealth.com, vimeo.com, vox.com, vrbo.com, vuejs.org, vultr.com, w3schools.com, walmart.com, washingtonpost.com, wayfair.com, webflow.com, webmd.com, wellsfargo.com, who.int, wikipedia.org, wiktionary.org, wired.com, wise.com, wix.com, worldbank.org, wpengine.com, writesonic.com, wsj.com, x.ai, xda-developers.com, xero.com, yoast.com, youtube.com, zalando.com, zdnet.com, zeit.de, zendesk.com, zoho.com, zoom.us.

The figures are the crawler view of the same dataset as the AI crawler access study and the sector and provider index. robots.txt is a voluntary protocol, so these figures report stated policy, not crawler behaviour.

IndexHalo. "AI crawler block rates across major websites." Measured 3 November 2026. https://www.indexhalo.com/ai-crawlers/

<a href="https://www.indexhalo.com/ai-crawlers/">AI crawler block rates across major websites, IndexHalo</a>