IndexHalo
DashboardGet my free AI report
MEASURED STUDY · 11 NOVEMBER 2026

Which reference and research
block AI crawlers?

Encyclopaedias, journals and universities — the citation bedrock of the web. 115 domains measured; 37 block at least one of the 30 tracked AI crawlers.

37of 115 reference and research block at least one AI crawler · median access 100%

The AI crawlers these sites block most

CCBot is blocked most often here, by 23 of 115 sites. What blocking each one means is in the AI crawler directory.

User-agentOperatorPurposeSites blocking
CCBotCommon Crawltraining dataset23 of 115
GPTBotOpenAImodel training23 of 115
ChatGPT-UserOpenAIon-demand fetch19 of 115
Applebot-ExtendedAppleApple Intelligence control18 of 115
Meta-ExternalAgentMetamodel training18 of 115
BytespiderByteDancemodel training15 of 115
AmazonbotAmazonmodel training14 of 115
ClaudeBotAnthropicmodel training14 of 115

Every domain in this category

Sorted by the share of tracked AI crawlers permitted, lowest first. Each report shows all 30 verdicts and the robots.txt rule behind them.

DomainAccessAllowedBlockedNot listed
linguee.com3%1290
wordreference.com7%2280
newscientist.com10%3270
sciencedirect.com13%4260
wikihow.com27%8220
academia.edu30%9210
nature.com47%14160
cell.com50%15150
genius.com50%15150
snopes.com53%16140
scientificamerican.com67%20100
weather.com70%2190
biorxiv.org73%2280
sagepub.com73%2280
livescience.com77%2370
fandom.com80%2460
science.org83%2550
sciencemag.org83%2550
timeanddate.com83%2550
etymonline.com87%2640
worldcat.org87%2640
dictionary.com90%2730
oxfordlearnersdictionaries.com90%2730
ssrn.com90%2730
thesaurus.com90%2730
vocabulary.com90%2730
wolframalpha.com90%2730
researchgate.net93%2820
cambridge.org97%2910
coursera.org97%2910
discogs.com97%2910
jstor.org97%2910
librarything.com97%2910
pnas.org97%2910
scholar.google.com97%2910
tandfonline.com97%2910
udacity.com97%2910
acm.org100%3000
allmusic.com100%0030
anu.edu.au100%3000
archive.org100%3000
arxiv.org100%3000
berkeley.edu100%3000
bl.uk100%3000
bls.gov100%0030
britannica.com100%3000
caltech.edu100%3000
cam.ac.uk100%3000
census.gov100%3000
cliffsnotes.com100%3000
cmu.edu100%3000
collinsdictionary.com100%3000
cornell.edu100%3000
data.gov100%3000
deepl.com100%3000
desmos.com100%3000
dictionary.cambridge.org100%3000
edx.org100%3000
europeana.eu100%3000
factcheck.org100%3000
frontiersin.org100%3000
futurelearn.com100%3000
geogebra.org100%3000
gutenberg.org100%3000
history.com100%0030
howstuffworks.com100%3000
imperial.ac.uk100%3000
khanacademy.org100%0030
litcharts.com100%3000
loc.gov100%3000
mathworld.wolfram.com100%3000
mcgill.ca100%3000
medrxiv.org100%3000
merriam-webster.com100%3000
metoffice.gov.uk100%3000
mit.edu100%3000
musicbrainz.org100%3000
nationalgeographic.com100%3000
noaa.gov100%3000
nus.edu.sg100%0030
ocw.mit.edu100%3000
open.edu100%3000
openlibrary.org100%3000
ourworldindata.org100%0030
ox.ac.uk100%3000
plos.org100%3000
politifact.com100%3000
princeton.edu100%3000
pubmed.ncbi.nlm.nih.gov100%3000
shmoop.com100%3000
smithsonianmag.com100%3000
springer.com100%3000
stanford.edu100%3000
statista.com100%3000
sydney.edu.au100%3000
symbolab.com100%3000
tcd.ie100%3000
ubc.ca100%3000
uchicago.edu100%3000
ucl.ac.uk100%3000
udemy.com100%3000
unimelb.edu.au100%3000
upenn.edu100%3000
urbandictionary.com100%0030
usgs.gov100%3000
utoronto.ca100%3000
wikidata.org100%3000
wikipedia.org100%3000
wikiquote.org100%3000
wikisource.org100%3000
wikivoyage.org100%3000
wiktionary.org100%3000
worldometers.info100%3000
yale.edu100%3000
zenodo.org100%3000

Method and caveats are described on the full study page: one robots.txt fetch per domain, resolved against 30 published AI user-agents, reported as the voluntary policy it is.

Who started blocking which crawler this week, measured from live robots.txt. Unsubscribe in one click.