AI crawler access:
ai companies
The AI industry's own websites, resolved against the industry's own crawlers. 28 domains measured; 2 block at least one of the fifteen tracked AI crawlers.
2of 28 ai companies block at least one AI crawler · median access 100%
Every domain in this category
Sorted by the share of tracked AI crawlers permitted, lowest first. Each report shows all fifteen verdicts and the robots.txt rule behind them.
| Domain | Access | Allowed | Blocked | Not listed |
|---|---|---|---|---|
| descript.com | 87% | 13 | 2 | 0 |
| together.ai | 93% | 14 | 1 | 0 |
| ai.meta.com | 100% | 15 | 0 | 0 |
| anthropic.com | 100% | 15 | 0 | 0 |
| character.ai | 100% | 15 | 0 | 0 |
| codeium.com | 100% | 15 | 0 | 0 |
| cohere.com | 100% | 15 | 0 | 0 |
| copy.ai | 100% | 15 | 0 | 0 |
| cursor.com | 100% | 15 | 0 | 0 |
| deepmind.google | 100% | 15 | 0 | 0 |
| elevenlabs.io | 100% | 15 | 0 | 0 |
| fireflies.ai | 100% | 15 | 0 | 0 |
| gamma.app | 100% | 15 | 0 | 0 |
| groq.com | 100% | 15 | 0 | 0 |
| huggingface.co | 100% | 15 | 0 | 0 |
| jasper.ai | 100% | 15 | 0 | 0 |
| mistral.ai | 100% | 15 | 0 | 0 |
| openai.com | 100% | 15 | 0 | 0 |
| otter.ai | 100% | 15 | 0 | 0 |
| perplexity.ai | 100% | 15 | 0 | 0 |
| replicate.com | 100% | 15 | 0 | 0 |
| runwayml.com | 100% | 15 | 0 | 0 |
| sourcegraph.com | 100% | 15 | 0 | 0 |
| stability.ai | 100% | 15 | 0 | 0 |
| synthesia.io | 100% | 15 | 0 | 0 |
| tabnine.com | 100% | 15 | 0 | 0 |
| writesonic.com | 100% | 15 | 0 | 0 |
| x.ai | 100% | 15 | 0 | 0 |
Method and caveats are described on the full study page: one robots.txt fetch per domain, resolved against fifteen published AI user-agents, reported as the voluntary policy it is. Check any site yourself with the free crawler access checker.