What is SemrushBot-SI? Semrush's SEO crawler
SemrushBot-SI is the Semrush bot behind On Page SEO Checker and similar tools.
42 of the 1662 major websites we measure block SemrushBot-SI in robots.txt (3%). 11 of those sites name it directly in their robots.txt; the rest are covered by a wildcard rule.
Key facts
| Fact | Detail |
|---|---|
| Operator | Semrush |
| Used for | Semrush On Page SEO Checker |
| Type | SEO tool. Collects web data for an SEO tool. Listed here because people look it up beside AI crawlers. It isn't counted in the AI crawler figures. |
| robots.txt token | SemrushBot-SI |
| Operator's documentation | Read it here |
What blocking SemrushBot-SI does
Semrush says each of its tool bots can be blocked on its own with a robots.txt group that names it, separately from SemrushBot.
How to allow or block SemrushBot-SI
Add one of these groups to the robots.txt file at the root of your domain. It is a stated preference that well-behaved crawlers read, not an enforcement mechanism.
User-agent: SemrushBot-SI Disallow: /
User-agent: SemrushBot-SI Allow: /
SemrushBot-SI user agent string
Semrush's page doesn't list a full string, so look for SemrushBot-SI in the user agent field of your logs.
How to verify a request is really SemrushBot-SI
Anyone can put a crawler's name in a user agent string, so the name alone proves nothing. We did not find a published IP list or reverse DNS rule from Semrush.
How to whitelist SemrushBot-SI in a firewall or CDN
robots.txt only states a preference. A firewall, CDN or bot protection service can still refuse the request, and the crawler then gets an error even though robots.txt allows it.
- Check that no group in your robots.txt blocks it, using the rule above.
- Add an allow rule in your firewall or CDN that matches the name in the user agent. Keep the rule narrow, because anyone can copy a name.
- Check your server logs for a 200 response from SemrushBot-SI afterwards.
On Cloudflare, also check AI Crawler and Agent access policies, which can block search, agent and training traffic across a site.
Why SemrushBot-SI gets a 403 or 405 error
A 403 means the server refused the request, which usually points to a firewall or bot rule. A 405 means the server refused the request's method, so check any rule that only permits certain methods. A crawler that obeys robots.txt won't request a page it is told to avoid, so an error in your logs points to your server's configuration. To stop it deliberately at the server, this nginx rule returns a 403:
if ($http_user_agent ~* "SemrushBot-SI") {
return 403;
}Who blocks SemrushBot-SI, by sector
| Sector | Sites measured | Blocking | Share |
|---|---|---|---|
| Social and media platforms | 115 | 10 | 9% |
| News publishers | 182 | 10 | 5% |
| Reference and research | 114 | 5 | 4% |
| AI companies | 128 | 4 | 3% |
| Finance and business media | 94 | 3 | 3% |
| Technology media | 126 | 4 | 3% |
| Government and institutions | 98 | 2 | 2% |
| E-commerce and retail | 92 | 1 | 1% |
| Hosting and infrastructure | 98 | 1 | 1% |
| SaaS platforms | 196 | 2 | 1% |
| Developer references | 159 | 0 | 0% |
| Health publishers | 88 | 0 | 0% |
| SEO and marketing tools | 86 | 0 | 0% |
| Travel | 86 | 0 | 0% |
Sites blocking SemrushBot-SI
academia.edu, afterpay.com, barrons.com, consilium.europa.eu, devclass.com, disneyplus.com, distill.pub, eluniversal.com.mx, etymonline.com, facebook.com, flickr.com, fortinet.com, golem.de, hackernoon.com, hrw.org, imdb.com, instacart.com, instagram.com, linguee.com, lobste.rs, magento.com, marketwatch.com, meta.ai, netflix.com, news.com.au, news24.com, poe.com, reddit.com, reuters.com, sap.com, sciencedirect.com, spiegel.de, theregister.com, theresanaiforthat.com, thesun.co.uk, thetimes.co.uk, threads.net, todayonline.com, wordreference.com, wsj.com, x.com, zeit.de.
Sites that allow SemrushBot-SI by name
bupa.co.uk, pinterest.com, semrush.com, slate.com, unsplash.com.
Other Semrush crawlers
SemrushBot (SEO index), SiteAuditBot (SEO site audit), SemrushBot-BA (SEO backlink audit), SemrushBot-SWA (SEO writing checks), SplitSignalBot (SEO A/B tests), SemrushBot-OCOB (SEO content tools), SemrushBot-FT (plagiarism checks), RyteBot (SEO site audit), SemrushBot-ESI (enterprise site data).
See all 36 AI crawlers, the rule change log or the full study. Being reachable is only the first step: how AI search engines choose what to cite covers the rest.
Cite this data
You are welcome to quote or republish these figures. Please credit IndexHalo with a link to this page (CC BY 4.0). All our datasets are on the data page.
IndexHalo. "SemrushBot-SI block rate across major websites." Measured 12 November 2026. https://www.indexhalo.com/ai-crawlers/semrushbot-si/
<a href="https://www.indexhalo.com/ai-crawlers/semrushbot-si/">SemrushBot-SI block rate across major websites, IndexHalo</a>