What is SemrushBot-FT? Semrush's SEO crawler
SemrushBot-FT is the Semrush bot behind Plagiarism Checker and similar tools.
40 of the 1662 major websites we measure block SemrushBot-FT in robots.txt (2%). 4 of those sites name it directly in their robots.txt; the rest are covered by a wildcard rule.
Key facts
| Fact | Detail |
|---|---|
| Operator | Semrush |
| Used for | Semrush Plagiarism Checker |
| Type | SEO tool. Collects web data for an SEO tool. Listed here because people look it up beside AI crawlers. It isn't counted in the AI crawler figures. |
| robots.txt token | SemrushBot-FT |
| Operator's documentation | Read it here |
What blocking SemrushBot-FT does
Semrush says each of its tool bots can be blocked on its own with a robots.txt group that names it, separately from SemrushBot.
How to allow or block SemrushBot-FT
Add one of these groups to the robots.txt file at the root of your domain. It is a stated preference that well-behaved crawlers read, not an enforcement mechanism.
User-agent: SemrushBot-FT Disallow: /
User-agent: SemrushBot-FT Allow: /
SemrushBot-FT user agent string
Semrush's page doesn't list a full string, so look for SemrushBot-FT in the user agent field of your logs.
How to verify a request is really SemrushBot-FT
Anyone can put a crawler's name in a user agent string, so the name alone proves nothing. We did not find a published IP list or reverse DNS rule from Semrush.
How to whitelist SemrushBot-FT in a firewall or CDN
robots.txt only states a preference. A firewall, CDN or bot protection service can still refuse the request, and the crawler then gets an error even though robots.txt allows it.
- Check that no group in your robots.txt blocks it, using the rule above.
- Add an allow rule in your firewall or CDN that matches the name in the user agent. Keep the rule narrow, because anyone can copy a name.
- Check your server logs for a 200 response from SemrushBot-FT afterwards.
On Cloudflare, also check AI Crawler and Agent access policies, which can block search, agent and training traffic across a site.
Why SemrushBot-FT gets a 403 or 405 error
A 403 means the server refused the request, which usually points to a firewall or bot rule. A 405 means the server refused the request's method, so check any rule that only permits certain methods. A crawler that obeys robots.txt won't request a page it is told to avoid, so an error in your logs points to your server's configuration. To stop it deliberately at the server, this nginx rule returns a 403:
if ($http_user_agent ~* "SemrushBot-FT") {
return 403;
}Who blocks SemrushBot-FT, by sector
| Sector | Sites measured | Blocking | Share |
|---|---|---|---|
| Social and media platforms | 115 | 11 | 10% |
| News publishers | 182 | 9 | 5% |
| Reference and research | 114 | 4 | 4% |
| AI companies | 128 | 4 | 3% |
| Finance and business media | 94 | 3 | 3% |
| Technology media | 126 | 4 | 3% |
| E-commerce and retail | 92 | 1 | 1% |
| Government and institutions | 98 | 1 | 1% |
| Hosting and infrastructure | 98 | 1 | 1% |
| SaaS platforms | 196 | 2 | 1% |
| Developer references | 159 | 0 | 0% |
| Health publishers | 88 | 0 | 0% |
| SEO and marketing tools | 86 | 0 | 0% |
| Travel | 86 | 0 | 0% |
Sites blocking SemrushBot-FT
academia.edu, afterpay.com, barrons.com, consilium.europa.eu, devclass.com, disneyplus.com, distill.pub, eluniversal.com.mx, facebook.com, flickr.com, fortinet.com, golem.de, hackernoon.com, imdb.com, instacart.com, instagram.com, linguee.com, lobste.rs, magento.com, marketwatch.com, meta.ai, netflix.com, news.com.au, pinterest.com, poe.com, reddit.com, reuters.com, sap.com, sciencedirect.com, semafor.com, theregister.com, theresanaiforthat.com, thesun.co.uk, thetimes.co.uk, threads.net, todayonline.com, wordreference.com, wsj.com, x.com, zeit.de.
Sites that allow SemrushBot-FT by name
Other Semrush crawlers
SemrushBot (SEO index), SiteAuditBot (SEO site audit), SemrushBot-BA (SEO backlink audit), SemrushBot-SI (SEO page checks), SemrushBot-SWA (SEO writing checks), SplitSignalBot (SEO A/B tests), SemrushBot-OCOB (SEO content tools), RyteBot (SEO site audit), SemrushBot-ESI (enterprise site data).
See all 36 AI crawlers, the rule change log or the full study. Being reachable is only the first step: how AI search engines choose what to cite covers the rest.
Cite this data
You are welcome to quote or republish these figures. Please credit IndexHalo with a link to this page (CC BY 4.0). All our datasets are on the data page.
IndexHalo. "SemrushBot-FT block rate across major websites." Measured 12 November 2026. https://www.indexhalo.com/ai-crawlers/semrushbot-ft/
<a href="https://www.indexhalo.com/ai-crawlers/semrushbot-ft/">SemrushBot-FT block rate across major websites, IndexHalo</a>