Semrush Enterprise Intelligence

What does Semrush Enterprise Intelligence do?

Semrush Enterprise Intelligence (SemrushBot-ESI) crawls web pages and links to populate Semrush's enterprise-level site intelligence tools. The collected data feeds products like Backlink Analytics, Site Audit, Backlink Audit, Link Building, On-Page SEO Checker, Content Toolkit, and Enterprise Site Intelligence. Semrush products present clickable links to source URLs in their reports, which can drive indirect referral traffic when other SEO professionals discover your site through Semrush data.

Should I allow and optimize for Semrush Enterprise Intelligence to drive organic growth?

Allowing Semrush crawlers ensures your site's backlink profile, technical SEO data, and content appear accurately in Semrush's tools. SEO professionals, marketers, and competitors use these tools daily. When your pages show up in Backlink Analytics or Site Audit reports, they include clickable links back to your URLs. This won't drive consumer traffic directly, but it keeps your site visible in the SEO ecosystem and ensures accurate representation in competitive analysis. Blocking the crawler means your data may be stale or incomplete in Semrush reports, which could affect how others perceive your site's authority.

Data Usage & Training

Crawled content is indexed to power Semrush's SEO product suite. Semrush's public documentation does not clearly state whether crawled page content is also used to train general-purpose AI models. If this concerns you, contact [email protected] for clarification.

How Semrush Enterprise Intelligence Accesses Content

Here's how Semrush Enterprise Intelligence accesses your site and understands your content:

• Fetches HTML and renders JavaScript fully
• Follows links to discover new pages across your site
• Respects robots.txt Disallow, Allow, Sitemap, and Crawl-delay directives
• Honors meta robots directives (noindex, nofollow)
• Treats robots.txt returning 4xx or 5xx as missing (assumes full access)
• Adapts crawl frequency based on server load and robots.txt settings

Recurring scheduled crawls from a crawl frontier. Site Audit crawls run on per-project schedules, while broader index crawls are continuous and periodic. The bot adapts frequency to server load and Crawl-delay directives.

How to Block or Control Semrush Enterprise Intelligence

To block via robots.txt, use the product-specific token: User-agent: SemrushBot-ESI Disallow: / To block all Semrush crawlers at once: User-agent: SemrushBot Disallow: / Other product-specific tokens include SiteAuditBot, SemrushBot-SI, SemrushBot-BA, SemrushBot-SWA, SemrushBot-FT, and SplitSignalBot. IP-based blocking is unreliable because Semrush crawlers do not use consecutive IP ranges. The one exception is SiteAuditBot, which uses the subnet 85.208.98.128/25. You can also use meta robots tags (noindex, nofollow) for page-level control. For additional help, contact [email protected].

Common Issues & Troubleshooting

Watch out for these common problems when working with Semrush Enterprise Intelligence:

• IP-based blocking is unreliable because Semrush crawlers don't use consecutive or published IP ranges
• A robots.txt returning 4xx or 5xx is treated as missing, giving the bot full access to your site
• CDNs, WAFs, and bot protection services (like Cloudflare) may block crawler requests unexpectedly
• Misconfigured robots.txt rules using the wrong user-agent token will have no effect on the intended crawler
• Multiple Semrush bot variants (SemrushBot, SemrushBot-ESI, SiteAuditBot, etc.) require separate robots.txt rules if you want granular control