Semrush Content Toolkit

What does Semrush Content Toolkit do?

SemrushBot-OCOB crawls web pages to collect URLs, hyperlinks, and page content that feed Semrush's Content Toolkit. The Content Toolkit uses this data alongside AI-powered insights to help Semrush users generate and optimize content for SEO. This bot does not drive referral traffic back to your site.

Should I allow and optimize for Semrush Content Toolkit to drive organic growth?

SemrushBot-OCOB feeds data into Semrush's Content Toolkit, which is an SEO tool used by marketers, not a consumer-facing search or AI assistant. It does not generate referral traffic or citations back to your site. Allowing it has no direct impact on your visibility to end users. The primary reason to allow this bot is if you want your site's data accurately represented within Semrush's SEO tools, which competitors and partners may use to evaluate your site.

Data Usage & Training

Whether content crawled by SemrushBot-OCOB is used to train AI models is unclear. Semrush states its Content Toolkit combines "trusted Semrush data" and "AI-powered insights" to generate content, but there is no explicit public statement confirming or denying that crawled content trains underlying models. Contact Semrush directly or review their data policy for clarification.

How Semrush Content Toolkit Accesses Content

Here's how Semrush Content Toolkit accesses your site and understands your content:

• Fetches HTML via standard HTTP requests using the user-agent string Mozilla/5.0 (compatible; SemrushBot-OCOB/1; +https://www.semrush.com/bot/)
• Collects page URLs, hyperlinks, and page content
• Respects robots.txt Disallow, Allow, Crawl-delay, and Sitemap directives
• Requires a top-level robots.txt returning HTTP 200 to recognize rules
• Treats 4xx robots.txt responses as "no robots.txt" (crawls freely) and 5xx as "do not crawl"

SemrushBot-OCOB uses an adaptive, continuous crawl pattern. It adjusts frequency based on server load and regularly revisits sites to detect updates. Changes to your robots.txt may take up to one hour or approximately 100 requests to be detected.

How to Block or Control Semrush Content Toolkit

To block SemrushBot-OCOB, add the following to your robots.txt: User-agent: SemrushBot-OCOB Disallow: / Your robots.txt must be at the top level of your domain and return HTTP 200 for rules to take effect. Semrush does not publish consecutive IP blocks, so IP-based blocking is unreliable and not recommended by Semrush. If you need additional help, contact [email protected].

Common Issues & Troubleshooting

Watch out for these common problems when working with Semrush Content Toolkit:

• A robots.txt returning 4xx is treated as absent, meaning the bot will crawl freely
• A robots.txt returning 5xx causes the bot to stop crawling entirely, which may not be your intent
• Robots.txt not placed at the top level of the domain will be ignored
• After a site migration, robots.txt rules may not carry over, leaving the bot unblocked
• IP-based blocking is unreliable because Semrush does not use consecutive IP blocks
• Robots.txt changes can take up to one hour or ~100 requests to be detected