SemrushSiteAudit

What does SemrushSiteAudit do?

SemrushSiteAudit crawls websites to perform technical SEO audits, checking for crawlability issues, broken links, JavaScript/CSS problems, and site health. The results feed into Semrush's Site Audit tool within their SEO Toolkit. It does not directly drive referral traffic to your site, but site owners or competitors using Semrush may discover your pages through audit data.

Should I allow and optimize for SemrushSiteAudit to drive organic growth?

SemrushSiteAudit doesn't generate direct referral traffic or citations. However, Semrush is one of the most widely used SEO platforms, and your site's data appears in competitor research, backlink analysis, and domain overview reports. Allowing the crawler ensures accurate representation of your site in Semrush's database, which SEO professionals and marketers rely on for decision-making. Blocking it could mean incomplete or outdated data about your site in a tool used by millions of marketers.

Data Usage & Training

Crawled data populates Semrush products like Site Audit and related analytics tools. There is no explicit public statement confirming or denying whether crawled content is used to train large language models, so training usage remains unclear from available documentation.

How SemrushSiteAudit Accesses Content

Here's how SemrushSiteAudit accesses your site and understands your content:

• Fetches HTML via standard HTTP requests using the user-agent string Mozilla/5.0 (compatible; SiteAuditBot/0.97; +http://www.semrush.com/bot.html)
• Supports partial JavaScript rendering when enabled in Semrush project settings
• Respects robots.txt Disallow, Allow, Sitemap, and Crawl-delay directives
• Honors meta robots tags (noindex, nofollow)
• Requires robots.txt at the site root returning HTTP 200 to parse rules; a missing or non-200 robots.txt is treated as no restrictions

Crawl frequency depends on whether audits are user-initiated or scheduled within a Semrush project. Semrush adjusts crawl rate based on server load and project settings. Broader background data collection may also occur outside of specific audit runs.

How to Block or Control SemrushSiteAudit

To block SemrushSiteAudit via robots.txt: User-agent: SiteAuditBot Disallow: / You can also use meta robots tags (noindex, nofollow) on specific pages. IP-based blocking is not recommended because Semrush uses many IP addresses that change over time. See https://www.semrush.com/kb/681-site-audit-troubleshooting for known IPs and troubleshooting guidance. For additional help, contact [email protected]. If you run your own Semrush projects, you can adjust crawl settings, change the user-agent, or use Web Bot Auth / file verification to control access.

Common Issues & Troubleshooting

Watch out for these common problems when working with SemrushSiteAudit:

• CDN or firewall rules (e.g., Cloudflare) block SiteAuditBot before it reaches your server
• Missing or non-200 robots.txt causes Semrush to treat your site as having no crawl restrictions
• IP-based blocking is unreliable because Semrush rotates across many addresses
• JavaScript-heavy pages return incomplete data when JS rendering is disabled in the Semrush project
• Large pages or complex site structures can cause crawl timeouts or incomplete audits