AhrefsSiteAudit

What does AhrefsSiteAudit do?

AhrefsSiteAudit crawls and renders your website pages to identify technical and on-page SEO issues for the Ahrefs Site Audit tool. It checks for broken links, performance problems, indexability issues, and other SEO factors, then surfaces results in the Ahrefs dashboard for verified site owners to fix. It does not drive referral traffic or generate public citations back to your site.

Should I allow and optimize for AhrefsSiteAudit to drive organic growth?

AhrefsSiteAudit doesn't drive direct referral traffic to your site. However, Ahrefs data feeds into one of the most widely used SEO toolkits, and the crawled data also powers the Yep search engine. Allowing AhrefsSiteAudit ensures your site's technical SEO data is accurate in Ahrefs, which helps you (and competitors analyzing your site) see a complete picture. If you use Ahrefs yourself, blocking this bot undermines your own Site Audit reports. Even if you don't use Ahrefs, the data contributes to Ahrefs' broader index and Yep search results.

Data Usage & Training

Crawled content populates Ahrefs' databases and powers Ahrefs products like Site Audit, backlink indexes, and the Yep search engine. Ahrefs' public documentation does not state whether crawled data is used for AI model training, so that use remains unclear.

How AhrefsSiteAudit Accesses Content

Here's how AhrefsSiteAudit accesses your site and understands your content:

• Fetches HTML and renders JavaScript fully (headless browser rendering)
• Identifies as Mozilla/5.0 (compatible; AhrefsSiteAudit/6.1; +http://ahrefs.com/robot/site-audit)
• Respects robots.txt Disallow, Allow, Crawl-delay, and Sitemap directives
• Honors meta robots tags
• Verified site owners can configure Site Audit to bypass robots.txt for their own sites
• JavaScript rendering may request many assets in parallel, making Crawl-delay difficult to honor during rendering

On-demand or scheduled, depending on how the site owner configures their Site Audit project. Crawl rate is capped at up to 30 URLs per minute, and verified owners can adjust this in the Ahrefs tool settings.

How to Block or Control AhrefsSiteAudit

To block AhrefsSiteAudit via robots.txt: User-agent: AhrefsSiteAudit Disallow: / You can also block by IP. Ahrefs publishes its IP ranges at https://help.ahrefs.com/en/articles/78658-what-is-the-list-of-your-ip-ranges. To verify a request is genuinely from Ahrefs, perform a reverse DNS lookup and confirm the hostname ends with ahrefs.com or ahrefs.net. The bot automatically backs off when it receives 4xx or 5xx responses. One caveat: verified site owners can configure Site Audit to ignore robots.txt for their own domains, so if you own the site in Ahrefs, your robots.txt rules won't necessarily apply to your own audits.

Common Issues & Troubleshooting

Watch out for these common problems when working with AhrefsSiteAudit:

• Cloudflare, Incapsula, and ModSecurity often block the Ahrefs user-agent or IPs by default
• A slow or failing robots.txt response can prevent crawling entirely
• WordPress security plugins frequently block Ahrefs bots without warning
• Crawl-delay directives may not be honored during JavaScript rendering, since rendering triggers parallel asset requests
• Firewall rules blocking the user-agent string cause incomplete Site Audit reports