Common Crawl

What does Common Crawl do?

CCBot is Common Crawl's web crawler that continuously fetches and archives large-scale samples of the open web into publicly available WARC archives and indexes. These datasets are widely used by researchers, analytics teams, and organizations building and training machine learning models. CCBot does not power any user-facing search or assistant product, so it does not drive referral traffic or citations back to your site.

Should I allow and optimize for Common Crawl to drive organic growth?

CCBot feeds open datasets, not a user-facing product. It won't send you referral traffic or generate citations. That said, Common Crawl data underpins training corpora used by many AI companies. Allowing CCBot means your content may influence models trained on those corpora, but the connection between your pages and any specific AI product's output is indirect and untraceable. Blocking CCBot is a reasonable choice if you want to limit broad redistribution of your content.

Data Usage & Training

Content crawled by CCBot is published as open WARC archives that anyone can download. These snapshots are the foundation for widely used AI training corpora including C4 and CCNet. Organizations download the raw data, filter and clean it, then use it to train large language models. If you don't want your content included, block CCBot via robots.txt or register with Common Crawl's opt-out process.

How Common Crawl Accesses Content

Here's how Common Crawl accesses your site and understands your content:

• Fetches HTML via standard HTTP requests using a Nutch-based crawler
• Does not render JavaScript
• Respects robots.txt Disallow, Allow, Crawl-delay, and Sitemap directives
• Honors nofollow in meta robots tags; compliance with noindex is not explicitly documented
• Uses adaptive, polite rate-limiting during crawls
• Identifies as CCBot/2.0 (https://commoncrawl.org/faq/)

CCBot crawls continuously with adaptive rate-limiting. Data is published in large CC-MAIN snapshots approximately once per month, each containing billions of pages.

How to Block or Control Common Crawl

To block CCBot via robots.txt: User-agent: CCBot Disallow: / You can also rate-limit instead of fully blocking: User-agent: CCBot Crawl-delay: 10 For IP-based blocking, Common Crawl publishes IP ranges at https://index.commoncrawl.org/ccbot.json. Verify requests using reverse DNS and the published IP ranges to distinguish real CCBot from spoofed user-agent strings. Common Crawl also operates an opt-out registry; contact them directly or check their blog for details on how to register.

Common Issues & Troubleshooting

Watch out for these common problems when working with Common Crawl:

• Other bots spoof the CCBot user-agent string, making log analysis unreliable without IP verification
• CDN and edge rate-limits sometimes block legitimate CCBot traffic before robots.txt is even checked
• Aggressive WAF rules can inadvertently block CCBot alongside malicious crawlers
• Robots.txt parser differences across platforms can cause unexpected allow/block behavior
• Meta robots noindex tags are captured in WARC files but may not prevent downstream use by third parties who process the archives differently