DuckDuckBot

What does DuckDuckBot do?

DuckDuckBot is DuckDuckGo's web crawler that fetches and indexes pages to power DuckDuckGo search results. It runs continuously to keep the search index up to date. Pages indexed by DuckDuckBot appear directly in DuckDuckGo search results with clickable links back to your site, making it a direct source of referral traffic.

Should I allow and optimize for DuckDuckBot to drive organic growth?

DuckDuckGo holds a smaller market share than Google or Bing, but its privacy-focused user base is loyal and growing. Allowing DuckDuckBot means your pages can appear in DuckDuckGo search results with direct citation links. DuckDuckGo users tend to click through to source sites since the platform doesn't personalize or filter results as aggressively as larger engines. Allow DuckDuckBot and treat it like any other search engine crawler to capture this traffic.

Data Usage & Training

DuckDuckGo's official documentation states that DuckDuckBot crawls pages to improve search results, but does not clarify whether crawled content is also used to train AI models. If this distinction matters to you, contact DuckDuckGo directly for clarification.

How DuckDuckBot Accesses Content

Here's how DuckDuckBot accesses your site and understands your content:

• Fetches HTML via standard HTTP requests
• Partial JavaScript rendering capability
• Identifies as DuckDuckBot/1.1; (+http://duckduckgo.com/duckduckbot.html)
• Crawls from IP addresses published at https://duckduckgo.com/duckduckbot.json
• Reads Sitemap directives from robots.txt

DuckDuckBot crawls continuously on a regular schedule as part of its search indexing process. There is no documented support for the Crawl-delay directive.

How to Block or Control DuckDuckBot

To block DuckDuckBot via robots.txt: User-agent: DuckDuckBot Disallow: / For IP-based blocking, DuckDuckGo publishes its crawler IP prefixes at https://duckduckgo.com/duckduckbot.json. You can use this list in your firewall or server config. Verify requests by matching the source IP against this published list, since user-agent strings can be spoofed. The IP list can change over time, so check it periodically if you rely on IP-based rules.

Common Issues & Troubleshooting

Watch out for these common problems when working with DuckDuckBot:

• DuckDuckBot has limited JavaScript rendering, so content behind client-side frameworks may not be indexed
• User-agent string matching alone is unreliable for verification; always cross-check against the published IP list
• Third-party bot lists and IP blocklists may contain stale DuckDuckBot IPs, causing false blocks or missed blocks
• Crawl-delay is not supported, so you cannot throttle DuckDuckBot through robots.txt alone
• CloudFlare or similar bot protection services may block DuckDuckBot if not explicitly allowlisted