Claude-SearchBot

What does Claude-SearchBot do?

Claude-SearchBot crawls the public web to index content and improve the relevance of Claude's search results. It powers Claude's search features, helping the AI assistant return accurate, up-to-date answers when users ask questions. When citations are enabled, Claude returns structured citations with source URLs, titles, and cited text blocks, which can drive referral traffic back to your site.

Should I allow and optimize for Claude-SearchBot to drive organic growth?

Claude-SearchBot directly feeds the search index that powers Claude's answers. When Claude cites your content, it includes structured citations with clickable source URLs, creating a direct referral traffic path. Blocking this bot removes your content from Claude's search index, which reduces your visibility and accuracy in Claude-powered responses. For most sites, allowing Claude-SearchBot is a straightforward way to maintain presence in a growing AI search channel.

Data Usage & Training

Content crawled by Claude-SearchBot is used to build and improve Claude's search index. Anthropic separates its crawlers by purpose: ClaudeBot handles model development and training, Claude-SearchBot handles search indexing, and Claude-User fetches pages at a user's direct request. Blocking Claude-SearchBot prevents your content from being indexed for search but does not affect training data collection (that requires blocking ClaudeBot separately).

How Claude-SearchBot Accesses Content

Here's how Claude-SearchBot accesses your site and understands your content:

• Fetches HTML via standard HTTP requests
• Respects robots.txt Disallow and Allow directives
• Supports non-standard Crawl-delay directive
• Recognizes Sitemap directives
• Rate-limits requests on a per-site basis
• JavaScript rendering capability is unknown

Claude-SearchBot performs ongoing, continuous indexing with rate-limited requests. It respects Crawl-delay directives and applies per-site rate limits to avoid overwhelming servers.

How to Block or Control Claude-SearchBot

To block Claude-SearchBot via robots.txt: User-agent: Claude-SearchBot Disallow: / Each Anthropic bot requires its own directive. Blocking ClaudeBot does not block Claude-SearchBot or Claude-User. You need separate rules for each. IP-based blocking is unreliable because Anthropic uses public cloud provider IPs, and blocking those IPs may also prevent the bot from reading your robots.txt. If you believe the crawler is not respecting your directives, contact Anthropic support with your domain details.

Common Issues & Troubleshooting

Watch out for these common problems when working with Claude-SearchBot:

• Blocking only ClaudeBot does not block Claude-SearchBot or Claude-User. Each token needs its own robots.txt directive.
• IP-based blocking is unreliable because the bot uses shared cloud provider IPs. Blocking those IPs may also prevent robots.txt from being read.
• Some site owners report apparent continued crawling after adding robots.txt rules. Anthropic asks you to contact support with domain details if this occurs.
• JavaScript-heavy pages may not be fully rendered or indexed, since Claude-SearchBot's JS rendering capability is unknown.
• Aggressive rate limiting or bot protection services (like CloudFlare) can block legitimate crawl requests.