Claude Code

What does Claude Code do?

Claude Code is the user-agent Anthropic uses when the Claude Code product fetches web content during development tasks. Unlike other AI bots that only surface content in chat responses, Claude Code can autonomously research dev tools, packages, datasets, and documentation on behalf of developers, then decide to integrate them into projects directly. Citations in Claude-powered results can drive referral traffic back to your site.

Should I allow and optimize for Claude Code to drive organic growth?

Claude Code fetches pages during active developer sessions where it researches tools, libraries, APIs, and datasets to help developers build software. Unlike traditional search bots, Claude Code doesn't just recommend your content in a response. It can autonomously decide to install a package, use an API, or follow your integration guide based on what it reads. That makes it one of the highest-value bots for developer-facing products: your documentation can directly influence adoption decisions. Claude supports structured citations with clickable links, so referral traffic follows. Blocking Claude Code removes your content from these autonomous development workflows entirely.

Data Usage & Training

Content fetched by Claude Code may contribute to model training. Anthropic uses different bots for different purposes: ClaudeBot handles training-focused crawling, Claude-User handles conversational retrieval, and Claude Code serves terminal and automation workflows. If you want to exclude your content from training, block the relevant tokens in your robots.txt.

How Claude Code Accesses Content

Here's how Claude Code accesses your site and understands your content:

• Fetches HTML via standard HTTP requests
• Respects robots.txt Disallow and Allow directives
• Supports Crawl-delay directives
• Honors Sitemap directives in robots.txt
• JavaScript rendering capability is unknown

Mixed pattern. Claude Code performs on-demand fetches when users initiate retrieval during coding sessions, but may also run scheduled or continuous crawling for search-quality indexing and model development.

How to Block or Control Claude Code

To block Claude Code, add the following to your top-level robots.txt: User-agent: Claude-Code Disallow: / You can also use Crawl-delay to rate-limit instead of fully blocking: User-agent: Claude-Code Crawl-delay: 10 Anthropic does not publish IP ranges for this bot, and IP-based blocking is not recommended. Anthropic warns that IP blocking can prevent bots from reading your robots.txt, which may cause the opposite of what you intend. Anthropic operates multiple crawlers with different tokens (ClaudeBot, Claude-User, Claude-Code), so block each one individually if you want to opt out entirely. Contact Anthropic support if blocking doesn't behave as expected.

Common Issues & Troubleshooting

Watch out for these common problems when working with Claude Code:

• IP-based blocking can prevent the bot from reading your robots.txt, causing unexpected behavior
• Robots.txt must be at the top level of your domain or it won't be found
• Anthropic uses multiple bot tokens, so blocking one doesn't block the others
• JavaScript-heavy pages may not render fully (rendering capability is unknown)
• Aggressive rate limiting or bot protection services may block legitimate fetches