Does YouBot respect robots.txt?

Yes. YouBot honors User-agent-specific Disallow and Allow rules, Crawl-delay directives, and Sitemap directives. Use the robots.txt token YouBot to target it specifically.

Does YouBot use my content for AI training?

This is unclear. You.com's documentation says crawled content powers search and feeds research/AI APIs, but does not explicitly confirm or deny model training. Contact spider@you.com if you need a definitive answer.

How do I verify that a request is actually from YouBot?

You have three options: reverse DNS lookup (hostnames should match youbot-{octets}.search.you.com), IP range validation against 68.67.112.0/24, or cryptographic verification using Cloudflare Web Bot Auth with Ed25519 signatures. The cryptographic method is the most reliable.

Can YouBot drive traffic to my site?

Yes. You.com search displays clickable links to source pages in its results. Content supplied through You.com's research and AI APIs can also reference source documents, though exact citation behavior varies by product.

How do I adjust YouBot's crawl rate without blocking it?

Add a Crawl-delay directive in your robots.txt under User-agent: YouBot. YouBot also adapts automatically based on your server's response codes. Returning 429 or 503 will signal it to slow down or pause.

Who do I contact about YouBot crawl issues?

Email spider@you.com. You can request crawl rate adjustments, report issues, or ask questions about how your content is being used.

Agent Directory You.comYouBot

YouBot

AI User Initiated

You.com agent fetching pages at user request for AI search answers.

What does YouBot do?

YouBot discovers and indexes web pages to power You.com's real-time search results and to supply grounded web content to You.com's research and AI APIs. It feeds into You.com search, where results appear as clickable links back to source pages. Content supplied through research and AI APIs can also reference source documents, giving your site visibility across You.com's product ecosystem.

Should I allow and optimize for YouBot to drive organic growth?

You.com search displays clickable links back to source pages, creating a direct referral traffic channel. Content indexed by YouBot also feeds You.com's research and AI APIs, where source documents can be referenced. Allowing YouBot gives your site visibility across You.com's search and AI products. Blocking it removes your content from these surfaces entirely.

Here's how to optimize for YouBot:

Allow YouBot in your robots.txt to ensure your pages appear in You.com search results
Add a Sitemap directive to help YouBot discover your pages efficiently
Use a reasonable Crawl-delay value if you need to manage server load without fully blocking the bot
Ensure pages return proper HTTP status codes (200 for live content, 404 for removed pages) so the index stays accurate
Include descriptive title tags and meta descriptions, as these may appear in You.com search results
Keep important content in the initial HTML rather than loading it entirely via JavaScript

Data Usage & Training

It is unclear whether You.com uses crawled content to train its own AI models. The official documentation states that crawling powers search and supplies content for research and AI APIs. Indexed content can also be supplied to AI workflows operated by downstream consumers through You.com's APIs. If this concerns you, contact [email protected] for clarification.

How YouBot Accesses Content

Here's how YouBot accesses your site and understands your content:

Fetches HTML via standard HTTP requests using a Chrome-based user-agent string
Respects robots.txt directives including Disallow, Allow, Crawl-delay, and Sitemap
Adapts crawl rate based on content-change frequency, site popularity, and server responses
Supports cryptographic verification via Cloudflare Web Bot Auth and HTTP message signatures (Ed25519)
Can be verified through reverse DNS (hostnames follow the pattern youbot-{octets}.search.you.com)
Operates from the IP range 68.67.112.0/24

YouBot uses adaptive, continuous crawling. It adjusts frequency based on how often your content changes, your site's popularity, server response codes, and any Crawl-delay directives you set. It can also perform on-demand crawls for specific user requests.

How to Block or Control YouBot

To block YouBot via robots.txt: User-agent: YouBot Disallow: / To slow it down instead of blocking entirely, use Crawl-delay: User-agent: YouBot Crawl-delay: 10 For IP-based blocking, YouBot operates from 68.67.112.0/24. Before blocking by IP, verify requests using reverse DNS (hostnames should match youbot-{octets}.search.you.com) or cryptographic verification via Cloudflare Web Bot Auth with Ed25519 public JWK keys. You can also return 429 or 503 status codes to signal YouBot to reduce or pause crawling. For custom crawl adjustments, contact [email protected] directly.

Common Issues & Troubleshooting

Watch out for these common problems when working with YouBot:

User-agent strings can be spoofed, so verify YouBot requests using reverse DNS or cryptographic signatures before making access decisions
Static IP-block rules can break if You.com's IP ranges change; prefer robots.txt or verification-based approaches
Returning 429 or 503 errors unintentionally (due to rate limiting or downtime) may cause YouBot to pause crawling your site
JavaScript-heavy pages may not be fully rendered, as YouBot's JavaScript rendering capability is unknown
Cloudflare or similar bot protection services may block YouBot if not configured to allow it