Amazonbot

What does Amazonbot do?

Amazonbot crawls web content to improve Amazon products and services, including training Amazon AI models. It feeds into Amazon search experiences like Alexa and Rufus. Amazon also operates related variants: Amzn-SearchBot (makes content eligible for Amazon search) and Amzn-User (fetches live information for user-initiated queries). Whether Amazonbot drives direct referral traffic back to your site is unclear.

Should I allow and optimize for Amazonbot to drive organic growth?

Allowing Amazonbot and its variants keeps your content eligible for Amazon search experiences like Alexa and Rufus. When users ask Alexa or Rufus questions, your content may surface in their answers. The exact referral and citation behavior is not well documented, so direct traffic benefits are uncertain. Still, Amazon's search products reach a massive user base, and blocking these crawlers removes your content from consideration entirely. If you're comfortable with training use, allow Amazonbot. If not, consider allowing only Amzn-SearchBot and Amzn-User to maintain search eligibility without contributing to AI model training.

Data Usage & Training

Content crawled by Amazonbot may be used to train Amazon's generative AI models. Amazon documents two related variants, Amzn-SearchBot and Amzn-User, that explicitly do not crawl for AI model training. If you want to opt out of training while remaining eligible for Amazon search experiences, you can block Amazonbot while allowing the other two variants. You can also add a noarchive meta tag to individual pages to signal that content should not be used for model training.

How Amazonbot Accesses Content

Here's how Amazonbot accesses your site and understands your content:

• Fetches HTML via standard HTTP requests using a Chrome-based user-agent string
• Respects robots.txt Allow and Disallow directives
• Honors meta robots tags including noindex, noarchive, and none
• Respects rel=nofollow on links
• Caches your robots.txt for up to approximately 30 days
• Does not support the non-standard Crawl-delay directive

Amazon does not publish specific crawl frequency data. The pattern likely varies by variant: Amazonbot crawls continuously for indexing purposes, while Amzn-User fetches pages on-demand in response to user queries. Robots.txt changes may take up to 30 days to take effect due to caching.

How to Block or Control Amazonbot

To block all Amazon crawlers, add the following to your robots.txt: User-agent: Amazonbot Disallow: / User-agent: Amzn-SearchBot Disallow: / User-agent: Amzn-User Disallow: / To block only training crawls while keeping search eligibility, block Amazonbot but allow the other two. You can also use page-level meta tags: noindex to prevent indexing, noarchive to opt out of model training, or none for both. For IP-based blocking, Amazon publishes IP ranges at https://developer.amazon.com/amazonbot/ip-addresses/. Be aware that robots.txt changes may take up to 30 days to take effect due to Amazon's caching behavior. For publisher-specific issues, contact amazonbot@amazon.com.

Common Issues & Troubleshooting

Watch out for these common problems when working with Amazonbot:

• Three separate agent tokens (Amazonbot, Amzn-SearchBot, Amzn-User) make robots.txt rules easy to misconfigure; each must be specified individually
• Robots.txt caching of up to 30 days means rule changes take a long time to propagate
• Amazon's IP ranges span many AWS prefixes, making IP-based filtering complex to maintain
• Crawl-delay is not supported, so you cannot rate-limit Amazonbot through robots.txt
• Blocking Amazonbot alone does not block Amzn-SearchBot or Amzn-User, which operate independently