Amazonbot

Operator

Amazon

User agent

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1) Chrome/W.X.Y.Z Safari/537.36

Robots token

Amazonbot

Obeys robots.txt

Yes, per the operator

Verification

Published IP ranges. Amazon publishes the crawler's IP addresses at https://developer.amazon.com/amazonbot/ip-addresses/. No reverse DNS hostname is documented, so the published list is the only check.

Operator docs

https://developer.amazon.com/amazonbot

Amazon's crawler:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1) Chrome/W.X.Y.Z Safari/537.36

W.X.Y.Z is a placeholder, the same convention Googlebot uses. The robots.txt token is Amazonbot.

Who operates it and what it feeds

Amazon. The stated purpose is broad: improving Amazon products and services, giving customers more accurate information, and possibly training Amazon AI models.

Read that as one bucket, because Amazon offers no way to split it. There is no training-only token here, no equivalent of Google-Extended or Applebot-Extended. Allowing Amazonbot means allowing all of those uses, and disallowing it refuses all of them. That is a cleaner design to document and a worse one to live with, since you cannot keep the product surfaces and skip the training.

Does it honor robots.txt

Yes. Amazon states that automated crawling from its listed user agents respects the Robots Exclusion Protocol.

It does not support Crawl-delay, and Amazon says so explicitly. This is the field most people get wrong, usually by copying a Crawl-delay line from a Bingbot example into an Amazonbot group where it is silently ignored. If the crawl rate is your problem, your options are narrowing the allowed path space or rate limiting at the edge with 429 responses and a Retry-After header.

How to verify a request

Match the source IP against the list Amazon publishes at https://developer.amazon.com/amazonbot/ip-addresses/.

No reverse DNS hostname is documented, so there is no DNS fallback and no way to check a single address quickly from a shell. That page is also substantially larger than the compact JSON files GPTBot and ClaudeBot publish, which makes automating the check more work than it should be. Fetch it on a schedule and cache the parsed result rather than hitting it per request.

Allow it

User-agent: Amazonbot
Allow: /

Block it

User-agent: Amazonbot
Disallow: /

Reduce the load without blocking

Since Crawl-delay does nothing, shrink the surface instead:

User-agent: Amazonbot
Disallow: /search
Disallow: /*?sort=
Disallow: /*?filter=
Allow: /

Those three patterns cover the generated URL space that consumes most of a site's crawl budget on every crawler, this one included. Fixing them once helps everywhere.

What blocking costs you

Hard to quantify, which is itself the answer. Amazonbot sends no attributed referral traffic the way a search crawler does, and Amazon does not publish where the crawled content surfaces. If you sell through Amazon, or your content describes products that Amazon's customer-facing answers might draw on, the cost is real but unmeasurable. If neither applies, blocking it is close to free.

One thing worth checking before you decide: whether the traffic in your logs is Amazonbot at all. AWS egress addresses carry a great deal of unrelated crawling, and a lot of it borrows recognizable agent strings. The published address list distinguishes the two, and a request claiming the name from outside that list deserves a 403 regardless of what you decide about the real one.

Find it in your logs

awk '$0 ~ /Amazonbot/ {print $1}' access.log | sort | uniq -c | sort -rn | head

Check the results against the published address list. With no reverse DNS method available, that file is the only line between Amazon's crawler and anything else running on a cloud address with a copied string.

Why the crawl rate is usually the real question

Most Amazonbot complaints are about volume rather than policy, and the missing Crawl-delay support turns the usual first move into a no-op. Three things actually change the rate.

Narrowing the crawlable URL space is the largest lever, because a faceted or parameterized site exposes an effectively unbounded number of URLs and any crawler will work through them. Returning correct status codes is the second: a 404 retires a URL, while a soft 404 keeps it in rotation forever because it looks like a page. Rate limiting at the edge is the third, and a 429 with a Retry-After header is the polite version of it.

Blanket-blocking is available but it is the blunt option, and with Amazon's single-token design it takes every use with it.

Check your site against Amazonbot