429 429 Too Many Requests

The client is sending requests faster than the server is willing to serve them. It is a rate limit, not a refusal of the content, and the distinction matters because crawlers act on it.

A 429 is temporary by definition. A compliant crawler slows down, comes back, and keeps the URL in its schedule. A 403 in the same situation says never, and eventually the URL leaves the index. Using 403 as a rate limiter is one of the more expensive configuration mistakes available.

What causes it for crawlers

Volume, usually from somewhere you were not watching. Bingbot crawling harder than Googlebot on the same site is common. An AI crawler working through a newly discovered section can produce a sustained burst that looks nothing like human traffic.

Shared rate limits are the second cause. A limit keyed on a network rather than a client treats every crawler from one cloud provider as one requester, and the whole group trips a threshold none of them approached alone.

Then there are limits applied to the wrong thing. A per-IP limit that counts static asset requests will throttle a crawler fetching one page with forty images, even though it asked for a single document.

How to diagnose it

Count 429s by verified crawler over time:

awk '$9 == 429 {print $1}' access.log | sort | uniq -c | sort -rn | head

Then verify the top addresses before concluding anything, because unverified traffic wearing a crawler name is a different problem with a different fix. The verification methods are on each bot page.

Check whether the pattern is steady or spiky. Steady 429s mean your limit is below the crawler's normal rate and needs raising. Spikes mean a burst, which is what Retry-After is for.

Watch your time to first byte alongside the error rate. A limit that was comfortable at 100ms becomes tight at 800ms, because the same request rate now holds far more connections open at once.

How to fix it

Send Retry-After on every 429. Seconds or an HTTP date both work:

HTTP/1.1 429 Too Many Requests
Retry-After: 120

Without it the crawler guesses, and its guess is usually more conservative than you wanted.

Give crawl control to crawlers that offer it. Crawl-delay works for Bingbot and ClaudeBot. It does nothing for Googlebot or Amazonbot, which is worth knowing before you add the line and assume it took effect. Google's rate control lives in Search Console.

Reduce demand rather than refusing it. Most heavy crawl volume is spent on generated URLs, so blocking parameter and faceted patterns in robots.txt cuts the request count at the source. That is a crawl budget fix that also makes the rate limit unnecessary.

Never return 429 on /robots.txt. A crawler that cannot read the rules cannot follow the very instruction that would have slowed it down.