Crawl-delay

Also called crawl delay, Crawl-delay directive

User-agent: bingbot
Crawl-delay: 10

A non-standard robots.txt line asking one crawler to leave that many seconds between requests to your host. It is not part of the robots exclusion protocol standard, nobody is obliged to implement it, and most large operators have not.

Bingbot reads it. So does ClaudeBot. Googlebot ignores it, and Amazonbot does too, which Amazon says in as many words. Before you write the line for a crawler, check that crawler's page for whether it supports one.

Placement follows the same rule as everything else in robots.txt. The directive belongs inside a User-agent group and applies only to the crawler that reads that group, so a Crawl-delay under User-agent: * does nothing for a bot that has a named group of its own.

The number is seconds, mostly. Operators who implement the directive do not all agree on what it means: some treat it as a literal gap between requests, others use it to size a window in which they fetch once. Neither reading is wrong, and the difference only matters at the edges.

The mistake

Writing the line and considering the rate handled.

A robots.txt parser that does not implement Crawl-delay skips the line and moves on. The file still parses, nothing warns you, no report says the directive was dropped. From the outside, a supported directive being honored and an unsupported one being ignored look identical: a valid file and a crawler that keeps doing whatever it was doing. The only place the difference shows up is your access log. Count requests from that user agent per minute before and after, over a window long enough that the crawler has refetched robots.txt, and you will know.

The second mistake is the value. Ten seconds sounds polite. Do the arithmetic and it is a hard ceiling of a few thousand fetches a day from that crawler, which on a large site means your new pages are found some other month. A delay is a crawl budget decision, not a courtesy, and the polite-sounding numbers are the dangerous ones.

What to reach for instead

If the problem is that your origin buckles under crawl volume, the delay is treating a symptom. Check time to first byte first, since a slow origin makes an ordinary crawl rate look like an attack, and fixing it raises your capacity instead of lowering their rate.

If you need a crawler to back off right now and it does not read the directive, answer it. A 429 with a Retry-After header is the documented way to say slow down, and a 503 is the way to say not now. Googlebot honors both. This works on every crawler that behaves, because it is part of HTTP rather than an extension somebody chose to support.

If the volume is real but the pages are not worth it, cut the path space with Disallow instead. Faceted URLs, internal search results and calendar pages are usually most of the traffic, and removing them from the crawl drops the rate without asking anyone's permission.