503 503 Service Unavailable

The server cannot handle the request right now and expects to be able to later. Overload, maintenance, a dependency that is down.

Crawlers treat it as temporary and that treatment is generous. A 503 for a few hours costs nothing: the crawler backs off, comes back, and the index is unchanged. This is the code to use when you need a site to be briefly unavailable, and it is much safer than the alternatives people reach for.

The generosity has a limit. Sustained 5xx responses stop reading as downtime and start reading as errors, and the URLs drop out. There is no published threshold, and the practical guidance is that hours are fine, days are risky, and a week is a problem you will be recovering from for longer than the outage lasted.

What causes it for crawlers specifically

Maintenance windows, correctly. Also incorrectly, when the maintenance page is served with a 200 and every URL on the site is briefly declared to be the same short notice.

Capacity, which is the crawler-specific one. Crawl traffic is not shaped like user traffic: it is steady, parallel, and goes to pages nothing else requests, so it misses every cache you have. A site that handles its peak human load comfortably can fall over under a crawler working through a large archive. Origin CPU and database connections run out before bandwidth does.

Dependency failures where the application returns 503 for a page that did not need the failed dependency at all. That one shows up as an oddly specific set of URLs erroring.

How to diagnose it

Separate crawler 5xx from user 5xx, because they usually have different causes:

awk '$9 ~ /^5/ {print $9, $7}' access.log | sort | uniq -c | sort -rn | head -20

If the erroring URLs are concentrated in one section, it is a dependency or a slow query rather than capacity. If they are spread evenly and correlate with request rate, it is capacity, and time to first byte under load is the number to look at.

Check /robots.txt specifically and separately. A 5xx there is the worst version of this problem: crawlers treat unavailable rules as a reason to stop, so a deploy that takes the rules file down for twenty minutes can pause crawling well beyond the deploy.

How to fix it

Send Retry-After with every 503:

HTTP/1.1 503 Service Unavailable
Retry-After: 3600

Serve /robots.txt from somewhere that does not depend on your application. A static file at the edge survives every deploy and every database outage, and it is the one URL where availability matters more than freshness.

If the cause is crawl volume rather than a fault, a 429 on the specific crawler is a better answer than a global 503, because it targets the source instead of refusing everyone. Reducing the crawlable URL space in robots.txt removes the load rather than rejecting it, which is the crawl budget version of the same fix.

Never substitute a 200. A maintenance notice with a 200 status is a soft 404 applied to your entire site at once, and the damage outlasts the outage by weeks.