Status codes

A status code is the whole conversation a crawler had with your server. These pages cover the ones that change whether a page stays indexed.

• 200 Soft 404 (a 200 that should be a 404)

  A soft 404 returns success for a page that has nothing. How to find them across a site, why crawlers keep fetching them, and the status codes that end it.

• 301 Redirect loop (a 301 or 302 cycle)

  A redirect loop sends a crawler in a circle until it gives up. The configurations that cause it, how to trace one, and why the page disappears from the index.

• 403 403 Forbidden

  A 403 tells a crawler the request was understood and refused. What causes it for bots, how to tell a deliberate 403 from an accident, and how to fix it.

• 404 404 Not Found

  A 404 tells a crawler the URL has no content. Why 404s are healthy, when they signal a real problem, and why 410 is sometimes the better answer.

• 429 429 Too Many Requests

  A 429 tells a crawler it is asking too fast. How crawlers respond, why Retry-After matters, and why 429 beats 403 for controlling crawl volume.

• 451 451 Unavailable For Legal Reasons

  A 451 says content is blocked for legal reasons. How crawlers treat it, when it is the honest code to return, and why it is rarer than it should be.

• 503 503 Service Unavailable

  A 503 tells a crawler the server is temporarily unable to respond. Why it is the right code for maintenance, and what a long-running 503 costs you.