Soft 404

Also called soft not found, false 200

A URL that has no content behind it but returns HTTP 200 anyway. The server says the request succeeded and the page says nothing is here. Those two claims disagree, and crawlers act on the status code.

The common shapes: an error template rendered with a 200, a product page for a deleted product, an empty search result, a category with no items left, and a framework that renders its not-found route without setting the status.

The mistake

Assuming the visible message is the signal. It is not. A crawler records a 200, treats the URL as a live page, adds it to the recrawl schedule, and comes back for it indefinitely. Multiply that by every deleted item on a catalogue site and a large share of your crawl budget goes to pages that stopped existing.

Google reports these in Search Console as soft 404s, detected from the content rather than the status, which means the detection is approximate and arrives late. Other crawlers, including GPTBot and Amazonbot, mostly do not detect them at all and simply keep fetching.

The fix is the status code:

HTTP/1.1 404 Not Found

Use 410 Gone when the removal is permanent and you want the URL retired faster. Both are correct answers, and both stop the recrawl cycle in a way that no amount of copy on the page will.

Check your own framework before assuming it does this. Many render a not-found view with a 200 by default, and single-page applications are the worst case: the server has no idea the route is invalid, so every unrecognized path returns the app shell with a 200. See prerendering for how to make the server answer correctly, and status 404 for what a real one does.