404 404 Not Found

The URL has no content behind it. The request was fine, the server understood it, and there is nothing there.

Crawlers handle this well. A 404 retires the URL from the recrawl schedule over time, with occasional retries in case it was temporary. Google has said repeatedly that 404s are normal and not a penalty. A site with zero 404s is usually a site with a bug, because links rot and content gets deleted.

What causes it for crawlers specifically

Crawlers reach URLs that no user ever clicked, which is why their 404 list looks different from yours.

They follow links from other sites, including mistyped ones and ones that reference a structure you changed years ago. They retry URLs that used to exist, long after you removed them. They parse anything link-shaped out of JavaScript and JSON, which produces requests for paths that were never real URLs. They read your sitemap, which is the one source where a 404 is unambiguously your fault.

How to diagnose it

Separate the harmless from the real. A 404 matters when something you control points at it.

Check the sitemap first. Every URL in it should return 200, and a 404 there is a direct contradiction of your own declaration:

curl -s https://example.com/sitemap.xml | grep -o '<loc>[^<]*' | cut -c6- | \
  while read u; do printf "%s %s\n" "$(curl -s -o /dev/null -w '%{http_code}' "$u")" "$u"; done | grep -v '^200'

Then check internal links, which is where a template bug shows up as thousands of identical broken paths. Then check which 404s have inbound links from other sites, because those are the ones worth a redirect: the link equity is real and a 404 discards it.

Everything else is noise, and chasing it wastes time that the sitemap and internal link problems deserve.

How to fix it

For a page that moved, a 301 to the new location, pointing directly at the destination rather than through a chain. Watch for the redirect loop that a careless rule creates.

For a page that is gone with no equivalent, leave the 404. If you are certain it is permanent, return 410 instead and the URL retires faster.

For a page that should exist, fix the routing. That sounds obvious and it is the case people skip, because the 404 gets classified as expected noise and nobody checks.

What not to do is redirect everything to the homepage. The destination does not answer the request, so it is read as a soft 404, and you have replaced a correct signal with a wrong one while also making the problem invisible in your own reports.

Custom 404 pages are fine and worth building. Just make sure the status code is still 404. A friendly error page served with a 200 is the single most common way a site ends up with thousands of phantom pages consuming its crawl budget.