How a crawler resolves a page

Most crawlability surprises are an ordering problem. A step you thought ran first ran last, or a step you thought ran at all was skipped. Here is the order.

One: robots.txt, before anything else

A compliant crawler fetches robots.txt from the root of the exact host and scheme it is about to request, and it does that before it asks for your page. Rules on your apex do not cover a subdomain. An https page is governed by the https file. That host and scheme pairing is the single most common reason a robots test disagrees with what somebody expected.

The status code on that file decides more than its contents do. A 404 means no rules exist and everything is allowed. A persistent 5xx is read by Google as a full disallow for the host until it recovers. So a deploy that leaves robots.txt returning a 500 is a crawl outage with no error anywhere in your logs that says so.

Two: group selection

The crawler then picks one group out of the file. Not the groups that apply, one group. It takes the group whose User-agent line most specifically names its token and ignores every other group, including the star group. Once inside that group, the longest matching path rule wins, and a tie between an Allow and a Disallow of equal length goes to Allow.

This is where a file that reads sensibly to a human behaves differently to a crawler. Adding a two line Googlebot group to a file with a carefully built star group does not add restrictions to the star rules. It replaces them for Googlebot.

Three: the fetch, and what sits in front of it

Only now does a request go out, carrying the crawler's own user agent string. Two different systems can refuse it, and they fail in ways that look alike from the outside. robots.txt refusing means no request was ever sent. A CDN, WAF or firewall rule refusing means the request was sent and something in front of your application answered 403 or 429. Nothing in your site is wrong in the second case, which is why nobody looks there.

The response status is read differently by a crawler than by a browser. A 503 with a Retry-After means come back, and the URL keeps its place. A 404 means the URL is gone and the listing follows it out. A soft 404, which is a 200 carrying an error page, is the worst of the three because the crawler has to guess.

Four: the redirect chain

Each hop is a separate request against your crawl allowance. Google follows about five hops in one pass and picks the chain up on a later visit if it runs out, which delays a move rather than stopping it. A canonical tag that names a third address on top of a redirect is a conflicting signal, and the crawler resolves that conflict on its own schedule.

Five: rendering, if it happens

Most crawlers stop at the HTML the server returned. Google splits the work into two passes, reading the raw HTML immediately and queueing the render for later, which means anything built by a script is indexed on a delay you do not control. JavaScript rendering covers what that queue actually is.

Links are the part people miss. If a category page builds its product grid in the browser, the raw HTML contains no links to those products at all, so a crawler that does not render has no route to them. The category page is fine. Everything underneath it is unreachable.

Observing the order on your own site

Every step above leaves a trace you can read. robots.txt and its status code are one fetch. The rule match is a text comparison you can do by hand. The redirect chain and the response status come from a request that does not follow redirects automatically. The rendering gap is the same URL fetched twice and diffed.

Ask as the crawler you care about. The answer changes with the user agent string, because the CDN rule and the robots group both key on it, so a check run as a browser tells you about browsers. Each crawler page carries the string to send and the token to match, and the crawler listing and the open source listing cover software that makes these requests for you, with what each one gives up.