How robots.txt decides what a crawler may fetch
Testing a URL against robots.txt means answering two questions in order: which group in the file does this crawler read, and which line inside that group matches the path. Almost every surprising answer comes from the first one.
What the file actually controls
robots.txt is a crawl instruction and nothing else. It decides whether a crawler is allowed to request a URL. It has no say in whether the page gets indexed, how it ranks, or whether a human can open it. Confusing those two jobs is behind most of the broken files worth looking at.
Matching is stricter than it looks. A crawler picks exactly one group, the one whose User-agent line most specifically names its token, and ignores every other group in the file. Once it has picked, the longest matching rule wins, and a tie between an Allow and a Disallow of equal length goes to Allow. So a file with a permissive star group and a restrictive Googlebot group does not combine them. Googlebot reads its own group and nothing else.
The token is not the user agent string. Googlebot sends a long string with a version and
a URL in it, and matches the group named Googlebot. Each
crawler page here lists the token its operator documents, which is the
string to write in the file.
The failure that keeps showing up
A team wants a page out of search, adds a Disallow for it, and waits. The page stays in the results with no description under it, sometimes for months. What happened is that the page already carries a noindex tag, and the Disallow now prevents any crawler from fetching the page to read that tag. The two directives cancel each other. The fix is to remove the Disallow, let the crawl happen, and let the noindex directive do the work it was written for.
The other common one is quieter. Someone tightens a firewall or a CDN rule, and now a named crawler gets a 403 even though robots.txt allows it. Nothing in the file is wrong, so nobody looks at the server. That is why the rule match and the response status have to be read side by side. If robots.txt says allowed and the status says 403, the problem is in front of your application, not in the file, and what a 403 means to a crawler is worth reading before you touch anything.
Reading a result and fixing what it shows
Work down from the crawlers that matter to you. For each one that comes back blocked, read the rule line that decided it. If that line is not the one you meant to write, the path pattern is wrong, usually a missing trailing slash or a wildcard that swallowed more than intended. If the rule is right and the block is not wanted, delete it rather than adding an Allow above it, because one clear file beats a stack of exceptions.
Serve the file as plain text with a 200, at the root of the host, and keep it small. A robots.txt that answers 500 is treated by some crawlers as a full block for as long as the error lasts, which turns a bad deploy into a crawl outage. The background is in the robots.txt entry in the glossary.
Checking your own file
Testing a real URL means reading two facts that can disagree: the rule match in your robots.txt, and the status your server returns to that crawler's user agent string. Each crawler page gives you both strings, the token for the first and the agent for the second.
For the order the steps happen in, and why robots.txt is fetched before anything else, see how a crawler resolves a page. The X-Robots-Tag header is the other half of this, since it does the job people keep asking Disallow to do.
Common questions
My robots.txt has no rule for GPTBot. Is it blocked?
No. A crawler with no group of its own falls back to the User-agent: * group, and if there is no star group either, everything is allowed. Silence in robots.txt means yes.
Why is a page still in search results after I disallowed it?
Disallow stops the fetch, not the listing. A URL that other pages link to can be indexed from those links alone, with no snippet, because the crawler is not allowed to read the noindex you put on the page. Allow the crawl and serve noindex instead.
Which file governs a URL on a subdomain?
The robots.txt of that exact host and scheme. Rules on the apex do not cover a subdomain, and an https page is governed by the https file, which is the most common reason a test disagrees with what someone expects.
Does a Crawl-delay line do anything?
Google ignores it outright. Bing and Yandex have documented support for it. Since the two operators that honor it are not usually the ones causing your load, rate limiting at the edge is the control that actually applies.