What robots.txt does not tell you

Published September 15, 2026

Reading robots.txt tells you what a site intends. It does not tell you what a crawler receives, and those come apart more often than they should.

A robots.txt file can allow every crawler while a CDN rule refuses two of them. A page can serve full HTML to a browser and a skeleton to anything without a JavaScript engine. Neither shows up in a robots tester, because a robots tester never sends a request.

Ask as the crawler

Send one request per crawler, carrying that crawler's published agent string, after applying the robots.txt rules for its token. Record the status code, the response size and the rule that matched. When robots says allowed and the response is a 403, the refusal sits in front of the site, and that is a different team and a different fix.

The strings and tokens to send are on the crawler pages, one per operator, each copied from the operator's own documentation. The order the steps run in is on how a crawler resolves a page.