What the bot matrix answers

Published September 15, 2026

Reading robots.txt tells you what a site intends. It does not tell you what a crawler receives, and those come apart more often than they should.

A robots.txt file can allow every crawler while a CDN rule refuses two of them. A page can serve full HTML to a browser and a skeleton to anything without a JavaScript engine. Neither shows up in a robots tester, because a robots tester never sends a request.

Asking as the crawler

The matrix sends one request per crawler, carrying that crawler's published agent string, after applying the robots.txt rules for its token. The row reports the status code, the response size and the rule that matched. When robots says allowed and the row says 403, the refusal is in front of the site, and that is a different team and a different fix.

The method, including how many requests a check sends, is on how it works.