robots.txt tester

What the test actually checks

robots.txt is a crawl instruction and nothing else. It decides whether a crawler is allowed to request a URL. It has no say in whether the page gets indexed, how it ranks, or whether a human can open it. Confusing those two jobs is behind most of the broken files this tool finds.

Matching is stricter than it looks. A crawler picks exactly one group, the one whose User-agent line most specifically names its token, and ignores every other group in the file. Once it has picked, the longest matching rule wins, and a tie between an Allow and a Disallow of equal length goes to Allow. So a file with a permissive star group and a restrictive Googlebot group does not combine them. Googlebot reads its own group and nothing else.

The failure that keeps showing up

A team wants a page out of search, adds a Disallow for it, and waits. The page stays in the results with no description under it, sometimes for months. What happened is that the page already carries a noindex tag, and the Disallow now prevents any crawler from fetching the page to read that tag. The two directives cancel each other. The fix is to remove the Disallow, let the crawl happen, and let the noindex directive do the work it was written for.

The other common one is quieter. Someone tightens a firewall or a CDN rule, and now a named crawler gets a 403 even though robots.txt allows it. Nothing in the file is wrong, so nobody looks at the server. That gap is why this tool reports the rule match and the actual response status side by side. If robots.txt says allowed and the status says 403, the problem is in front of your application, not in the file, and what a 403 means to a crawler is worth reading before you touch anything.

How to fix what the table shows

Work down from the crawlers that matter to you. For each row marked blocked, read the rule line the table prints. If that line is not the one you meant to write, the path pattern is wrong, usually a missing trailing slash or a wildcard that swallowed more than intended. If the rule is right and the block is not wanted, delete it rather than adding an Allow above it, because one clear file beats a stack of exceptions.

Serve the file as plain text with a 200, at the root of the host, and keep it small. A robots.txt that answers 500 is treated by some crawlers as a full block for as long as the error lasts, which turns a bad deploy into a crawl outage. The background is in the robots.txt entry in the glossary.

For the rendering verdict, the redirect chain and the rest of the picture, run the full report. The measurement rules behind every row, including how many requests a check sends, are on the how it works page.