Crawlability checks, explained
Five checks people run against a site, each with a page on what it actually looks at, how a crawler resolves it, and how to read a result once you have one.
-
robots.txt matching
How a crawler picks one group out of your file, why the longest rule wins, and the Disallow that keeps a page in search results instead of removing it.
-
llms.txt
The shape the format asks for, the three mistakes that make a file unreadable, and what publishing one does not do.
-
Redirect chains
What 301, 302 and 307 each claim, how hops stack up without anyone deciding to stack them, and the loop that removes a page nobody deleted.
-
JavaScript rendering
Why the gap between raw HTML and the rendered page is the number that matters, and which crawlers never close it.
-
Structured data
Why a JSON-LD block that fails to parse is discarded whole and silently, and the template habit that causes it.
Why these five
They are the checks that come up when a page stops behaving, and the four of them that are not rendering are cheap to get wrong. A missing trailing slash in a path pattern, a second redirect rule stacked on an old one, an apostrophe in a product title breaking the JSON. None of those look like anything in a browser, and each one changes what a crawler ends up with.
Start from the order a crawler does things in when the symptom is vague. Half of what looks like a rendering problem is a robots rule that stopped the fetch before rendering was ever on the table.
Running the check itself
Reading how a check works gets you the interpretation. Getting the answer for your own URL means sending a request as the crawler you care about and reading the response, because a CDN rule and a robots group both key on the user agent string. The crawler pages carry the string to send and the token to match.
For software that makes those requests for you, the crawler listing and the open source listing name what each option does and where it stops working. Neither list has a winner in it.
Common questions
Can I run these checks on this site?
No. These are explainers. Running a check means sending requests to your server as a named crawler, which takes a crawler rather than a web page. The crawler pages here give you the agent string and the token to send.
Which of these should I read first?
Whichever matches the symptom. A page that vanished from results is usually robots.txt or a redirect. A page that ranks but shows the wrong snippet is usually rendering or structured data.
Do these checks interact?
Yes, and the order matters. A page blocked in robots.txt has no rendering verdict worth reading, because nothing compliant fetched it. A redirect chain ending on a 404 makes the structured data count zero for a reason that has nothing to do with your markup.