How the crawlability check works
Every number this site publishes was measured this way. If a verdict looks wrong, the method below is where to look first.
The modes
A full check runs the same URL three ways and compares the results. Nothing is inferred from the framework, the headers or the shape of the HTML.
Raw
One HTTP GET, no scripts, redirects followed and recorded. This is what a crawler that does not render gets, which is still most of them.
Rendered
The same URL in a real browser engine, with ads, analytics, stylesheets and images blocked so the measurement is about content rather than page weight. The difference between raw and rendered is the JavaScript dependence verdict. If a heading, a price or a link exists only in the rendered result, the report names it.
Bot matrix
One request per crawler, sending that crawler's published user agent string, with the robots.txt rules for its token applied first. Each row reports the status code, the response size, whether robots.txt allowed the fetch, and the rule line that matched.
What each verdict means
| Verdict | What was observed |
|---|---|
| Allowed | robots.txt permits the path for that token and the fetch returned a 2xx. |
| Blocked by robots | A Disallow rule in the group for that token matched the path. The request was not sent. |
| Blocked by server | robots.txt allowed it, but the server answered 401, 403 or 429 to that agent. Usually a CDN or firewall rule rather than anything in your site. |
| Needs JavaScript | The rendered result contains content the raw result does not. Crawlers that do not render will miss it. |
| Timeout | No response inside the budget for that mode. |
| Not checked | An earlier step failed, so running this one would report noise. The report says which step. |
Politeness
The rules are short enough to state in full, and stating them is the point. One request per check per mode, never in parallel against one host. Every request carries an agent string that says what it is, and the bot matrix requests carry the string of the crawler being tested, which is what makes the test meaningful. robots.txt is fetched first and its rules are applied before any request goes out. No proxies, no fingerprint evasion, no retry storm: a failed check is reported as failed.
Rate limits run per client and per target host, so a site cannot be hammered by many people checking it at once. Results are cached briefly, so a second check of the same URL inside that window may return the earlier report rather than re-fetching your page.
Limits
- Public pages only. No cookies, no credentials, no login.
- One URL per check. seoder does not crawl a site, follow internal links, or read a sitemap.
- http and https only, on their standard ports. Private, loopback and link local addresses are refused.
- A blocked result is reported, never worked around, so some sites cannot be measured from here at all.
- The rendered fetch uses one viewport and one engine. A site that serves different markup by device may look different to a phone crawler.
Questions
How many requests does one check make to my server?
One for robots.txt, one for llms.txt, one raw fetch, one rendered fetch, and one per crawler in the matrix. A default run is under ten requests, sent in sequence rather than at once.
Will a check show up in my analytics?
The rendered fetch blocks analytics scripts, so it should not. The raw and bot fetches never run scripts at all, so they only appear in server logs.
Why does the tool identify itself as the crawler it is testing?
Because the answer changes with the user agent. A CDN rule or a robots.txt group that applies to GPTBot can only be observed by asking as GPTBot, so that is what the matrix does, and every row says which agent string was sent.
What does a timeout verdict mean?
The request did not finish inside the budget for that mode. It is reported as a timeout rather than a block, because those have different causes and different fixes.