About seoder

A reference for one question that is annoyingly hard to answer: what does a crawler actually get when it asks for this page?

Why this exists

Every few months someone finds out that half their catalogue was invisible to search for a year, and the cause is a two line robots.txt rule or a price that only appears after a script runs. The facts needed to catch that early are scattered across an operator's documentation here, a blog post from 2019 there, and a forum thread that was right at the time. This site collects them in one place, per crawler and per file, and dates each page.

The crawler pages are the part we could not find anywhere reliable. Search crawlers and AI crawlers get different treatment from CDNs, firewalls and robots rules, often without the site owner knowing, and most bot directories publish user agent strings copied out of server logs rather than out of an operator's reference. Where an operator publishes nothing, the page says so instead of filling the gap with a confident sentence.

How a claim gets onto a page

Every user agent string, robots token and verification method here comes from the operator's own documentation, and the page names it. Where no such documentation exists, as with Bytespider, the page says that is the most useful fact about the crawler and stops there. Where a number comes from a crawl, the output is published alongside it so you can check the arithmetic.

Pages carry a date because crawler behavior moves. An operator adds a published IP range, a token changes, a documented cache window gets revised. A page with an old date is a page to verify, not a page to trust.

Who writes it

The team at Spider writes seoder. Spider sells a crawling API and maintains open source crawlers, which is why we had these facts lying around and why we care whether they are right.

That also means some entries on the resource listings are our own products. Each one says so under its name, and links to them are marked sponsored. They get the same treatment as every other entry: what it does, what it suits, and what it does not suit, written down next to it. No list here names one product as the one to use. Nothing on the site requires an account, nothing is gated, and there is nothing to buy on it.

Getting in touch

If something here is wrong or has gone stale, say which page and what the operator documentation says now. Corrections that come with a source get made the same week.