Blog
Every post that carries a number links to the crawl it came from, so you can download the data and check it.
-
What robots.txt does not tell you
A robots.txt file states an intention. What a crawler receives is a separate fact, and the two come apart on sites where every other check looks clean.