seoder

What crawlers get from your pages

Ten search and AI crawlers, each with its user agent string, its robots.txt token, how to verify a request really came from it, and what blocking it costs. Plus the files and status codes they act on, written from operator documentation and dated so you can see when it was last checked.

What is on this site

Six subjects, each with the part people get wrong written down rather than implied. Where a claim comes from an operator's own reference, the page says which one.

  • Crawler pages, one per crawler, with the agent string, the robots token and the verification method the operator publishes.
  • How robots.txt matching works, including the group selection rule that decides which lines a crawler reads at all.
  • What rendering costs you, and which crawlers never do it.
  • Redirect chains, what each status code claims, and the chains that quietly stack up.
  • llms.txt, what the format asks for and what it does not control.
  • Status codes read the way a crawler reads them, where a 503 and a 403 lead to very different outcomes.

The crawlers documented here

Each page carries the crawler's user agent string, its robots token, the verification method its operator publishes, and what you give up by blocking it.

  • Amazonbot

    Crawling to improve Amazon products and services, including answers surfaced to customers, and it may be used to train Amazon AI models.

  • Applebot-Extended

    Controlling whether content already crawled by Applebot may be used to train Apple's generative foundation models.

  • Bingbot

    Crawling for the Bing index, which also backs Copilot answers and several licensed search products.

  • Bytespider

    Collecting web content for ByteDance. The operator publishes no statement of purpose, so any description of what it feeds is inference.

  • CCBot

    Building the free public Common Crawl archive, which many research projects and several model training pipelines read downstream.

  • ClaudeBot

    Collecting public web content for model development. Search and user-directed retrieval run under separate tokens.

  • Google-Extended

    Controlling whether crawled content may be used for training Gemini models and for grounding in Gemini Apps and Vertex AI.

  • Googlebot

    Crawling for Google Search indexing and the Search-derived surfaces built on that index.

  • GPTBot

    Collecting web content used to train OpenAI's foundation models. It does not drive ChatGPT search citations.

  • Meta-ExternalAgent

    Crawling for use cases such as training foundation AI models and improving products by indexing content directly.

  • PerplexityBot

    Indexing pages so they can appear as cited sources in Perplexity answers. Perplexity states it does not collect content for model training.

Why the pages are checkable

Every user agent string, robots token and verification method here is copied from the operator's own reference, and the page names it. Where an operator publishes nothing, as ByteDance does for Bytespider, the page says so rather than repeating a string somebody pulled out of a server log. Pages carry a date, because this moves.

Nothing here fetches your site. It is a reference, so how a crawler resolves a page is where to start when a result surprises you, and the resource listings cover the software that does the fetching, with what each one stops being good at.

Common questions

Does a Disallow rule keep a page out of search results?

No. Disallow stops the fetch. A URL other pages link to can still be listed, with no description under it, because the crawler is not allowed to read the noindex you put on the page. Allow the crawl and serve noindex instead.

Do AI crawlers run JavaScript?

Most do not. They read the HTML the server returns and stop there, so a page whose text arrives through client-side fetches reads as close to empty to them even when robots.txt allows the crawl. Google renders on a second pass with its own queue.

Why does one crawler get a 403 when robots.txt allows it?

Because the block is in front of your application. A CDN or firewall rule keyed on the user agent string answers before your server sees the request, so nothing in robots.txt is wrong and nothing in your site changed. The rule is where to look.

Can I tell a real Googlebot request from something wearing the name?

For some operators, yes. Google and Bing verify through reverse DNS plus a forward lookup back to the same address. OpenAI, Anthropic and Perplexity publish IP range files instead. ByteDance publishes nothing, so Bytespider traffic is unverifiable in both directions.