CCBot

Operator

Common Crawl Foundation

User agent

CCBot/2.0 (https://commoncrawl.org/faq/)

Robots token

CCBot

Obeys robots.txt

Yes, per the operator

Verification

Reverse DNS lookup. Reverse DNS resolves to a hostname of the form [dashed-ip].crawl.commoncrawl.org, and Common Crawl recommends forward-confirmed reverse DNS. The same ranges are published as JSON at https://index.commoncrawl.org/ccbot.json.

Operator docs

https://commoncrawl.org/ccbot

The shortest crawler string on this list:

CCBot/2.0 (https://commoncrawl.org/faq/)

No browser preamble, no Chrome version, nothing to go stale. The robots.txt token is CCBot.

Who operates it and what it feeds

The Common Crawl Foundation, a nonprofit that has published a free web archive since 2008. The corpus is the raw material behind a large amount of academic work on the web, and it is also read by commercial model training pipelines, which is why a bot that predates the current argument by fifteen years now shows up in AI blocking guides.

That downstream reach is the thing to understand before you decide. CCBot crawls once. What reads the archive afterwards is out of Common Crawl's hands and out of yours.

Does it honor robots.txt

Yes, and it has a long record of doing so. Common Crawl also runs an opt-out registry for content already in the archive, which is a removal path rather than a crawl rule. A Disallow stops future fetches and leaves existing captures alone, so if your concern is content already published, the registry is the part you want.

How to verify a request

Common Crawl documents forward-confirmed reverse DNS:

host 18.97.14.84
# 18-97-14-84.crawl.commoncrawl.org
host 18-97-14-84.crawl.commoncrawl.org
# expect 18.97.14.84 back

The same ranges are published as JSON at https://index.commoncrawl.org/ccbot.json, which carries both IPv4 and IPv6 prefixes and a note recommending the DNS check for IPv4 addresses. Having both a DNS method and a range file puts CCBot in better shape than most AI-era crawlers, which publish one or neither.

Allow it

User-agent: CCBot
Allow: /

Block it

User-agent: CCBot
Disallow: /

What blocking costs you

Directly, almost nothing. Common Crawl sends no referral traffic and shows no results. Nobody arrives at your site from the archive.

Indirectly, you leave a widely used research dataset, and you remove one input to an unknown number of training pipelines. That second effect is easy to overstate. Blocking CCBot does not block GPTBot, ClaudeBot, Amazonbot or Bytespider, all of which crawl your site themselves and each of which needs its own rule. A robots.txt with CCBot disallowed and nothing else is a common and largely symbolic configuration.

There is also a citation argument in the other direction. Content in the archive is part of the corpus that models learn the shape of a topic from, and some site owners want to be in that corpus for exactly the reason others want out. Neither position is measurable, which is worth saying out loud rather than dressing either one up.

Crawl volume

CCBot fetches broadly and shallowly, and its passes are periodic rather than continuous. If it is showing up heavily in your logs, check whether your crawl budget is being spent on a large generated URL space, and whether those URLs are returning real 404 responses or soft 404 pages that look like content to every crawler that sees them.

Find it in your logs

awk '$0 ~ /CCBot/ {print $1}' access.log | sort | uniq -c | sort -rn | head

Run the DNS pair on each address. CCBot is verifiable in both directions, so an address that fails both the lookup and the range file is straightforwardly not Common Crawl, and you can treat it as ordinary unwanted traffic.

Reading the archive instead of your logs

Common Crawl publishes an index, which means you can check what the archive actually holds for your domain rather than inferring it from crawl volume. That is unusual. For every other operator on this list the corpus is private and you are guessing.

It is worth doing once before you decide anything. Sites often find that the archive holds far less of them than expected, because the crawl is broad rather than deep, or that it holds a large number of near-identical faceted URLs that never should have been crawlable in the first place. The second finding is a crawl budget problem with every crawler, not a Common Crawl problem, and fixing it improves your position with Googlebot at the same time.

A caveat on timing

Robots.txt changes apply to future passes. Common Crawl runs periodic crawls rather than continuous ones, so the interval between your edit and any observable change is measured in weeks. Do not read the first week of unchanged behavior as non-compliance.

Check your site against CCBot