ClaudeBot

Operator

Anthropic

User agent

ClaudeBot

Robots token

ClaudeBot

Obeys robots.txt

Yes, per the operator

Verification

Published IP ranges. Anthropic publishes the IP ranges for its bots as JSON at https://claude.com/crawling/bots.json. No reverse DNS hostname is documented, and Anthropic warns that IP blocking can stop the bot reading robots.txt at all.

Operator docs

https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler

Anthropic documents three bots, each with its own robots.txt token:

ClaudeBot         model development
Claude-User       fetches a page a user asked about
Claude-SearchBot  indexes content for search results

Anthropic publishes the product token rather than a full HTTP user agent string. That distinction matters more than it sounds. Bot directories across the web print a full-looking ClaudeBot string with a contact address in it, and none of those strings come from Anthropic's documentation. Write your rules against the ClaudeBot substring and you are matching the part the operator actually commits to.

Does it honor robots.txt

Anthropic states that its bots honor standard robots.txt directives, respect anti-circumvention measures, and do not attempt to bypass CAPTCHAs. It also supports the non-standard Crawl-delay extension, which puts ClaudeBot in the small group of large crawlers that read it. Bingbot is the other one worth knowing.

How to verify a request

Match the source IP against https://claude.com/crawling/bots.json, a CIDR list with a creationTime field. There is no documented reverse DNS hostname, so this file is the only check.

Anthropic adds a caveat that most operators leave out, and it is a good one: blocking by IP can be counterproductive, because a bot that cannot reach your server also cannot read the robots.txt that tells it to leave. If your intent is "do not crawl me", say so in the file the crawler reads. Save the firewall for traffic that ignores the file.

Allow it

User-agent: ClaudeBot
Allow: /

Rate-limited instead of blocked:

User-agent: ClaudeBot
Crawl-delay: 1
Allow: /

Block it

User-agent: ClaudeBot
Disallow: /

Block model development, keep user-directed fetches and search:

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

What blocking costs you

Blocking ClaudeBot alone costs you nothing you can measure today. It affects future training corpora, which carry no attribution and send no traffic.

Blocking all three is a different decision. Claude-User fires when a person is actively looking at your page, and Claude-SearchBot is how you appear in Claude's search results with a link attached. Those are referral surfaces. A blanket User-agent: Claude prefix match, or a firewall rule on the word Claude, takes out all three at once, which is usually not what the person writing it meant.

Getting the rule right

Robots.txt tokens are matched by prefix against the crawler's own identifier, and the most specific matching group wins. ClaudeBot and Claude-User are distinct tokens, not variations, so each needs its own group. A single group listing several User-agent lines applies the same rules to all of them, which is fine when that is the intent and a trap when it is not.

If your robots.txt is already long, check what it actually serves before editing it. A 403 on /robots.txt from a WAF rule, or a 503 during a deploy, is read as "rules unavailable" rather than "no rules", and behavior during that window is not what your file says. See robots.txt for how fetch failures are interpreted, and GPTBot for the same training-versus-retrieval split at OpenAI.

Find it in your logs

awk '$0 ~ /ClaudeBot/ {print $1}' access.log | sort | uniq -c | sort -rn | head

Compare those addresses against bots.json. Anything outside the file is not Anthropic, whatever the string says, and it is the part of the traffic your robots.txt was never going to influence.

A note on the three tokens in one file

Robots.txt groups are matched by the most specific token, not by how many groups mention a crawler. A file with a ClaudeBot group and a separate * group gives ClaudeBot only the rules in its own group, and the wildcard group is ignored entirely for it. This trips people up when the wildcard carries the real restrictions and the named group was added later to allow one path.

If you want ClaudeBot to obey your general rules plus one extra, the extra rules have to be repeated inside the ClaudeBot group. There is no inheritance in the format. The same rule applies to GPTBot, Bingbot and every other named token, and it is the single most common way a carefully written robots.txt does something other than what its author intended.

Check your site against ClaudeBot