Googlebot

Operator

Google

User agent

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36

Robots token

Googlebot

Obeys robots.txt

Yes, per the operator

Verification

Reverse DNS lookup. Reverse DNS must resolve to googlebot.com, google.com or googleusercontent.com, and a forward lookup on that hostname must return the original IP. Google also publishes CIDR ranges as JSON.

Operator docs

https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers

Google runs two Search crawlers under one robots.txt token. The desktop string:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36

The smartphone string, which does the large majority of the crawling since mobile-first indexing:

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

W.X.Y.Z is a placeholder in Google's own documentation, not a literal. It tracks the current Chrome release, so any rule you write that pins a version number will break on the next Chrome bump. Match the substring Googlebot instead. Google also documents two rarer short forms that carry the same token.

The robots.txt token is Googlebot.

Who operates it and what it feeds

Google. The fetches build the Search index, and everything Google derives from that index reads the same crawled copy. That includes the AI surfaces in Search. There is no separate opt-out token for AI Overviews, which is the single most common misreading of the Google crawler docs. Google-Extended covers Gemini training and grounding, and Google states plainly that it does not affect Search inclusion.

Does it honor robots.txt

Yes. Googlebot is the reference implementation of the robots.txt protocol, and Google open-sourced its parser. Two caveats matter in practice.

A Disallow rule blocks the fetch, not the indexing. Google can still list a disallowed URL from anchor text on other sites, with no snippet. If you want a page gone, let Googlebot fetch it and return noindex, either as a meta tag or an X-Robots-Tag header. The two directives fight each other: a disallowed page's noindex is never read.

Googlebot also ignores Crawl-delay. Rate control lives in Search Console, or in your own 429 and 503 responses, which Googlebot does back off from.

How to verify a request

User agent strings are free to type, so treat the string as a claim and the IP as the evidence. Google documents two checks.

Manual, for one address in a log:

host 66.249.66.1
# expect a hostname ending in googlebot.com, google.com or googleusercontent.com
host crawl-66-249-66-1.googlebot.com
# expect the original IP back

Both directions have to agree. A reverse lookup alone can be forged by anyone who controls the PTR record for their own address.

At scale, match against the published CIDR list at https://developers.google.com/static/crawling/ipranges/common-crawlers.json. Google publishes separate files for special-case crawlers and for user-triggered fetchers, so pick the file that matches the token you are checking rather than the general goog.json, which covers all Google egress.

Allow it

Googlebot is allowed by default. An empty or missing robots.txt permits everything, and you only need an explicit block if a broader rule already catches it:

User-agent: Googlebot
Allow: /

Block it

User-agent: Googlebot
Disallow: /

What blocking costs you

Everything. Organic Google traffic stops, and the decay is slow enough to be confusing: existing index entries persist for days or weeks, so the graph in Search Console falls after the decision rather than with it. Rich results, Discover and the Search-derived AI surfaces all go with it, because they read the same index.

There is one narrow case where a partial block earns its keep. Faceted navigation, internal search results and session-parameter URLs can absorb most of a site's crawl budget without ever being worth indexing. Disallowing those path patterns is a real fix. Blanket-blocking Googlebot is not a tuning decision.

Watch out for the accidental version. A staging robots.txt shipped to production, a WAF rule that returns 403 to anything without a browser-shaped header, or a CDN rule keyed on an old Chrome version will all block Googlebot while your robots.txt says otherwise. Check the server response, not the file.

Find it in your logs

awk '$0 ~ /Googlebot/ {print $1}' access.log | sort | uniq -c | sort -rn | head

Take the top addresses and run the two DNS lookups on each. On a site with any traffic at all, expect a meaningful share to fail the check. Googlebot is the most impersonated agent string on the web, because so many sites allow it unconditionally and a surprising number serve it different content, which is cloaking and is treated as a violation when Google notices.

Check your site against Googlebot