Google-Extended

Operator

Google

User agent

None. Google-Extended has no separate HTTP user agent string; crawling happens under existing Google agents.

Robots token

Google-Extended

Obeys robots.txt

Yes, per the operator

Verification

No published method. Google-Extended sends no requests, so no verification applies. Verify Googlebot instead, with forward-confirmed reverse DNS under googlebot.com or against Google's published CIDR JSON files.

Operator docs

https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers

Google-Extended sends no requests. Google's own wording:

Google-Extended doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity.

So there is no string to match, no IP range to check, and nothing that will ever appear in a log file. The token is a permission setting that Google reads from your robots.txt and applies to content its crawlers already fetched.

What it controls

Whether content crawled from your site may be used to train Gemini models, and for grounding in Gemini Apps and Vertex AI.

Google also states the boundary directly: Google-Extended does not impact a site's inclusion in Google Search, and it is not used as a ranking signal. That sentence exists because publishers worried that opting out of AI training would cost them rankings. It does not.

What it does not control

This is where the confusion concentrates. AI surfaces built on the Search index read the Search index, which Googlebot fills. Disallowing Google-Extended does not remove you from them. The only robots.txt rule that governs Search crawling is the one on the Googlebot token, and that rule costs you organic Search traffic as well.

There is no partial opt-out for Search-derived AI features. A lot of published advice implies one exists. It does not, and pretending otherwise leads people to ship a rule that achieves nothing they wanted.

Does it honor robots.txt

The token is a robots.txt instruction, so the thing honoring it is Google's downstream data handling. You cannot observe compliance from your logs, because there is no traffic to observe. That is a policy commitment rather than a technical control, and it is worth naming as one.

Opt out of Gemini training and grounding

User-agent: Google-Extended
Disallow: /

Opt in

Leave it out of the file. No rule means no restriction, per the robots.txt default. An explicit allow is valid if you want the decision visible to whoever edits the file next:

User-agent: Google-Extended
Allow: /

Partial control

The token takes ordinary path rules, so you can open documentation and close everything else:

User-agent: Google-Extended
Disallow: /
Allow: /docs/

What opting out costs you

Nothing measurable, and the honest version of that sentence has two halves. Gemini training inclusion sends no traffic and carries no attribution, so there is no number to lose. Grounding is a different matter: a grounded answer can cite and link a source, and opting out removes you from that pool along with the training corpus. Google bundles both behaviors under one token, so you cannot separate them.

Compare Applebot-Extended, which splits the same way for Apple Intelligence, and GPTBot, where OpenAI splits training and search across two distinct crawlers instead of one control token.

Checking that it applied

There is no positive signal. You cannot confirm from logs, from Search Console, or from any Google report that the token was read, because no request corresponds to it. What you can check is that the rule is served at all. Fetch your own robots.txt from outside your network and confirm it returns a 200 with the group intact, rather than a 403 from a bot-mitigation rule a 404, or a stale cached copy.

A rule that is easy to write wrong

The token is Google-Extended exactly. It is not a prefix of Googlebot and it is not matched by a Google group, because robots.txt tokens are matched against the crawler's identifier rather than by loose substring on your side. A group written as User-agent: Google matches nothing Google runs.

Watch the interaction with your wildcard group too. Robots.txt has no inheritance: if a Google-Extended group exists, Google-Extended reads that group and ignores the * group entirely. A file whose real restrictions live under *, with a narrow Google-Extended group added later, has quietly given the token more access than the wildcard allowed. The same trap catches ClaudeBot and every other named token.

Check your site against Google-Extended