GPTBot

Operator

OpenAI

User agent

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot

Robots token

GPTBot

Obeys robots.txt

Yes, per the operator

Verification

Published IP ranges. OpenAI publishes GPTBot's CIDR ranges as JSON at https://openai.com/gptbot.json, with separate files for OAI-SearchBot and ChatGPT-User. There is no documented reverse DNS check.

Operator docs

https://developers.openai.com/api/docs/bots

OpenAI's training crawler:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot

The robots.txt token is GPTBot.

That version number has already moved once, so any rule pinned to GPTBot/1.4 will quietly stop matching. Match the substring GPTBot.

Three OpenAI crawlers, three decisions

This is where most GPTBot advice goes wrong. OpenAI documents three separate agents, each with its own token and its own published range file:

GPTBot collects training data. OAI-SearchBot builds the index behind ChatGPT's search results and citations. ChatGPT-User fetches a page because a user asked for that page, usually by pasting a link.

Blocking GPTBot stops training use. It does not remove you from ChatGPT answers, and it does not stop a user-initiated fetch. Sites that want the traffic but not the training keep OAI-SearchBot allowed and disallow GPTBot, which is a coherent position and the most common one.

Does it honor robots.txt

OpenAI documents robots.txt support for all three agents and publishes the tokens for exactly that purpose. Expect a delay between the edit and the behavior change, because the crawler caches robots.txt rather than refetching it per request.

How to verify a request

Match the source IP against https://openai.com/gptbot.json, a JSON file of CIDR prefixes with a creationTime field. The companion files are https://openai.com/searchbot.json and https://openai.com/chatgpt-user.json.

OpenAI publishes no reverse DNS hostname, so the range file is the whole verification story. Anything claiming to be GPTBot from an address outside that file is something else wearing the name, and there is a lot of it: GPTBot is one of the most commonly spoofed agent strings, because sites tend to allow it without checking.

Allow it

User-agent: GPTBot
Allow: /

Block it

User-agent: GPTBot
Disallow: /

To block training while keeping search citations:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

What blocking costs you

Less than people assume, and the honest answer is that nobody can measure it well. Blocking GPTBot affects whether your content is in a future training corpus. It does not affect today's model, which was trained on a snapshot taken before your edit, and it does not affect retrieval-time citations.

The real cost is optionality. Training-corpus inclusion is not attributed and not linked, so it sends no traffic and builds no measurable brand. Against that, once content is in a corpus there is no removal mechanism. Reasonable people land in different places on that trade, and the one indefensible position is blocking GPTBot while believing it removes you from ChatGPT.

If you want to state preferences in a machine-readable file rather than only in access rules, llms.txt is the proposal people reach for, though no major operator has committed to reading it.

The enforcement gap

Disallow is a request. Every honest operator honors it and dishonest traffic ignores it, which is the entire design of robots.txt. If you need enforcement rather than a preference, the rule has to live where the request is served: match the published ranges at the edge and return 403 to anything else claiming the name. Compare with ClaudeBot, which publishes ranges the same way, and with Bytespider, which publishes nothing at all.

Find it in your logs

awk '$0 ~ /GPTBot/ {print $1}' access.log | sort | uniq -c | sort -rn | head

Check the addresses against gptbot.json before drawing conclusions. In practice a large fraction of GPTBot-labelled traffic on small sites is unverifiable, and treating it as OpenAI's volume will make you reach for a rule that does not apply to it.

What GPTBot sees

OpenAI documents no JavaScript execution for GPTBot. Plan for a fetch that reads the HTML your server returns and nothing more, which is the same assumption you should make for most non-search crawlers. If your content arrives through client-side hydration or a framework that renders after load, GPTBot's copy of your page is the empty shell.

That cuts both ways. A client-rendered site is effectively opted out of GPTBot by accident, and the owners usually have no idea. If inclusion is what you want, prerendering or server rendering is the fix, and the check is to compare the raw HTML against the rendered DOM. See JavaScript rendering for what that comparison tells you.

Check your site against GPTBot