robots.txt

Also called robots file, robots exclusion protocol, REP

A plain text file at the root of a host, https://example.com/robots.txt, listing which crawlers may request which paths. It is a request, not a control. Compliant crawlers read it and honor it; nothing stops one that does not, which is why Bytespider can fetch the file and ignore it.

Rules are grouped by User-agent. A crawler reads the single most specific group whose token matches its own identifier, applies those rules, and ignores every other group in the file.

User-agent: *
Disallow: /admin/

User-agent: GPTBot
Disallow: /

The mistake

Two, and they are the same mistake in different clothes.

The first is expecting Disallow to deindex. It does not. The rule prevents the fetch, so a noindex directive on the blocked page is never read and the URL can still appear as a bare link. To remove a page, allow the crawl and serve noindex.

The second is assuming groups combine. They do not. In the file above, GPTBot reads only its own group, so the /admin/ rule does not apply to it. Add a named group for one crawler and it silently stops obeying everything in the wildcard group. Whatever you want a named crawler to follow has to be repeated inside its group.

One more thing worth checking rather than assuming: what the file returns. Bot mitigation that serves a 403 to non-browser agents, or a 503 during a deploy, means your rules are not being read at all. Fetch it yourself from outside your network, with a crawler's user agent, and look at the status code before you trust the contents.

For AI training preferences specifically, the tokens are per-operator. See Google-Extended, Applebot-Extended and llms.txt.