Applebot-Extended

Operator

Apple

User agent

None. Applebot-Extended does not crawl webpages and sends no requests of its own.

Robots token

Applebot-Extended

Obeys robots.txt

Yes, per the operator

Verification

No published method. Applebot-Extended issues no requests, so there is nothing to verify. Verify Applebot instead: reverse DNS resolves under applebot.apple.com, and Apple publishes CIDR ranges at https://search.developer.apple.com/applebot.json.

Operator docs

https://support.apple.com/en-us/119829

Applebot-Extended has no user agent string, because it is not a crawler. Apple states it plainly:

Applebot-Extended does not crawl webpages.

It is a robots.txt token and nothing else. Adding it to your file does not stop a fetch, because no fetch is attributed to it. What it changes is permission: whether content Applebot has already collected may be used to train Apple's generative foundation models.

Every page that prints an "Applebot-Extended user agent string" invented it. There is nothing to match in a log, no IP range to check, and no request to block at the edge.

The pair it belongs to

Applebot is the crawler. Apple documents both forms:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)

Mozilla/5.0 (iPhone; CPU iPhone OS 17_4_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Mobile/15E148 Safari/604.1 (Applebot/0.1; +http://www.apple.com/go/applebot)

Those fetches feed Spotlight, Siri and Safari suggestions. Applebot-Extended then decides what may be done with the result.

The split is the useful part of Apple's design, and it gives you an option most operators do not offer: stay in Apple's search surfaces, opt out of model training, in two lines of robots.txt. Google-Extended works on exactly the same principle for Gemini.

Does it honor robots.txt

The question inverts here. Applebot-Extended is a robots.txt instruction, so the thing honoring it is Apple's internal data handling rather than a crawler. There is no fetch to allow or deny, and no way to observe compliance from your own logs. You are taking a documented policy statement on trust, which is worth stating clearly rather than implying a technical control exists.

How to verify

There is nothing to verify for Applebot-Extended. For Applebot:

host 17.58.101.179
# 17-58-101-179.applebot.apple.com
host 17-58-101-179.applebot.apple.com
# expect 17.58.101.179 back

Apple also publishes CIDR prefixes as JSON at https://search.developer.apple.com/applebot.json, refreshed regularly. Both methods are documented, which puts Applebot alongside Googlebot and CCBot rather than the range-only operators.

Opt out of AI training, stay in search

User-agent: Applebot-Extended
Disallow: /

Opt in

Say nothing. Absence of a rule is permission, which is how robots.txt has always worked. If you want the intent recorded for a future reader of the file, an explicit allow is legal and harmless:

User-agent: Applebot-Extended
Allow: /

Block the crawler too

User-agent: Applebot
Disallow: /

What blocking costs you

Disallowing Applebot-Extended costs you nothing observable. No traffic depends on training-corpus inclusion, and Apple states that search inclusion is unaffected.

Disallowing Applebot is the expensive one. Siri answers, Spotlight results and Safari suggestions all read what Applebot collected, and those surfaces reach a large iOS audience that never opens a search engine. People conflate the two rules constantly, usually because a blog post told them to "block Applebot for AI", and the result is a site that has quietly left Siri.

Check the live file rather than the copy in your repository before you trust either rule. A 403 on /robots.txt from a bot-mitigation product is common, and a robots.txt that never loads carries none of your intent.

Find Applebot in your logs

awk '$0 ~ /Applebot/ {print $1}' access.log | sort | uniq -c | sort -rn | head

You will never see Applebot-Extended here. If something in your logs carries that string, it is not Apple, because Apple states the agent does not crawl.

What Applebot reads

Apple documents JavaScript execution for Applebot, which puts it in a small group with Googlebot. Most crawlers on this list read the HTML your server returns and stop there. That still does not make client rendering safe: execution is subject to timeouts and resource limits, and a page whose content depends on a slow request after load can be captured empty. Compare the raw HTML against the rendered DOM if you want to know what is actually visible, which is what a JavaScript rendering check measures.

Check your site against Applebot-Extended