Bytespider

Operator

ByteDance

User agent

Bytespider (ByteDance publishes no user agent specification; match the token substring, not a full string)

Robots token

Bytespider

Obeys robots.txt

Not documented

Verification

No published method. ByteDance publishes no IP ranges, no ASN list and no reverse DNS method. There is no way to confirm that a request claiming to be Bytespider came from ByteDance.

Every other page on this site starts with a documented string copied from an operator's own reference. This one cannot, and the reason is the most useful fact about Bytespider.

ByteDance publishes no crawler documentation, no user agent specification, no IP address list and no verification method. The strings you find in bot directories were read out of other people's access logs. They vary, they change without notice, and none of them carry an operator's commitment.

So the only thing to match is the token:

Bytespider

Case-insensitive substring, applied to the User-Agent header. Log-derived variants include lowercase spellings and strings referencing zhanzhang.toutiao.com, ByteDance's webmaster platform, which hosts no crawler documentation reachable from its front page.

What it feeds

Unknown, in the sense that matters. ByteDance operates search and model products, and Bytespider is widely reported as the crawler collecting content for them. No operator statement confirms this, so anything more specific is inference dressed as fact, including the confident sentences you will find elsewhere.

That gap is not a footnote. Without a published purpose there is no opt-out that means anything, no distinction between training and retrieval like GPTBot offers, and no control token like Google-Extended.

Does it honor robots.txt

Assume not. Several independent analyses of server logs report the same pattern: Bytespider fetches /robots.txt, then requests URLs listed under Disallow. ByteDance has made no public commitment to the protocol and offers no contact route for reporting a violation.

This is the practical difference between Bytespider and the rest of the list. Robots.txt has always been a request rather than a control, and it works because large operators choose to honor it. When one does not, the file is a note nobody reads.

How to verify a request

There is no method. Nothing is published, so a request claiming to be Bytespider cannot be confirmed as ByteDance, and a request from ByteDance cannot be distinguished from an imitator.

Compare Meta-ExternalAgent, which has the same gap but at least documents its purpose and its robots.txt handling. Bytespider is missing all three.

The robots.txt rule, for completeness

User-agent: Bytespider
Disallow: /

Write it. It costs nothing and it records your intent for anything that does read the file. Do not expect it to change your traffic.

The rule that works

Enforcement has to happen where the request is served.

nginx:

if ($http_user_agent ~* "bytespider") {
    return 403;
}

Apache:

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} bytespider [NC]
RewriteRule ^ - [F,L]

Cloudflare and similar products can express the same thing as a WAF expression on http.user_agent. A 403 is the right response: it is cheap, it is final, and it does not invite the retries that a 429 or a 503 does.

A user agent match is defeated by changing the user agent, which is a real limitation and not a reason to skip it. It stops the traffic that identifies itself honestly, which is most of the volume, and the remainder becomes a rate-limiting problem rather than an identity problem.

What blocking costs you

Very little that you can name. ByteDance products send no attributed referral traffic to most sites, and with no published purpose there is no specific surface you are opting out of.

Against that, Bytespider is one of the heaviest crawlers by request volume on many sites. If your origin is thin or your time to first byte is already marginal, the load is a concrete cost you can measure today, unlike the benefit.

One check before you write the rule: confirm the traffic is what it says. Since nothing is verifiable, some share of any Bytespider volume in your logs is other software wearing the name. That does not change the response, since you are blocking the string either way, but it does change what you conclude about ByteDance from a graph.

Check your site against Bytespider