Crawlers

Bytespider: user agent, robots.txt, and verification

Toutiao’s webmaster documentation identifies Bytespider as its search crawler and explains discovery, indexing, and retrieval. That official page does not establish an AI-training purpose for every request carrying this name.

User agent and purpose

Documented purpose: Toutiao search.

Mozilla/5.0 (compatible; Bytespider; https://zhanzhang.toutiao.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.0.0 Safari/537.36

Toutiao’s official Bytespider article (Chinese) provides desktop and mobile examples. The browser version is an example, not an authentication signal.

Robots.txt control

User-agent: Bytespider
Disallow: /

This expresses a whole-site crawl restriction. Toutiao publishes robots matching guidance with its own query-parameter rules. The documentation is not a guarantee that every request claiming Bytespider will comply. Use access controls when you require enforcement.

Review this example in the context of your existing groups and paths. Publish it at the origin’s /robots.txt and check the actual response. A policy change can take time to be read; it does not erase earlier copies of content.

Verify origin before trusting the name

The official article lists ten IPv4 /24 ranges and describes reverse DNS under .bytedance.com. Recheck the source instead of copying a permanent allowlist. We recommend also forward-resolving the returned hostname and requiring the original address; that additional check is our operational recommendation.

A User-Agent is self-reported and can be spoofed. A matching header is a claimed identity, not an authenticated bot. Use the operator’s current IP or DNS procedure for origin confidence; those checks are not cryptographic request signatures and do not authorize access to private data.

What AnalyticFast detects today

AnalyticFast does not currently classify Bytespider in its named crawler aggregate. This directory entry is research, not a claim of detection support. AnalyticFast does not authenticate bot identities or perform the IP/rDNS checks described here.

A useful next check

Compare the exact user agent and source network with the first-party article. If they differ, retain an unverified label. Do not infer TikTok distribution, AI training, or a specific downstream model from the Bytespider token alone.

Official sources and review date

Last checked 2026-10-31. Verify the latest policy and network data before changing production access rules.

Back to the crawler directory.

Questions, answered.

Is Bytespider officially documented only as an AI-training bot?

No. The official Toutiao article describes a search crawler. It does not verify the training purpose often assigned to this name by third-party bot directories.