Crawlers

ClaudeBot: user agent, robots.txt, and verification

ClaudeBot is Anthropic’s training crawler. Claude-SearchBot improves search indexing, while Claude-User retrieves pages at a user’s direction. These names describe different purposes.

User agent and purpose

Documented purpose: Training.

ClaudeBot (documented crawler token; no full request string is asserted here)

Anthropic’s official help page identifies ClaudeBot, Claude-SearchBot, and Claude-User. Match the exact purpose-specific token instead of treating every Claude request as training.

Robots.txt control

User-agent: ClaudeBot
Disallow: /

Anthropic documents robots.txt opt-outs and support for Crawl-delay. Apply the rules to each subdomain you intend to restrict. Disallowing ClaudeBot alone is a training choice; use separate groups for its other bots.

Review this example in the context of your existing groups and paths. Publish it at the origin’s /robots.txt and check the actual response. A policy change can take time to be read; it does not erase earlier copies of content.

Verify origin before trusting the name

Anthropic now links an official crawler IP list. Compare the connecting address with that live list. This supports an Anthropic network-origin check; a User-Agent alone does not authenticate ClaudeBot. No official rDNS suffix is asserted here.

A User-Agent is self-reported and can be spoofed. A matching header is a claimed identity, not an authenticated bot. Use the operator’s current IP or DNS procedure for origin confidence; those checks are not cryptographic request signatures and do not authorize access to private data.

What AnalyticFast detects today

AnalyticFast does not currently classify ClaudeBot in its named crawler aggregate. This directory entry is research, not a claim of detection support. AnalyticFast does not authenticate bot identities or perform the IP/rDNS checks described here.

A useful next check

Choose separately whether to permit training, search indexing, and user-directed retrieval. When investigating excessive traffic, distinguish an observed ClaudeBot header from a request whose source address matches the operator’s list.

Official sources and review date

• Anthropic crawler controls and verification source

Last checked 2026-10-31. Verify the latest policy and network data before changing production access rules.

Back to the crawler directory.