Crawlers

CCBot: user agent, robots.txt, and verification

CCBot collects pages for Common Crawl’s publicly accessible web corpus. That corpus can be used by researchers and other organizations; a CCBot request does not identify a particular downstream model or company using the data.

User agent and purpose

Documented purpose: Open web corpus.

CCBot/2.0 (https://commoncrawl.org/faq/)

Common Crawl’s official CCBot page documents this user agent and warns about impersonation.

Robots.txt control

User-agent: CCBot
Disallow: /

Common Crawl publishes this whole-site opt-out example. Treat it as a preference for future fetching; do not assume changing robots.txt erases previously distributed crawl data or changes every downstream copy.

Review this example in the context of your existing groups and paths. Publish it at the origin’s /robots.txt and check the actual response. A policy change can take time to be read; it does not erase earlier copies of content.

Verify origin before trusting the name

Use the official IPv4/IPv6 feed. For IPv4, the official page demonstrates reverse DNS under .crawl.commoncrawl.org followed by a forward lookup. It says reverse DNS is not yet supported over IPv6, so use the published prefixes for that case.

A User-Agent is self-reported and can be spoofed. A matching header is a claimed identity, not an authenticated bot. Use the operator’s current IP or DNS procedure for origin confidence; those checks are not cryptographic request signatures and do not authorize access to private data.

What AnalyticFast detects today

AnalyticFast does not currently classify CCBot in its named crawler aggregate. This directory entry is research, not a claim of detection support. AnalyticFast does not authenticate bot identities or perform the IP/rDNS checks described here.

A useful next check

If you want to opt out, check the published robots file on each origin and review Common Crawl’s current removal or opt-out information. If you allow crawling, keep those requests separate from visits by prospective customers.

Official sources and review date

• Official CCBot identity, opt-out, and verification

Last checked 2026-10-31. Verify the latest policy and network data before changing production access rules.

Back to the crawler directory.