AnalyticCrawlers
AI and search crawler directory: user agents, robots rules, and verification
Crawler activity tells you which automated clients requested content. Human referrals tell you who arrived from a search or assistant link. A matched payment adds another observation. Keep these signals separate when judging whether AI discovery is helping your business.
Choose a crawler
| Name | Purpose or control | Runtime classification |
|---|---|---|
| GPTBot | Training | Self-reported UA on valid ingest requests |
| ClaudeBot | Training | Not in the named aggregate |
| PerplexityBot | Search | Not in the named aggregate |
| Google-Extended | Usage control token | Not in the named aggregate |
| Bytespider | Toutiao search | Not in the named aggregate |
| Baiduspider | Search | Not in the named aggregate |
| MistralAI-User | User-requested retrieval | Not in the named aggregate |
| Kimi | Training, search, and user retrieval | Not in the named aggregate |
| Applebot | Search with separate usage preferences | Not in the named aggregate |
| CCBot | Open web corpus | Not in the named aggregate |
| Googlebot | Search | Self-reported UA on valid ingest requests |
| Bingbot | Search | Self-reported UA on valid ingest requests |
| OAI-SearchBot | Search | Self-reported UA on valid ingest requests |
| ChatGPT-User | User-requested retrieval | Self-reported UA on valid ingest requests |
What a user-agent check can prove
A User-Agent is self-reported and can be spoofed. A matching header is a claimed identity, not an authenticated bot. Use the operator’s current IP or DNS procedure for origin confidence; those checks are not cryptographic request signatures and do not authorize access to private data.
For an IP feed, match the connecting address against the current CIDRs for the correct bot. For a documented DNS procedure, reverse-resolve the address, validate the exact domain boundary, and forward-resolve the hostname back to the same address. A proxy must preserve trustworthy network information; a client-supplied forwarding header is not sufficient.
Robots preferences and access control
Publish robots.txt on the relevant origin, preserve existing rules, and distinguish training from indexing and live user retrieval. Some user-requested fetchers have different behavior. Robots.txt is a public preference file, not a password or a way to keep a private URL secret.
Google-Extended has no separate request User-Agent. Applebot-Extended also controls use rather than making independent crawl requests. A header checker cannot measure either preference by searching for a corresponding HTTP request.
What AnalyticFast currently measures
The named classifier recognizes ChatGPT-User, OAI-SearchBot, GPTBot, Googlebot, and Bingbot. Opt-in aggregates count matching valid analytics ingest requests, not all requests to customer pages. Most page-fetching crawlers need not run a browser tracker. Use your server or CDN logs for a complete request inventory; this directory does not install server logging or expand the classifier.
Use the analytics demo to explore visitor and revenue reporting, and installation documentation to understand the data you collect. Allowing a crawler never guarantees indexing, citation, referral traffic, or a purchase.
Primary references
- OpenAI official crawler reference
- Anthropic official crawler reference
- Perplexity official crawler reference
- Google official crawler reference
- Apple official crawler reference
- Kimi official crawler reference
- Common Crawl official crawler reference
Research checked . The operator’s current instructions take precedence over this snapshot.
Questions, answered.
Does a crawler visit mean an AI assistant cited my site?
No. A fetch, an index entry, an answer citation, a human referral, and a payment are different observations. Measure them separately.
Does AnalyticFast verify all the crawlers in this directory?
No. Its current named aggregate uses five self-reported user-agent patterns on valid analytics ingest requests. It performs no bot IP or reverse-DNS authentication, and directory coverage does not imply detection support.