AnalyticCrawlers
Google-Extended: user agent, robots.txt, and verification
Google-Extended is a robots.txt product token governing certain uses of content Google has crawled. It has no separate HTTP request User-Agent. Do not create a Google-Extended request count by searching server logs for that name.
User agent and purpose
Documented purpose: Usage control token.
No distinct HTTP User-Agent exists for Google-Extended.Google’s official Google-Extended section separates the robots control from the existing Google request user agents that perform crawling.
Robots.txt control
User-agent: Google-Extended
Disallow: /Google documents this token for future Gemini model training and specified Gemini/Vertex AI grounding uses. It does not affect inclusion or ranking in Google Search. This example changes the usage preference; it is not a network-level block on Googlebot.
Review this example in the context of your existing groups and paths. Publish it at the origin’s /robots.txt and check the actual response. A policy change can take time to be read; it does not erase earlier copies of content.
Verify origin before trusting the name
There is no Google-Extended-specific IP identity to authenticate. Verify the actual Google crawler using Google’s crawler verification instructions. Even a verified Googlebot request does not reveal which downstream use of the fetched content will occur.
A User-Agent is self-reported and can be spoofed. A matching header is a claimed identity, not an authenticated bot. Use the operator’s current IP or DNS procedure for origin confidence; those checks are not cryptographic request signatures and do not authorize access to private data.
What AnalyticFast detects today
AnalyticFast does not currently classify Google-Extended in its named crawler aggregate. This directory entry is research, not a claim of detection support. AnalyticFast does not authenticate bot identities or perform the IP/rDNS checks described here.
A useful next check
Keep your Search crawl policy and content-use policy in separate groups. To check implementation, inspect the published robots file rather than simulating a fictitious Google-Extended header. A header checker can report a string match, but cannot establish this usage preference.
Official sources and review date
Last checked . Verify the latest policy and network data before changing production access rules.
Back to the crawler directory.
Questions, answered.
Can I detect Google-Extended in an HTTP User-Agent?
No. Google says Google-Extended has no separate request User-Agent. AnalyticFast correctly returns no crawler classification for that token.