Crawlers

OAI-SearchBot: user agent, robots.txt, and verification

OAI-SearchBot helps surface pages in ChatGPT search. It is distinct from the GPTBot training crawler and ChatGPT-User user-directed retrieval. A crawl is an opportunity for discovery, not a guaranteed citation.

User agent and purpose

Documented purpose: Search.

OAI-SearchBot/1.4 (documented token; version can change)

Check the current request formats in OpenAI’s reference; the token describes a search purpose.

Robots.txt control

User-agent: OAI-SearchBot
Allow: /

This permits the search bot for all paths within its group. Add any necessary exclusions to that group before publishing. A separate GPTBot restriction can express a different training preference.

Review this example in the context of your existing groups and paths. Publish it at the origin’s /robots.txt and check the actual response. A policy change can take time to be read; it does not erase earlier copies of content.

Verify origin before trusting the name

Compare the source address with the official searchbot feed. Matching a name or a different OpenAI bot’s address list is not enough.

A User-Agent is self-reported and can be spoofed. A matching header is a claimed identity, not an authenticated bot. Use the operator’s current IP or DNS procedure for origin confidence; those checks are not cryptographic request signatures and do not authorize access to private data.

What AnalyticFast detects today

AnalyticFast currently recognizes the self-reported OAI-SearchBot token on valid analytics ingest requests. Opt-in crawler aggregates count those requests only; they do not automatically observe every page fetch on your site. AnalyticFast does not authenticate bot identities or perform the IP/rDNS checks described here.

A useful next check

Keep the robots file, landing page, and essential content reachable without login when search discovery is intended. Measure subsequent human referrals separately; indexing activity itself does not establish customer acquisition.

Official sources and review date

• Official OpenAI search crawler reference

Last checked 2026-10-31. Verify the latest policy and network data before changing production access rules.

Back to the crawler directory.