Free tools

AI crawler robots.txt generator

Generate robots.txt rules for common AI crawlers. Robots directives are voluntary instructions, not authentication or a firewall. Keep human visitors and ordinary search crawlers in mind when choosing rules.

Training and search are different controls

GPTBot is separate from OAI-SearchBot and ChatGPT-User. Google-Extended is a robots control token, not a distinct user-agent string. CCBot collects an open corpus that downstream projects may reuse; it does not identify a particular model trainer. Bytespider has documented search uses and is excluded from the training-only group. Crawler operators may change behavior, so review the linked official guidance in the crawler directory before publishing rules.

Keep exploring

Crawlers

GPTBot: user agent, robots.txt, and verification

Understand GPTBot user agents, independent robots.txt training controls, official IP verification, and the limits of AnalyticFast crawler counts.

Read guide Crawlers

Google-Extended: user agent, robots.txt, and verification

Google-Extended is a robots.txt usage control, not a distinct HTTP crawler. Learn what it controls and why a User-Agent match cannot identify it.

Read guide