MarketLensIs your robots.txt still written for a world where “crawler” meant Googlebot and Bingbot? Most are. Meanwhile, a dozen AI user agents with completely different jobs — training collectors, answer-engine indexers, on-demand page fetchers — are reading that same file and applying whatever blanket rule they find, or no rule at all.
The stakes cut both ways. An overly permissive file donates your entire archive to model training you may never have agreed to. An overly aggressive one — the reflexive Disallow: / for anything with “AI” in the name — quietly removes you from ChatGPT Search and Perplexity citations, which is traffic real buyers arrive from. Getting this right requires knowing what each user agent actually does, because the correct policy is almost never the same for all of them.
Here is the complete agent-by-agent map, a decision framework, and a production-ready file you can adapt in ten minutes.
What Does Each AI User Agent Actually Do?
The single most common robots.txt mistake is treating one company as one bot. Every major operator runs multiple agents with separate functions and separate controls:
| User Agent | Operator | Function | Blocking It Means |
|---|---|---|---|
| GPTBot | OpenAI | Training data collection | Content excluded from future GPT training |
| OAI-SearchBot | OpenAI | ChatGPT Search index | No ChatGPT Search citations |
| ChatGPT-User | OpenAI | User-triggered live fetch | ChatGPT can’t open your URLs on request |
| ClaudeBot | Anthropic | Training data collection | Content excluded from Claude training |
| Claude-User | Anthropic | User-triggered live fetch | Claude can’t open your URLs on request |
| PerplexityBot | Perplexity | Answer-engine index | No Perplexity citations |
| Google-Extended | Gemini training control token | Opts out of Gemini training; Search unaffected | |
| CCBot | Common Crawl | Open web archive | Excluded from Common Crawl (feeds many labs) |
Note what is not on the list: Googlebot. Google AI Overviews draw on the ordinary Search index, so there is no robots.txt lever that removes you from AI Overviews without also removing you from Search. Google-Extended only governs Gemini training and grounding.
Which Policy Fits Your Business?
Three coherent positions, in descending order of openness:
1. Full participation. Allow everything. Right for brands whose economics depend on being known — B2B SaaS, consultancies, local services — where a training-data appearance in next year’s models is marketing, not theft. If discovery matters more to you than control of the corpus, staying maximally visible is the coherent position.
2. Citations yes, training no. Allow OAI-SearchBot, PerplexityBot, and the user-fetch agents; disallow GPTBot, ClaudeBot, CCBot, and Google-Extended. Right for publishers whose content is the product. You keep the referral and citation surface while withholding the training corpus.
3. Full withdrawal. Disallow all of the above. Coherent only for paywalled or licensing-first businesses — and remember it removes you from answer engines your buyers increasingly use, a trade-off worth weighing against what the Pew Research browsing study found about click behaviour on AI-summarised results.
The middle position fits most content businesses. Here it is as a complete file:
# --- Search engines: unrestricted ---
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /
# --- AI answer engines: allowed (citations wanted) ---
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-User
Allow: /
# --- AI training collectors: refused ---
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
Sitemap: https://www.example.com/sitemap.xmlSpelling and casing matter — GPT-Bot or Claudebot matches nothing. And keep Bingbot allowed if you want ChatGPT Search visibility: OpenAI’s search experience leans heavily on Bing’s index, so a Bingbot block is a de facto ChatGPT Search block regardless of your OAI-SearchBot rule.
How Do You Know the Rules Are Working?
Robots.txt is a request. Verification is a log exercise:
- Confirm compliant bots obey. After deploying, watch your access logs for a week. Disallowed agents should stop appearing on content paths (a robots.txt fetch itself is fine and expected).
- Authenticate the user agents you do see. OpenAI and Anthropic publish official crawler IP ranges as JSON. Any “GPTBot” arriving from a residential proxy is a scraper wearing a costume — robots.txt was never going to stop it, and it belongs in your firewall rules instead. Refusing compliant bots and repelling hostile ones are different jobs, and the second one belongs at the edge — see our guide to Cloudflare WAF rules and edge caching.
- Watch for new agents quarterly. The user-agent roster changes several times a year. A quarterly log scan for unrecognized bot-like agents, plus a check of each operator’s crawler documentation page, keeps the file current.
Beyond Permission: Make the Allowed Fetches Count
Once you have decided who gets in, two adjacent moves multiply the value of every permitted crawl. First, performance: allowed bots still abandon slow pages, and the timing budgets involved overlap almost exactly with the ones covered in our companion piece on Core Web Vitals and mobile page speed. Second, format: pointing agents at a curated plain-text index via the llms.txt standard hands them your best content pre-flattened, instead of leaving them to infer structure from your templates.
Ship It This Week
Pull your current robots.txt, diff it against the agent table above, and write an explicit rule for every row — silence is a policy too, just an accidental one. Then verify in logs after seven days. The whole exercise costs an hour and settles a question most competitors are still answering by default.