This is a reference, not an essay. Pick the policy that matches your decision, paste it, change the paths, and test.

First, know who's knocking

AI user agents fall into two groups. The distinction matters, because "block training" and "stay out of AI answers" are different decisions.

Agents that fetch pages for AI answers

Agents that collect training data

Two notes before you copy anything:

• Googlebot is separate. Google Search, AI Overviews and AI Mode use Googlebot. Blocking Google-Extended doesn't remove you from Search, but it can keep your pages out of Gemini's answers, because the token covers grounding as well as training.
• User-triggered fetchers may not obey robots.txt. Some operators say agents like Perplexity-User act on a person's behalf and may ignore the file. If you need a hard block, do it at your CDN or firewall.

Template 1: Welcome every AI crawler

For brands that want to be read, cited and recommended everywhere.

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /account/
Disallow: /checkout/

Sitemap: https://www.example.com/sitemap.xml

That's genuinely enough, as long as no named group exists for an AI crawler further down. The moment you add one, see the mistakes section below.

Template 2: Yes to AI answers, no to training

The most common choice for publishers and SaaS companies: be cited, but don't donate your content to training sets.

# AI answer and search agents: allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: DuckAssistBot
Allow: /
Disallow: /admin/
Disallow: /account/
Disallow: /checkout/

# Training crawlers: blocked
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: CCBot
User-agent: Bytespider
Disallow: /

# Everyone else, including Googlebot
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /account/
Disallow: /checkout/

Sitemap: https://www.example.com/sitemap.xml

Stacking several User-agent lines over one set of rules is valid under the robots.txt standard (RFC 9309), and it keeps the file readable.

Google-Extended is deliberately left out. Add it to the blocked group only if you accept losing Gemini answers along with Gemini training.

Template 3: Block all AI, keep Google Search

For sites with licensing concerns or paid content.

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: DuckAssistBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: CCBot
User-agent: Bytespider
Disallow: /

User-agent: *
Allow: /
Disallow: /admin/

Sitemap: https://www.example.com/sitemap.xml

Remember the caveat above: this states your wishes. Enforcement happens at the edge, so pair it with your CDN's bot rules.

Template 4: Section by section

Sometimes the answer is "docs yes, everything else no", for example to let assistants help your users without exposing your blog archive.

User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: Claude-SearchBot
Allow: /docs/
Allow: /pricing
Disallow: /

User-agent: *
Allow: /
Disallow: /admin/

Under RFC 9309 the most specific (longest) matching rule wins, so Allow: /docs/ beats Disallow: / for anything under /docs/.

The mistakes that quietly undo all of this

1. Named groups drop your wildcard rules. A crawler that finds a group with its own name follows only that group and ignores User-agent: * entirely. Add User-agent: GPTBot / Allow: / to be friendly, and every Disallow under * stops applying to GPTBot. Repeat your private paths in every named group, as the templates above do.

2. Leftover blanket blocks. Lots of sites copied User-agent: GPTBot / Disallow: / in 2023. Fine if you still mean it. But check that nobody extended the same block to OAI-SearchBot or ChatGPT-User, which removes you from ChatGPT's answers rather than just its training data.

3. Typos fail silently. GPT-Bot, Claudebot-User, Perplexitybot/1.0: a token that matches no crawler is ignored without error. Copy names from the operators' docs, or from a list you trust.

4. Assuming robots.txt is a wall. It's a request. Your CDN or firewall decides what a crawler actually receives, and it can block an agent your file allows, or let through one your file forbids.

5. Forgetting to test. After any change, request a page as each crawler and compare the result with a normal browser request.

Tools to save some typing

If you'd rather tick boxes than hand-edit, I help build Presence Scout, and we made a free robots.txt generator for AI crawlers with these same presets. It runs in your browser, no sign-up. Then paste a page into the free AI crawler checker to see what each crawler actually receives.

The list of AI user agents changes every few months. Whatever you pick, put a reminder in your calendar to review it each quarter.

Mayank is the Digital Marketing Manager at Presence Scout, which tracks how brands appear in AI answers and Google search.