robots.txt for the AI era: copy-paste templates for GPTBot, ClaudeBot, PerplexityBot and Google-ExtendedFour ready-to-use policies, a reference table of AI user agents, and the mistakes that quietly undo them.Oct 12, 2026 · 6 min read