Robots.txt Generator (with AI crawler rules)
Create a robots.txt with separate rules for search engines, AI search bots and AI training crawlers like GPTBot, ClaudeBot and Google-Extended.
- Free
- No sign-up
- Runs in your browser
- SEO & Content
About the Robots.txt Generator (with AI crawler rules)
Choose what to allow for regular search engines, AI search/answer engines and AI model-training crawlers, add your sitemap, and copy the file.
How it works
Rules are grouped by user-agent. AI training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended, Meta-ExternalAgent, Bytespider) are listed separately from AI search bots (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot), so you can block training while staying visible in AI answers.
Frequently asked questions
Does blocking GPTBot remove me from ChatGPT answers?
No. GPTBot is OpenAI’s training crawler. ChatGPT search uses OAI-SearchBot and ChatGPT-User, which have their own rules here.
Is robots.txt enforced?
It is a voluntary standard. Reputable crawlers follow it; it is not a security control.