Free tool · runs in your browser
robots.txt generator for AI crawlers.
Choose which AI crawlers you let in: the ones behind linked answers in ChatGPT, Claude and Perplexity, and the ones that collect data for model training. The generator builds a ready robots.txt. Nothing is sent anywhere; it all runs in your browser.
Which crawler does what
| Crawler | Purpose | Group |
|---|---|---|
OAI-SearchBot | ChatGPT search | search |
ChatGPT-User | ChatGPT opening a page a user asked for | search |
Claude-SearchBot | Claude search | search |
Claude-User | Claude opening a page a user asked for | search |
PerplexityBot | Perplexity search index | search |
Perplexity-User | Perplexity opening a page a user asked for | search |
GPTBot | OpenAI model training | training |
ClaudeBot | Anthropic model training | training |
Google-Extended | Gemini (training and answers in Gemini apps); does not affect Google Search or AI Overviews | training |
Applebot-Extended | Apple model training | training |
CCBot | Common Crawl, an open dataset used to train many models | training |
Meta-ExternalAgent | Meta model training | training |
Bytespider | ByteDance (TikTok) | training |
Crawler names from the documentation of OpenAI, Anthropic, Perplexity, Google, Apple, Common Crawl, Meta and ByteDance (as of October 2026). Companies add new crawlers, so check back every few months.
The “search yes, training no” version
# robots.txt generated at aiagentbuilder.pl/en/tools/robots-txt-generator/ # All crawlers (search engines and others) User-agent: * Disallow: /admin/ Allow: / # AI crawlers you let in (they also get the paths blocked for everyone) User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: Claude-SearchBot User-agent: Claude-User User-agent: PerplexityBot User-agent: Perplexity-User Disallow: /admin/ Allow: / # AI crawlers you block User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: CCBot User-agent: Meta-ExternalAgent User-agent: Bytespider Disallow: /
After you upload the file
- Open yourdomain.com/robots.txt and check you see the new version (you may need to clear the hosting or Cloudflare cache).
- In Google Search Console, open the robots.txt report (Settings): it shows syntax errors.
- Run the page through our AI visibility check: it shows whether your firewall lets AI crawlers in too, because robots.txt does not override blocks in e.g. Cloudflare.
- Add an llms.txt file listing your key pages.
Frequently asked questions
Should I block AI crawlers?
If you want AI assistants to recommend your business, do not block the search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot): they decide whether your pages can appear in answers with a link. Blocking training crawlers (GPTBot, ClaudeBot, Google-Extended) is a separate decision: it protects your content from model training, but models may then know less about your business.
Does blocking GPTBot remove me from ChatGPT?
No. GPTBot collects training data. ChatGPT search uses OAI-SearchBot. You can block GPTBot and keep OAI-SearchBot, and your pages can still appear in ChatGPT answers.
Does Google-Extended affect my Google rankings?
No. Google says Google-Extended does not affect Google Search rankings or inclusion. AI Overviews in Google Search are controlled by the regular Googlebot, so blocking Google-Extended does not remove you from them.
Why are my blocked paths repeated for AI crawlers?
A crawler that has its own User-agent group ignores the “*” group. If you only block /admin/ under “*”, a bot with its own group would still crawl it. That is why the generator copies your blocked paths into the group of AI crawlers you let in.
Do all crawlers respect robots.txt?
Major companies document that their crawlers do, but robots.txt is a request, not a lock. Some providers say that user-initiated fetches (ChatGPT-User, Perplexity-User) may not follow it. If you need a hard block, use firewall rules.
Contact
Tell us what eats your team's time.
Two or three sentences are enough: what your business does and which process you would hand over first. For fixed-price packages, use Fiverr or Contra.