> why_wont_ai_
Scan

robots.txt generator

The robots.txt generator is a free tool that builds a robots.txt file for search and AI crawlers. Pick which groups of bots may read your site, add private paths and your sitemap, then copy the file.

Bot groups
Search and answers
Fetches when a user asks
Model training
Change one bot
Googlebot (Google, search and answers)
Bingbot (Microsoft, search and answers)
OAI-SearchBot (OpenAI, search and answers)
ChatGPT-User (OpenAI, fetches when a user asks)
GPTBot (OpenAI, model training)
Claude-SearchBot (Anthropic, search and answers)
Claude-User (Anthropic, fetches when a user asks)
ClaudeBot (Anthropic, model training)
PerplexityBot (Perplexity, search and answers)
Perplexity-User (Perplexity, fetches when a user asks)
Applebot (Apple, search and answers)
Amazonbot (Amazon, search and answers)
CCBot (Common Crawl, model training)
Meta-ExternalAgent (Meta, model training)
Bytespider (ByteDance, model training)
Google-Extended (Google, training-use switch)
Applebot-Extended (Apple, training-use switch)

The file is built in your browser. Nothing is sent or stored.

Check your current file

See which of 17 crawlers your live robots.txt allows, and which rule decides each one.

Open the robots.txt checker

Check what AI crawlers get

Tags and files only help when crawlers can read the page. The free scan requests your pages as GPTBot, ClaudeBot, PerplexityBot and CCBot and shows what arrives.

Run the free scan All free tools

Questions

Should I block AI crawlers in robots.txt?

Block the crawlers you do not want, but know what each one does. Search and answer bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) decide whether AI search can show your pages. Training bots (GPTBot, ClaudeBot, CCBot) collect text for future models. Many sites allow the first group and block the second.

Does blocking GPTBot remove my site from ChatGPT search?

No. OpenAI uses OAI-SearchBot for ChatGPT search and GPTBot for training. Block GPTBot to opt out of training and keep OAI-SearchBot allowed to stay in search results.

What do Google-Extended and Applebot-Extended do?

They are not crawlers. They are tokens that tell Google and Apple whether they may use pages they already crawl for AI training. Blocking them does not change normal search.

Where do I put robots.txt?

At the root of your domain, for example https://yoursite.com/robots.txt, as a plain text file. On single-page apps, make sure the server returns the file and not the app's HTML page.