ai-crawler-robots-txt/SKILL.md
Goal
Make robots.txt allow every crawler the owner wants, block only the crawlers the owner does not want, and stay valid.
Facts to use
- robots.txt lives at the root of each host, for example https://example.com/robots.txt. A file in a subfolder has no effect.
- Google uses only one group per crawler: the group with the most specific
User-agentthat matches. Rules from the*group do not add to a specific group. If no group matches, the crawler followsUser-agent: *. - Path values are case-sensitive. A
Disallow:line with no path has no effect. - robots.txt applies only to the host, protocol and port where it is served.
- robots.txt controls crawling, not indexing. To keep a page out of search results, use a
noindexmeta tag or header and let the page be crawled. - Search and training are different tokens for some companies. Check each owner's current docs before you change a rule:
- OpenAI:
OAI-SearchBot(ChatGPT search),GPTBot(training),ChatGPT-User(actions a user starts; OpenAI says robots.txt rules may not apply to it). Each setting is independent: a GPTBot block does not change ChatGPT search. - Anthropic:
Claude-SearchBot(search),ClaudeBot(training),Claude-User(actions a user starts). - Perplexity:
PerplexityBot(search results),Perplexity-User(actions a user starts; Perplexity says it generally ignores robots.txt). - Google:
Googlebot(Search).Google-Extendedcontrols whether Google may use your content to train Gemini models. Google says it does not affect inclusion or ranking in Google Search. - Apple:
Applebot(Spotlight, Siri and Safari search).Applebot-Extendedopts content out of Apple model training. It does not crawl pages, and a block does not remove the site from search. - Microsoft:
Bingbot(Bing). Copilot fetches web results from Bing. - Common Crawl:
CCBot.
- OpenAI:
Workflow
- Fetch
/robots.txtfor every host the user names (apex and www). Record the HTTP status. For Google, a 5xx status stops crawling of the site for the first 12 hours, then Google uses the last good copy for up to 30 days. A 4xx status (except 429) means "no rules". - List every
User-agentgroup and its rules. Note groups that block/for a search crawler. - Ask the owner one question if it is not clear: "Do you want AI assistants to cite your pages, to train on them, both, or neither?"
- Write the new file:
- Allow the search tokens (
OAI-SearchBot,Claude-SearchBot,PerplexityBot,Googlebot,Bingbot,Applebot) if the owner wants citations. - Add training tokens (
GPTBot,ClaudeBot,Google-Extended,Applebot-Extended,CCBot) as allow or disallow from the owner's answer. - Keep existing rules for admin, cart, search result and API paths.
- Add one
Sitemap:line per sitemap, with an absolute URL.
- Allow the search tokens (
- Check that the site firewall or CDN does not block the same bots (see the cloudflare-ai-crawler-check skill). A 403 from the firewall overrides an allow in robots.txt.
Checks
- The file returns status 200 and
text/plain. - No group blocks
/for a crawler the owner wants. - Every
Sitemap:URL returns 200. - Google's robots.txt report in Search Console shows no errors.
Output
Return the full new robots.txt in a code block, then a table: crawler, owner, purpose, old rule, new rule.
Sources
- Google: How Google interprets the robots.txt specification. https://developers.google.com/search/reference/robots_txt
- Google: List of Google's common crawlers (Google-Extended). https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers
- OpenAI: Overview of OpenAI crawlers. https://developers.openai.com/api/docs/bots
- Anthropic: Does Anthropic crawl data from the web. https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
- Perplexity: Perplexity crawlers. https://docs.perplexity.ai/docs/resources/perplexity-crawlers
- Apple: About Applebot. https://support.apple.com/en-us/119829
- Microsoft: Web search in Microsoft Copilot. https://learn.microsoft.com/microsoft-365-copilot/manage-public-web-access