---
name: ai-crawler-robots-txt
description: Audit and rewrite a site's robots.txt so the AI and search crawlers the owner wants can reach the site. Use when a user asks why ChatGPT, Claude, Perplexity or Gemini do not cite their site, asks to allow or block AI bots, or shares a robots.txt file.
---

# AI crawler robots.txt

## Goal
Make robots.txt allow every crawler the owner wants, block only the crawlers the owner does not want, and stay valid.

## Facts to use
- robots.txt lives at the root of each host, for example https://example.com/robots.txt. A file in a subfolder has no effect.
- Google uses only one group per crawler: the group with the most specific `User-agent` that matches. Rules from the `*` group do not add to a specific group. If no group matches, the crawler follows `User-agent: *`.
- Path values are case-sensitive. A `Disallow:` line with no path has no effect.
- robots.txt applies only to the host, protocol and port where it is served.
- robots.txt controls crawling, not indexing. To keep a page out of search results, use a `noindex` meta tag or header and let the page be crawled.
- Search and training are different tokens for some companies. Check each owner's current docs before you change a rule:
  - OpenAI: `OAI-SearchBot` (ChatGPT search), `GPTBot` (training), `ChatGPT-User` (actions a user starts; OpenAI says robots.txt rules may not apply to it). Each setting is independent: a GPTBot block does not change ChatGPT search.
  - Anthropic: `Claude-SearchBot` (search), `ClaudeBot` (training), `Claude-User` (actions a user starts).
  - Perplexity: `PerplexityBot` (search results), `Perplexity-User` (actions a user starts; Perplexity says it generally ignores robots.txt).
  - Google: `Googlebot` (Search). `Google-Extended` controls whether Google may use your content to train Gemini models. Google says it does not affect inclusion or ranking in Google Search.
  - Apple: `Applebot` (Spotlight, Siri and Safari search). `Applebot-Extended` opts content out of Apple model training. It does not crawl pages, and a block does not remove the site from search.
  - Microsoft: `Bingbot` (Bing). Copilot fetches web results from Bing.
  - Common Crawl: `CCBot`.

## Workflow
1. Fetch `/robots.txt` for every host the user names (apex and www). Record the HTTP status. For Google, a 5xx status stops crawling of the site for the first 12 hours, then Google uses the last good copy for up to 30 days. A 4xx status (except 429) means "no rules".
2. List every `User-agent` group and its rules. Note groups that block `/` for a search crawler.
3. Ask the owner one question if it is not clear: "Do you want AI assistants to cite your pages, to train on them, both, or neither?"
4. Write the new file:
   - Allow the search tokens (`OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot`, `Googlebot`, `Bingbot`, `Applebot`) if the owner wants citations.
   - Add training tokens (`GPTBot`, `ClaudeBot`, `Google-Extended`, `Applebot-Extended`, `CCBot`) as allow or disallow from the owner's answer.
   - Keep existing rules for admin, cart, search result and API paths.
   - Add one `Sitemap:` line per sitemap, with an absolute URL.
5. Check that the site firewall or CDN does not block the same bots (see the cloudflare-ai-crawler-check skill). A 403 from the firewall overrides an allow in robots.txt.

## Checks
- [ ] The file returns status 200 and `text/plain`.
- [ ] No group blocks `/` for a crawler the owner wants.
- [ ] Every `Sitemap:` URL returns 200.
- [ ] Google's robots.txt report in Search Console shows no errors.

## Output
Return the full new robots.txt in a code block, then a table: crawler, owner, purpose, old rule, new rule.

## Sources
- Google: How Google interprets the robots.txt specification. https://developers.google.com/search/reference/robots_txt
- Google: List of Google's common crawlers (Google-Extended). https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers
- OpenAI: Overview of OpenAI crawlers. https://developers.openai.com/api/docs/bots
- Anthropic: Does Anthropic crawl data from the web. https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
- Perplexity: Perplexity crawlers. https://docs.perplexity.ai/docs/resources/perplexity-crawlers
- Apple: About Applebot. https://support.apple.com/en-us/119829
- Microsoft: Web search in Microsoft Copilot. https://learn.microsoft.com/microsoft-365-copilot/manage-public-web-access
