Crawl access

AI crawler robots.txt

Audit and rewrite a site's robots.txt so the AI and search crawlers the owner wants can reach the site. Use when a user asks why ChatGPT, Claude, Perplexity or Gemini do not cite their site, asks to allow or block AI bots, or shares a robots.txt file.

ai-crawler-robots-txt/SKILL.md

Goal

Make robots.txt allow every crawler the owner wants, block only the crawlers the owner does not want, and stay valid.

Facts to use

  • robots.txt lives at the root of each host, for example https://example.com/robots.txt. A file in a subfolder has no effect.
  • Google uses only one group per crawler: the group with the most specific User-agent that matches. Rules from the * group do not add to a specific group. If no group matches, the crawler follows User-agent: *.
  • Path values are case-sensitive. A Disallow: line with no path has no effect.
  • robots.txt applies only to the host, protocol and port where it is served.
  • robots.txt controls crawling, not indexing. To keep a page out of search results, use a noindex meta tag or header and let the page be crawled.
  • Search and training are different tokens for some companies. Check each owner's current docs before you change a rule:
    • OpenAI: OAI-SearchBot (ChatGPT search), GPTBot (training), ChatGPT-User (actions a user starts; OpenAI says robots.txt rules may not apply to it). Each setting is independent: a GPTBot block does not change ChatGPT search.
    • Anthropic: Claude-SearchBot (search), ClaudeBot (training), Claude-User (actions a user starts).
    • Perplexity: PerplexityBot (search results), Perplexity-User (actions a user starts; Perplexity says it generally ignores robots.txt).
    • Google: Googlebot (Search). Google-Extended controls whether Google may use your content to train Gemini models. Google says it does not affect inclusion or ranking in Google Search.
    • Apple: Applebot (Spotlight, Siri and Safari search). Applebot-Extended opts content out of Apple model training. It does not crawl pages, and a block does not remove the site from search.
    • Microsoft: Bingbot (Bing). Copilot fetches web results from Bing.
    • Common Crawl: CCBot.

Workflow

  1. Fetch /robots.txt for every host the user names (apex and www). Record the HTTP status. For Google, a 5xx status stops crawling of the site for the first 12 hours, then Google uses the last good copy for up to 30 days. A 4xx status (except 429) means "no rules".
  2. List every User-agent group and its rules. Note groups that block / for a search crawler.
  3. Ask the owner one question if it is not clear: "Do you want AI assistants to cite your pages, to train on them, both, or neither?"
  4. Write the new file:
    • Allow the search tokens (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot, Applebot) if the owner wants citations.
    • Add training tokens (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot) as allow or disallow from the owner's answer.
    • Keep existing rules for admin, cart, search result and API paths.
    • Add one Sitemap: line per sitemap, with an absolute URL.
  5. Check that the site firewall or CDN does not block the same bots (see the cloudflare-ai-crawler-check skill). A 403 from the firewall overrides an allow in robots.txt.

Checks

  • The file returns status 200 and text/plain.
  • No group blocks / for a crawler the owner wants.
  • Every Sitemap: URL returns 200.
  • Google's robots.txt report in Search Console shows no errors.

Output

Return the full new robots.txt in a code block, then a table: crawler, owner, purpose, old rule, new rule.

Sources

  • Google: How Google interprets the robots.txt specification. https://developers.google.com/search/reference/robots_txt
  • Google: List of Google's common crawlers (Google-Extended). https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers
  • OpenAI: Overview of OpenAI crawlers. https://developers.openai.com/api/docs/bots
  • Anthropic: Does Anthropic crawl data from the web. https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
  • Perplexity: Perplexity crawlers. https://docs.perplexity.ai/docs/resources/perplexity-crawlers
  • Apple: About Applebot. https://support.apple.com/en-us/119829
  • Microsoft: Web search in Microsoft Copilot. https://learn.microsoft.com/microsoft-365-copilot/manage-public-web-access
See all 30 skills