---
name: cloudflare-ai-crawler-check
description: Check whether Cloudflare settings block or challenge AI and search crawlers, and read which bots visit which pages. Use when a site is on Cloudflare and AI assistants do not cite it, or when a user asks to see AI bot traffic.
---

# Cloudflare AI crawler check

## Goal
Find Cloudflare settings that stop crawlers the owner wants, and show which AI bots reach the site.

## Facts to use
- A firewall block or a challenge page stops a crawler even when robots.txt allows it. Crawlers do not solve challenges.
- Cloudflare has AI Crawler and Agent access policies (formerly the Block AI bots setting), with an action per behavior: Search, Agent and Training. AI Crawl Control lets you allow or block each AI crawler. Cloudflare can also manage robots.txt for the zone. Check the dashboard for the current names.
- Bot Fight Mode, Super Bot Fight Mode, WAF custom rules, rate limiting rules and "Under Attack" mode can all challenge or block bots.
- Cloudflare marks known good bots as verified bots. The AI access policies apply to verified bots too, not only to unverified bots. Rules that act on "not verified" traffic can still hit bots that are not on the list.

## Workflow
1. Ask the owner which crawlers they want: search and citation only, or training too.
2. In the Cloudflare dashboard for the zone, read:
   - The AI bot block setting and the AI crawler page (which crawlers are allowed or blocked).
   - Security > Bots: Bot Fight Mode or Super Bot Fight Mode state.
   - Security > WAF: custom rules and rate limiting rules that match user agents, ASNs or countries.
   - Whether Cloudflare manages robots.txt for the zone, and what it adds.
3. Test from outside: request a page with each crawler user agent (see the crawler-access-test skill). A 403, a 503 or a challenge page body means a block. Note that some rules check IP ranges too, so a user agent test can differ from the real bot.
4. Read the crawler activity report: requests per bot, top paths, status codes. Look for bots that only get 403 or 429.
5. Change only what the owner approves. Prefer an allow rule for the wanted bots over turning off bot protection for the whole zone.

## Checks
- [ ] Each wanted crawler gets status 200 on the home page and on a deep page.
- [ ] No challenge HTML appears in the crawler responses.
- [ ] robots.txt served by Cloudflare matches what the owner wants.

## Output
A table: crawler, Cloudflare setting that affects it, status code seen, change made or proposed.

## Sources
- Cloudflare: AI Crawler and Agent access policies. https://developers.cloudflare.com/bots/additional-configurations/block-ai-bots
- Cloudflare: Manage AI crawlers (AI Crawl Control). https://developers.cloudflare.com/ai-crawl-control/features/manage-ai-crawlers
- Cloudflare: Managed robots.txt. https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt
- Cloudflare: Bots concepts. https://developers.cloudflare.com/bots/concepts/bot
