Making your site agent-ready with Cloudflare
Published · Last updated
If your domain runs through Cloudflare, its settings can overrule your robots.txt, so a site that says "crawlers welcome" can still turn them away at the door. Being agent-ready means checking Cloudflare first and robots.txt second.
Why it matters
Cloudflare sits in front of your site and answers every request before your pages do. Bot protection, AI-crawler blocking and a managed robots.txt are all switched on from its dashboard, often by default or by a single toggle.
Three places to look
- Bot protection settings: some are on by default and can challenge legitimate crawlers.
- AI crawler controls: blocking AI training bots can have side effects on the search crawlers you want.
- Managed robots.txt: Cloudflare can serve its own version in place of the file you deployed.
Decide on purpose
Some owners want AI assistants to cite them, some want to opt out of training, and some want both. Each is a valid choice, but it has to be a choice you made rather than a default you inherited.
What this does not tell you
The exact rules to set, the order to apply them in, and how to let real crawlers through without opening the door to impersonators. A wrong rule here can block Google as easily as it blocks an AI bot.