On this page
We read the robots.txt file of 130 scanned websites. We checked the rules for 6 crawlers: Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot and CCBot. 127 sites (98%) had a robots.txt file that we could read. Only 3 sites (2%) blocked an AI crawler from the whole site. No site blocked Googlebot or Bingbot from the whole site.
This page is about the rules in robots.txt only. Our study of what AI crawlers get from these sites is a different question. It found that rendering, not robots.txt, was the main reason that AI crawlers got no text.
Whole-site blocks by crawler
| Crawler | What it does | Blocks the whole site | Does not block the whole site |
|---|---|---|---|
| Googlebot | Google Search | 0 | 127 |
| Bingbot | Bing search | 0 | 127 |
| GPTBot | OpenAI model training | 2 | 125 |
| ClaudeBot | Anthropic model training | 2 | 125 |
| PerplexityBot | Perplexity search and answers | 0 | 127 |
| CCBot | Common Crawl data set | 3 | 124 |
The counts use the 127 sites with a readable robots.txt file. A whole-site block is a Disallow rule for "/" that applies to that crawler. All three groups are under 10 sites, so read them with care.
What else we found
- 3 sites (2%) had no readable robots.txt file. One returned 404. Two refused the request with 401 or 403.
- No site blocked all six crawlers. So no robots.txt in the sample shut out every crawler.
- 2 sites blocked GPTBot, ClaudeBot and CCBot, but not PerplexityBot. These sites block training crawlers but leave one answer crawler open.
- 1 site blocked only CCBot.
- 116 of the 127 files (91%) named a sitemap with a Sitemap line.
- 1 site listed pages in its sitemap that its own robots.txt blocked for Googlebot.
Partial blocks for Googlebot
Most sites do not block a crawler from everything. They block some paths. Our scan stores the full path rules for Googlebot only. 67 of the 127 files (53%) had at least one Disallow rule for Googlebot. The other 60 files (47%) had no Disallow rule for Googlebot.
| Platform | Sites | Disallow rules for Googlebot | Share |
|---|---|---|---|
| Next.js | 31 | 21 | 68% |
| Lovable, server-rendered (TanStack Start) | 17 | 12 | 71% |
| React single-page app | 10 | 4 | 40% |
| Lovable, browser-rendered (Vite) | 23 | 7 | 30% |
| Other platforms (Astro, WordPress, Webflow, Replit, Base44) | 13 | 9 | 69% |
| Platform not detected | 36 | 14 | 39% |
| All sites | 130 | 67 | 52% |
The platform rows count all 130 sites, so the total share is 52% here. The "Other platforms" row joins five small groups. Read it with care.
The blocked paths were mostly private app areas. Of the 67 sites with Disallow rules for Googlebot, 37 had a rule that starts with /api. 31 had a rule for /admin, 26 for /dashboard, 16 for /auth and 12 for /settings.
What this means
In this sample, robots.txt rarely kept AI crawlers out. Almost every site had the file, and almost every file let the six crawlers in. When a site used robots.txt to block an AI crawler, it blocked training crawlers more often than answer crawlers. If an AI assistant does not cite your site, robots.txt is one item to check. It is not often the cause in this data. OpenAI's crawlers have different jobs, and our guide to GPTBot, OAI-SearchBot and ChatGPT-User explains them.
How we got these numbers
We took the latest scan of each site that was scanned with Why Won't AI from November 11 to November 17, 2026: 130 sites. We left out our own three sites. Many of the sites are new products whose founders listed them on launch directories, so the sample leans to small, new sites. It is not a random sample of the web. We do not name any site.
The scan gets robots.txt once per site. For each of the six crawlers, it records if the rules block the whole site. It records the full path rules for Googlebot only. So this study does not measure partial blocks for the AI crawlers. It also does not measure OAI-SearchBot, ChatGPT-User, Claude-SearchBot or Google-Extended. The scan detects the platform from the page code. It could not detect a platform for 36 sites.
Do most sites block AI crawlers in robots.txt?
Not in this data. Only 3 of the 127 sites with a readable robots.txt blocked an AI crawler from the whole site.
Which AI crawlers were blocked most often?
CCBot on 3 sites, GPTBot on 2 sites and ClaudeBot on 2 sites. No site blocked PerplexityBot from the whole site.
Can I check my own robots.txt?
Yes. The free AI crawler robots.txt checker reads your file and shows what it tells each crawler.