Research

What robots.txt tells AI crawlers on 130 websites (November 2026)

127 of 130 sites we scanned had a robots.txt file. Only 3 blocked an AI crawler from the whole site, and none blocked Googlebot or Bingbot.

By the Why Won't AI teamPublished 3 min readAsk about this page:ChatGPTClaude
On this page

We read the robots.txt file of 130 scanned websites. We checked the rules for 6 crawlers: Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot and CCBot. 127 sites (98%) had a robots.txt file that we could read. Only 3 sites (2%) blocked an AI crawler from the whole site. No site blocked Googlebot or Bingbot from the whole site.

This page is about the rules in robots.txt only. Our study of what AI crawlers get from these sites is a different question. It found that rendering, not robots.txt, was the main reason that AI crawlers got no text.

Whole-site blocks by crawler

CrawlerWhat it doesBlocks the whole siteDoes not block the whole site
GooglebotGoogle Search0127
BingbotBing search0127
GPTBotOpenAI model training2125
ClaudeBotAnthropic model training2125
PerplexityBotPerplexity search and answers0127
CCBotCommon Crawl data set3124

The counts use the 127 sites with a readable robots.txt file. A whole-site block is a Disallow rule for "/" that applies to that crawler. All three groups are under 10 sites, so read them with care.

What else we found

  • 3 sites (2%) had no readable robots.txt file. One returned 404. Two refused the request with 401 or 403.
  • No site blocked all six crawlers. So no robots.txt in the sample shut out every crawler.
  • 2 sites blocked GPTBot, ClaudeBot and CCBot, but not PerplexityBot. These sites block training crawlers but leave one answer crawler open.
  • 1 site blocked only CCBot.
  • 116 of the 127 files (91%) named a sitemap with a Sitemap line.
  • 1 site listed pages in its sitemap that its own robots.txt blocked for Googlebot.

Partial blocks for Googlebot

Most sites do not block a crawler from everything. They block some paths. Our scan stores the full path rules for Googlebot only. 67 of the 127 files (53%) had at least one Disallow rule for Googlebot. The other 60 files (47%) had no Disallow rule for Googlebot.

PlatformSitesDisallow rules for GooglebotShare
Next.js312168%
Lovable, server-rendered (TanStack Start)171271%
React single-page app10440%
Lovable, browser-rendered (Vite)23730%
Other platforms (Astro, WordPress, Webflow, Replit, Base44)13969%
Platform not detected361439%
All sites1306752%

The platform rows count all 130 sites, so the total share is 52% here. The "Other platforms" row joins five small groups. Read it with care.

The blocked paths were mostly private app areas. Of the 67 sites with Disallow rules for Googlebot, 37 had a rule that starts with /api. 31 had a rule for /admin, 26 for /dashboard, 16 for /auth and 12 for /settings.

What this means

In this sample, robots.txt rarely kept AI crawlers out. Almost every site had the file, and almost every file let the six crawlers in. When a site used robots.txt to block an AI crawler, it blocked training crawlers more often than answer crawlers. If an AI assistant does not cite your site, robots.txt is one item to check. It is not often the cause in this data. OpenAI's crawlers have different jobs, and our guide to GPTBot, OAI-SearchBot and ChatGPT-User explains them.

How we got these numbers

We took the latest scan of each site that was scanned with Why Won't AI from November 11 to November 17, 2026: 130 sites. We left out our own three sites. Many of the sites are new products whose founders listed them on launch directories, so the sample leans to small, new sites. It is not a random sample of the web. We do not name any site.

The scan gets robots.txt once per site. For each of the six crawlers, it records if the rules block the whole site. It records the full path rules for Googlebot only. So this study does not measure partial blocks for the AI crawlers. It also does not measure OAI-SearchBot, ChatGPT-User, Claude-SearchBot or Google-Extended. The scan detects the platform from the page code. It could not detect a platform for 36 sites.

Do most sites block AI crawlers in robots.txt?

Not in this data. Only 3 of the 127 sites with a readable robots.txt blocked an AI crawler from the whole site.

Which AI crawlers were blocked most often?

CCBot on 3 sites, GPTBot on 2 sites and ClaudeBot on 2 sites. No site blocked PerplexityBot from the whole site.

Can I check my own robots.txt?

Yes. The free AI crawler robots.txt checker reads your file and shows what it tells each crawler.