Research

What AI crawlers get from 129 websites (November 2026)

In our scans of 129 sites, 75% of browser-rendered sites sent AI crawlers an empty homepage. Server-rendered sites averaged 82 of 100.

By the Why Won't AI teamPublished 2 min readAsk about this page:ChatGPTClaude
On this page

We requested the pages of 129 websites as 6 crawlers: Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot and CCBot. Then we compared each answer with the page that a real browser builds. The main result: how a site builds its pages decides what AI crawlers get. 18 of the 24 browser-rendered sites (75%) sent the AI crawlers an empty homepage. None of the 84 server-rendered sites did.

Results by rendering type

How the site builds its pagesSitesAverage score (of 100)Empty homepage for AI crawlersNavigation links only after JavaScript
Server-rendered: the server sends the full HTML84820%4%
Mixed: some pages full, some pages empty186522%39%
Browser-rendered: the browser builds the page with JavaScript244175%88%
Pre-rendered snapshots3323 of 32 of 3
All sites1297119%26%

An empty homepage means that GPTBot, ClaudeBot, PerplexityBot and CCBot got little or no text in the raw HTML. These crawlers do not run JavaScript. If the text is not in the HTML, the crawler has nothing to quote.

What else we found

  • On 29% of the sites (37 of 129), every crawler we tested saw little or no text on most of the pages we checked.
  • On 26% of the sites (33 of 129), the navigation links exist only after JavaScript runs. A crawler that does not run scripts cannot follow them to the other pages.
  • robots.txt blocked AI crawlers on only 3 sites (2%). A firewall blocked our crawler requests on 5 sites (4%). Rendering, not blocking, was the main reason AI crawlers got no content.
  • 66 sites (51%) scored as healthy, 27 (21%) as needs work and 36 (28%) as critical.

How we got these numbers

We took the latest scan of each site that was scanned with Why Won't AI from November 11 to November 17, 2026: 129 sites. We left out our own three sites. Many of the sites are new products whose founders listed them on launch directories, so the sample leans to small, new sites. It is not a random sample of the web. We do not name any site.

Each crawler request uses that crawler's published user agent. We count a page as empty when the raw HTML has little or no visible text but the browser view has the full copy. The score has six parts: crawl and render, indexable body copy, intent match, internal linking, demand capture and AI citability.

Do AI crawlers run JavaScript?

In our tests, GPTBot, ClaudeBot, PerplexityBot and CCBot read the HTML that the server sends and do not run scripts. That is why 75% of the browser-rendered sites in this data sent them an empty homepage.

Is robots.txt the main reason AI crawlers miss a site?

Not in this data. robots.txt blocked AI crawlers on 3 of 129 sites. Pages that only build in the browser were a much more common reason.

Can I check my own site?

Yes. The free scan runs the same checks on up to 10 pages of your site and shows what each crawler got.