Crawl access

Crawler access test

Request a page the way Googlebot, Bingbot and AI crawlers do and compare what each one gets with what a browser shows. Use when a user asks what GPTBot or ClaudeBot sees, or when a page is missing from AI answers.

crawler-access-test/SKILL.md

Goal

Show, per crawler, the status code and the text each bot gets for a page, and compare it with the rendered page.

Workflow

  1. Pick 3 to 10 URLs: the home page, pricing, one product page, one article.
  2. For each URL and each user agent, run a request without JavaScript: Use the current user agent strings from each owner's docs: Googlebot, Bingbot, GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot.
    curl -sS -L -o body.html -w "%{http_code} %{content_type} %{size_download}\n" \
      -A "Mozilla/5.0 (compatible; GPTBot/1.0; +https://openai.com/gptbot)" https://example.com/pricing
  3. From each body, extract: <title>, meta description, canonical, <h1>, and the visible text length (strip scripts and tags).
  4. Render the same URL in a headless browser and extract the same fields.
  5. Compare. Flag a page when:
    • The status is not 200 for a wanted bot.
    • The raw HTML has almost no visible text but the rendered page has a lot. This is a client-side rendering problem (see client-side-rendering-fix).
    • The raw title or canonical differs from the rendered one.
    • The body is a challenge or error page.

Decision rules

  • Google renders JavaScript, but it does so after the first crawl and can miss content. Many AI crawlers do not run JavaScript. Treat the raw HTML as what an AI crawler gets.
  • A user agent test does not prove what the real bot gets, because some firewalls check IP ranges. Use server or CDN logs to confirm.

Output

A table per URL: crawler, status, title, H1, visible words in raw HTML, visible words after render, problem.

Sources

  • Google: Understand JavaScript SEO basics. https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics
  • OpenAI crawlers. https://developers.openai.com/api/docs/bots
  • Anthropic crawlers. https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
  • Perplexity crawlers. https://docs.perplexity.ai/docs/resources/perplexity-crawlers
See all 30 skills