AI search glossary

Plain definitions of AI search terms: AEO and GEO, GPTBot and other AI crawlers, robots.txt, llms.txt, rendering, structured data, citations and more.

AEO (answer engine optimization)
Work that helps AI assistants and answer engines find, read and quote your site. It overlaps with SEO, but the goal is to be named in an answer, not only to rank in a list of links.
How to measure AI visibility
Agent-ready site
A site that AI agents can read and use: clear text in the HTML, stable links, and machine-readable files that describe what the site offers.
Make your site agent-ready
AI agent
An AI system that takes steps for a person, for example it opens pages, calls tools or fills in forms.
Make your site agent-ready
AI crawler
A program that an AI company uses to fetch web pages. Some AI crawlers collect pages for training, some find pages for search answers, and some fetch a page when a person asks for it.
AI crawler robots.txt checker
AI visibility
How often AI assistants name or cite your product when people ask questions about your category. You measure it by asking the same buyer questions over time.
How to measure AI visibility
Answer engine
A tool that answers a question in its own words instead of only listing links. ChatGPT, Perplexity, Gemini, Claude and Google AI Overviews are examples.
Bing Webmaster Tools
Microsoft's free tool for site owners. It shows Bing search data and lets you submit sitemaps and URLs.
Bingbot
Microsoft's crawler for Bing. Bing results also feed other tools that use the Bing index.
Is my site indexed?
Buyer question
A question a real buyer asks before they choose a product, for example "best tool to check if AI can read my site". AI visibility tools track answers to these questions.
Choose buyer questions to track
Canonical URL
The one address you want search engines to treat as the main copy of a page. You set it with a link tag in the page head.
Meta tag checker
CCBot
The crawler of Common Crawl, a public archive of web pages. Many AI models use Common Crawl data for training.
ChatGPT Business Hub
OpenAI's place to claim your business in ChatGPT, update its details and see mentions and website clicks.
ChatGPT Business Hub guide
ChatGPT-User
OpenAI's agent that fetches a page when a person in ChatGPT asks for it, for example when they paste a link.
GPTBot, OAI-SearchBot and ChatGPT-User
Citation
A link to a source that an AI answer shows with its text. A page that gets cited is a page the answer used.
Find the pages AI cites
Citation gap
A page type or source that AI answers cite for your competitors but not for you.
Find the pages AI cites
ClaudeBot
Anthropic's web crawler. Like many AI crawlers, it reads the HTML that the server sends.
See what AI crawlers get
Client-side rendering (CSR)
A site setup where the browser builds the page with JavaScript. The server sends an almost empty page, so crawlers that do not run JavaScript get little text.
Why crawlers can't see JavaScript sites
Crawl
One fetch of a page by a crawler. A crawler reads the response and may follow the links it finds.
Direct answer
An opening paragraph that says plainly what you do, for whom, with a specific detail. It is the text an AI answer can quote.
Empty shell
A page whose raw HTML has almost no text because the content appears only after JavaScript runs. AI crawlers that read raw HTML have nothing to quote.
AI App Builder Index
Fact check
A check that compares what an AI assistant says your product does with the facts on your own site, and flags claims that contradict them.
Fix wrong AI answers about your product
GEO (generative engine optimization)
Another name for work that helps generative AI tools mention and cite your site. Most people use GEO and AEO for the same thing.
How to measure AI visibility
Google Search Console
Google's free tool for site owners. It shows which searches show your pages, the clicks they get, and which pages Google indexed.
Is my site indexed?
Google-Extended
A robots.txt token that Google reads to decide whether your content can be used for Gemini models. It is not a separate crawler, and it does not change Google Search.
Googlebot
Google's main crawler for Google Search. Googlebot can run JavaScript, but it can do so later than the first fetch.
Is my site indexed?
GPTBot
OpenAI's crawler that collects content that may be used to train OpenAI's models. Blocking it does not remove a site from ChatGPT search.
GPTBot, OAI-SearchBot and ChatGPT-User
Grounding
When an AI assistant bases its answer on pages it fetched or searched for, instead of only on what the model learned in training.
Hallucination
A claim in an AI answer that is not true and has no source. Wrong claims about your product are one kind.
Fix wrong AI answers about your product
Indexing
The step where a search engine stores a page so it can show it in results. A page that is crawled is not always indexed.
Is my site indexed?
IndexNow
A protocol that lets a site tell search engines such as Bing that a page is new or changed, so they can crawl it sooner.
JSON-LD
A format for structured data. It sits in a script tag in the page and uses the Schema.org vocabulary.
Schema markup generator
llms.txt
A proposed plain-text Markdown file at the root of a site that gives AI systems a short map of the site. It is a convention, not a standard that every AI company reads.
What is llms.txt?
MCP (Model Context Protocol)
An open protocol that lets AI assistants and coding agents use outside tools and data. An MCP server offers tools that an agent can call.
Why Won't AI MCP server
Mention
An AI answer that names your product. A mention can come with or without a link to your site.
How to measure AI visibility
Meta description
A short summary in the page head. Search engines and some AI tools can show it under the title.
Meta tag checker
OAI-SearchBot
OpenAI's crawler that finds sites to show in ChatGPT search answers. You control it separately from GPTBot in robots.txt.
GPTBot, OAI-SearchBot and ChatGPT-User
Orphan page
A page that no other page on the site links to. Crawlers may not find it.
PerplexityBot
Perplexity's crawler that finds pages for Perplexity's answers.
See what AI crawlers get
Prerendering
Making a ready HTML copy of a JavaScript page so crawlers get the full text. It is one way to fix an empty shell.
Prompt tracking
Asking AI assistants the same set of buyer questions on a schedule and recording who they name and cite.
How to measure AI visibility
Query fan-out
When an AI assistant splits one question into several searches and uses the results of all of them in its answer.
Why AI assistants give different answers
Raw HTML
The HTML that the server sends before any JavaScript runs. Crawlers that do not run JavaScript see only this.
See what AI crawlers get
Rendered HTML
The page after a browser runs its JavaScript. It can hold much more text than the raw HTML.
Pages that look identical to crawlers
robots.txt
A plain-text file at the root of your site that tells crawlers which paths they may fetch. Each crawler follows the group of rules that names its user agent.
robots.txt generator
Schema.org
A shared vocabulary of types (Organization, Product, FAQPage, Article and more) that search engines read in structured data.
Schema markup generator
Server-side rendering (SSR)
A site setup where the server sends the full page text in its HTML. Every crawler can read it without running JavaScript.
Why crawlers can't see JavaScript sites
Share of voice
Your share of the mentions in a set of AI answers, compared with your competitors.
Single-page app (SPA)
A site that loads one HTML page and changes the content with JavaScript. Without server rendering, crawlers often get an empty shell.
AI App Builder Index
Sitemap
An XML file that lists the pages you want search engines to find, often with the date each page last changed.
How to set up a sitemap
Soft 404
A page that says "not found" but answers with status 200 (OK). Search engines can treat it as an error page or index the wrong thing.
Is my site indexed?
Static site generation (SSG)
A site setup where pages are built into HTML files before anyone visits. Crawlers get the full text on the first fetch.
Structured data
Machine-readable facts about a page, for example the product name or the questions it answers. Most sites add it as JSON-LD.
Schema markup generator
Title tag
The page title in the page head. It names the page in search results and browser tabs.
Meta tag checker
Training data
The text a model learned from before release. Facts that changed after that date reach the model only through search or grounding.
User agent
The name a crawler or browser sends with each request, for example GPTBot or Googlebot. robots.txt rules use the user agent name to target one crawler.
AI crawler robots.txt checker

Want to see these terms on your own site? Run a free scan.