AI search glossary
Plain definitions of AI search terms: AEO and GEO, GPTBot and other AI crawlers, robots.txt, llms.txt, rendering, structured data, citations and more.
- AEO (answer engine optimization)
- Work that helps AI assistants and answer engines find, read and quote your site. It overlaps with SEO, but the goal is to be named in an answer, not only to rank in a list of links.
- How to measure AI visibility
- Agent-ready site
- A site that AI agents can read and use: clear text in the HTML, stable links, and machine-readable files that describe what the site offers.
- Make your site agent-ready
- AI agent
- An AI system that takes steps for a person, for example it opens pages, calls tools or fills in forms.
- Make your site agent-ready
- AI crawler
- A program that an AI company uses to fetch web pages. Some AI crawlers collect pages for training, some find pages for search answers, and some fetch a page when a person asks for it.
- AI crawler robots.txt checker
- AI visibility
- How often AI assistants name or cite your product when people ask questions about your category. You measure it by asking the same buyer questions over time.
- How to measure AI visibility
- Answer engine
- A tool that answers a question in its own words instead of only listing links. ChatGPT, Perplexity, Gemini, Claude and Google AI Overviews are examples.
- Bing Webmaster Tools
- Microsoft's free tool for site owners. It shows Bing search data and lets you submit sitemaps and URLs.
- Bingbot
- Microsoft's crawler for Bing. Bing results also feed other tools that use the Bing index.
- Is my site indexed?
- Buyer question
- A question a real buyer asks before they choose a product, for example "best tool to check if AI can read my site". AI visibility tools track answers to these questions.
- Choose buyer questions to track
- Canonical URL
- The one address you want search engines to treat as the main copy of a page. You set it with a link tag in the page head.
- Meta tag checker
- CCBot
- The crawler of Common Crawl, a public archive of web pages. Many AI models use Common Crawl data for training.
- ChatGPT Business Hub
- OpenAI's place to claim your business in ChatGPT, update its details and see mentions and website clicks.
- ChatGPT Business Hub guide
- ChatGPT-User
- OpenAI's agent that fetches a page when a person in ChatGPT asks for it, for example when they paste a link.
- GPTBot, OAI-SearchBot and ChatGPT-User
- Citation
- A link to a source that an AI answer shows with its text. A page that gets cited is a page the answer used.
- Find the pages AI cites
- Citation gap
- A page type or source that AI answers cite for your competitors but not for you.
- Find the pages AI cites
- ClaudeBot
- Anthropic's web crawler. Like many AI crawlers, it reads the HTML that the server sends.
- See what AI crawlers get
- Client-side rendering (CSR)
- A site setup where the browser builds the page with JavaScript. The server sends an almost empty page, so crawlers that do not run JavaScript get little text.
- Why crawlers can't see JavaScript sites
- Crawl
- One fetch of a page by a crawler. A crawler reads the response and may follow the links it finds.
- Direct answer
- An opening paragraph that says plainly what you do, for whom, with a specific detail. It is the text an AI answer can quote.
- Empty shell
- A page whose raw HTML has almost no text because the content appears only after JavaScript runs. AI crawlers that read raw HTML have nothing to quote.
- AI App Builder Index
- Fact check
- A check that compares what an AI assistant says your product does with the facts on your own site, and flags claims that contradict them.
- Fix wrong AI answers about your product
- GEO (generative engine optimization)
- Another name for work that helps generative AI tools mention and cite your site. Most people use GEO and AEO for the same thing.
- How to measure AI visibility
- Google Search Console
- Google's free tool for site owners. It shows which searches show your pages, the clicks they get, and which pages Google indexed.
- Is my site indexed?
- Google-Extended
- A robots.txt token that Google reads to decide whether your content can be used for Gemini models. It is not a separate crawler, and it does not change Google Search.
- Googlebot
- Google's main crawler for Google Search. Googlebot can run JavaScript, but it can do so later than the first fetch.
- Is my site indexed?
- GPTBot
- OpenAI's crawler that collects content that may be used to train OpenAI's models. Blocking it does not remove a site from ChatGPT search.
- GPTBot, OAI-SearchBot and ChatGPT-User
- Grounding
- When an AI assistant bases its answer on pages it fetched or searched for, instead of only on what the model learned in training.
- Hallucination
- A claim in an AI answer that is not true and has no source. Wrong claims about your product are one kind.
- Fix wrong AI answers about your product
- Indexing
- The step where a search engine stores a page so it can show it in results. A page that is crawled is not always indexed.
- Is my site indexed?
- IndexNow
- A protocol that lets a site tell search engines such as Bing that a page is new or changed, so they can crawl it sooner.
- Internal links
- Links from one page on your site to another. They help crawlers find pages and show which pages matter most.
- JSON-LD
- A format for structured data. It sits in a script tag in the page and uses the Schema.org vocabulary.
- Schema markup generator
- llms.txt
- A proposed plain-text Markdown file at the root of a site that gives AI systems a short map of the site. It is a convention, not a standard that every AI company reads.
- What is llms.txt?
- MCP (Model Context Protocol)
- An open protocol that lets AI assistants and coding agents use outside tools and data. An MCP server offers tools that an agent can call.
- Why Won't AI MCP server
- Mention
- An AI answer that names your product. A mention can come with or without a link to your site.
- How to measure AI visibility
- Meta description
- A short summary in the page head. Search engines and some AI tools can show it under the title.
- Meta tag checker
- OAI-SearchBot
- OpenAI's crawler that finds sites to show in ChatGPT search answers. You control it separately from GPTBot in robots.txt.
- GPTBot, OAI-SearchBot and ChatGPT-User
- Orphan page
- A page that no other page on the site links to. Crawlers may not find it.
- PerplexityBot
- Perplexity's crawler that finds pages for Perplexity's answers.
- See what AI crawlers get
- Prerendering
- Making a ready HTML copy of a JavaScript page so crawlers get the full text. It is one way to fix an empty shell.
- Prompt tracking
- Asking AI assistants the same set of buyer questions on a schedule and recording who they name and cite.
- How to measure AI visibility
- Query fan-out
- When an AI assistant splits one question into several searches and uses the results of all of them in its answer.
- Why AI assistants give different answers
- Raw HTML
- The HTML that the server sends before any JavaScript runs. Crawlers that do not run JavaScript see only this.
- See what AI crawlers get
- Rendered HTML
- The page after a browser runs its JavaScript. It can hold much more text than the raw HTML.
- Pages that look identical to crawlers
- robots.txt
- A plain-text file at the root of your site that tells crawlers which paths they may fetch. Each crawler follows the group of rules that names its user agent.
- robots.txt generator
- Schema.org
- A shared vocabulary of types (Organization, Product, FAQPage, Article and more) that search engines read in structured data.
- Schema markup generator
- Server-side rendering (SSR)
- A site setup where the server sends the full page text in its HTML. Every crawler can read it without running JavaScript.
- Why crawlers can't see JavaScript sites
- Single-page app (SPA)
- A site that loads one HTML page and changes the content with JavaScript. Without server rendering, crawlers often get an empty shell.
- AI App Builder Index
- Sitemap
- An XML file that lists the pages you want search engines to find, often with the date each page last changed.
- How to set up a sitemap
- Soft 404
- A page that says "not found" but answers with status 200 (OK). Search engines can treat it as an error page or index the wrong thing.
- Is my site indexed?
- Static site generation (SSG)
- A site setup where pages are built into HTML files before anyone visits. Crawlers get the full text on the first fetch.
- Structured data
- Machine-readable facts about a page, for example the product name or the questions it answers. Most sites add it as JSON-LD.
- Schema markup generator
- Title tag
- The page title in the page head. It names the page in search results and browser tabs.
- Meta tag checker
- Training data
- The text a model learned from before release. Facts that changed after that date reach the model only through search or grounding.
- User agent
- The name a crawler or browser sends with each request, for example GPTBot or Googlebot. robots.txt rules use the user agent name to target one crawler.
- AI crawler robots.txt checker
Want to see these terms on your own site? Run a free scan.