On this page
As of November 2026, AI agents do more than read pages. They look for files that say what your service does and how to use it, and some can call your service for a user. An agent-ready site has pages that crawlers can read, an llms.txt map, and, if you offer an API, an MCP server or clear API docs with standard sign-in metadata.
The pieces and what each does
| Piece | What it does | Who needs it |
|---|---|---|
| Readable pages | Gives crawlers and agents your real content without JavaScript | Every site |
| llms.txt | A short map of your most useful pages | Every site, as cheap insurance |
| MCP server | Lets AI apps such as Claude, Cursor and VS Code call your tools | Services with an API |
| OAuth protected resource metadata | Tells an agent where to sign in to use your API | APIs that need a user's permission |
| API reference and OpenAPI file | Tells agents exactly how to call each endpoint | Services with an API |
How we did it on our own site
- Our pages render on the server, so every crawler gets the full text.
- Our llms.txt lists every guide and tool with one line each.
- We run an MCP server at mcp.whywontai.com, listed in the official MCP Registry, with one tool that needs no key and others that use the account's plan.
- Agents can sign in with OAuth, and our metadata files tell them where.
Start with the basics
Most sites do not need an MCP server. They need pages that crawlers can read, a sitemap, a robots.txt that lets the search crawlers in, and an llms.txt. Add agent files when you have an API that agents should use.
Questions people ask
Do I need an MCP server?
Only if you have a service that AI apps should call. A content site gets more from readable pages and an llms.txt.
Is MCP a standard?
MCP is an open protocol with its own specification and an official registry of servers.
Do agents follow robots.txt?
Crawlers that collect pages usually do. Agents acting for a person may fetch a page even when robots.txt blocks crawlers, as OpenAI says of ChatGPT-User.
What this does not tell you
Which agent files your site serves today, whether they are correct, and what agents get when they request them.
Sources
- 1.Model Context Protocol: the protocol AI apps use to call tools on other services
- 2.RFC 9728: OAuth 2.0 Protected Resource Metadata: how a service tells agents where to sign in
- 3.llmstxt.org: the llms.txt proposal
- 4.OpenAI: Overview of OpenAI crawlers: OpenAI's search and training crawlers