The Fundamental Flaw of LLMs.txt
Publishers are generating LLMs.txt files to get their sites surfaced by AI. Google's John Mueller explains why that was never what the file does — and why it can't.
Search runs on a 5-stage pipeline. LLMs.txt touches none of it.
Every page that ranks passes through the same sequence. It starts with Discovery — the engine learning the URL exists. Skip that first box and the page is invisible, no matter what else you optimize.
The file does have a purpose — just not the one it's being used for.
Mueller spoke with one of the proposal's creators. The intent was narrow, and it was never about getting found.
Site owners publish LLMs.txt hoping AI systems will find, trust, and surface their content. This use case conflicts with the file's design — it was never meant to drive discovery.
If an AI already knows your site and wants to see what else is there, LLMs.txt can point the way. It's an internal index for the known — not a beacon for the unknown.
It's self-reported — so an AI can't lean on it to compare sites.
LLMs.txt is the site owner describing their own content, which may not match the actual HTML. Every site claims to be the best, so the signal carries no weight for differentiation.
By design, an LLM can't trust a file in which a site simply declares it has the best pages and you must buy its products.
LLMs.txt vs. WebMCP
If AI agents become how users interact with sites, the useful standard is one that lets them act — not one that merely describes. Neither, however, gets you discovered.
Test the agentic layer today — and prep your site to be callable.
WebMCP is a proposed web standard (built by Google's Chrome team, incubated through the W3C, with the goal of any agentic browser implementing it). It lets a page expose its actions as structured tools an AI agent can call — turning your site into an API agents use without you maintaining a separate one. It supports tool discovery, JSON Schema for inputs/outputs, and shared page state. It's in origin trial from Chrome 149, and available behind a local flag for development.
- 01Chrome with WebMCP — the Chrome 149+ origin trial for live users, or the local-dev flag for testing.
- 02Local dev: set chrome://flags/#enable-webmcp-testing to Enabled, then relaunch.
- 03Model Context Tool Inspector extension to register, call, and verify a page's tools.
- 04An origin-isolated document — WebMCP is disabled if document.domain / Origin-Agent-Cluster: ?0 is set.
- 05The tools permissions policy — defaults to self; cross-origin iframes need allow="tools".
- 06An open tab — no headless support; a visible browsing context is required to call tools.
Use a recent Chrome build for local testing.
Set the WebMCP testing flag to Enabled.
Restart Chrome so the flag takes effect.
Add the Tool Inspector; prompt it in natural language.
Add toolname and tooldescription attributes to a clean HTML form. The browser auto-generates the schema and fills the fields when an agent calls the tool.
Use navigator.modelContext to register a tool with a name, description, input schema, and execute function. Tools appear and disappear with page state (e.g. checkout only when the cart has items).
Discovery and ranking are still bound to HTML.
Don't ship LLMs.txt for visibility. It plays no role in how AI or search finds your pages.
Keep optimizing the HTML. Discovery, crawling, and ranking all happen at the page level.
Watch WebMCP for ecommerce. Agentic standards are still unsettled — none has won yet.