Google’s John Mueller responded to a question about whether Google’s advice on LLMs.txt was contradictory. His response shows why Google considers LLMs.txt speculative at the moment and why WebMCP, a Google-backed proposal, might be what SEOs and site owners should look into.
LLMs.txt, Chrome Lighthouse and Google tips
A discussion on Reddit asked about Google’s seemingly contradictory advice. The questioner noted that Google Search Central says publishers don’t need special AI files like LLMs.txt to appear in AI search experiences. But Chrome’s Lighthouse Audit documentation includes a check for the LLMs.txt file, which makes it seem like there’s a conflict between the two directives.
The editor asked:
“Google Search Central says you don’t need special AI files like LLMs.txt to appear in generative AI search, but Chrome developers say LLMs.txt can help agents understand the structure and main content of a site.
I’m anti-LLMs.txt, but is this advice contradictory from Google? Or am I confusing “Google search visibility” with “AI agent usability”?
Classic technical writing that fails to communicate
The person asking the question apparently didn’t understand what the Chrome Lighthouse Audit documentation said, as it is poorly written.
Contrary to what the Redditor believes, the page does not say that “LLMs.txt can help agents understand the structure and main content of a site. »
What is this do to say is that LLMS.txt is an “emerging convention” and that without this file AI agents”can spend more time exploring the site to understand its high-level structure and main content. This word “can» does a lot of work because it’s a hidden word that communicates a possibility, not a certainty.
- The Redditor believes there is an advantage to Chrome’s advice.
- But the wording itself only suggests one possibility.
The Chrome Lighthouse guidelines are jargon because they use the phrase “emerging convention” instead of correctly saying that LLMS.txt is just a proposal and not a standard.
Using the word “may” is a poor choice because it forces the reader to infer that the benefit of an LLMs.txt is hypothetical and not something that is currently happening.
Mueller’s ironic response
Mueller’s response is ironic. He emphasizes that if an AI platform needs a file, give it to them. No AI sites actually require or use LLMs.txt, so this is purely hypothetical. The irony is that he points out that website owners and SEOs use LLMs to parse HTML content to generate an LLMs.txt so that another LLM doesn’t have to do it.
Mueller’s response:
“When an AI platform that brings you clients complains that they need the file for your site, then I recommend you take the time to create one. (Plus, if you use an LLM to create the file for you, doesn’t that mean the LLM could just… create it for itself too?)”
Mueller points out the inherent redundancy of the LLMs.txt file.
LLMs.txt is purely speculative at this time
After Mueller answered To the original question, another Redditor continued the discussion by suggesting that the benefit of the proposed LLMs.txt standard is that it would reduce the amount of crawling and bandwidth used while also improving interactions between a website and AI agents.
The editor wrote:
“Don’t you think it would be a good thing to provide user-friendly, lightweight interfaces between your site and LLMs? I consider this similar to page speed a good thing for user experience. LLMs are very resource intensive, so why not do what we can to reduce resource consumption during recovery?”
Mueller responded:
“I don’t think anyone knows – it’s purely speculative at the moment (the file has been around for years, yet no AI systems are using it – what does that mean?).
I like the WebMCP approach, as well as the business integrations – they have clear goals and processes: “Given the agent is already on your site, how can it *properly* perform task X?” (for example, determining the final price of a product, including all fees and potential discounts).
Mueller introduces an interesting point about WebMCP. WebMCP is a proposed web standard that leverages the Model Context Protocol to do much more than LLMs.txt.
WebMCP allows AI agents to discover and use a website’s functionality, making it easier for them to interact with a website instead of using HTML code created for humans. This allows AI agents to compare products, add items to a cart and fill out a contact form. WebMCP is currently supported in Chrome.
Perhaps the biggest problem is whether sites block agents
Mueller’s response continues with an additional observation that is perhaps the most important part of the discussion. He suggested that the most basic form of agent optimization is simply ensuring that agents are not blocked from accessing a site. He said this hurdle might be more important to most publishers than questions about LLMs.txt.
He continues his response:
“I don’t think there is the agent equivalent of ‘let me look at 10 sites and see where I can buy
So, speculatively, I’m guessing that if an agent is already on your site and tasked with doing something, they’ll happily just click and try to complete the task with the UI as well.
Of course, all of this assumes that the most basic agent optimization is in place, namely: don’t block agents. I think this hurdle will be the biggest for most sites.
The reality today is that AI agents can use HTML UI designed for humans. So even if the site doesn’t implement WebMCP, the agent can still accomplish tasks. The bottom line is that it is increasingly important to ensure that AI agents are not blocked.
