AI NewsGo-to-marketReported
Semrush ran its own llms.txt file for two and a half months and saw no visits from any AI crawler
Semrush's Tushar Pol says a two and a half month server-log test of the llms.txt file on Search Engine Land recorded zero visits from Google-Extended, GPTBot, PerplexityBot or ClaudeBot, matching Google's John Mueller who wrote in June that AI systems do not use the file.
Image: Semrush
Why it mattersA marketing or SEO team can stop treating llms.txt as a required file for AI search visibility, and spend the time on the pages the AI crawlers actually read.
Every few weeks a new blog post says an AI-friendly file called llms.txt is the next thing every website has to publish. Semrush ran the file on Search Engine Land, its own SEO news site, for two and a half months and watched the server logs. No AI crawler read it once.
The post was published by Tushar Pol at Semrush on 28 September 2026, with Dana Nicole contributing. Search Engine Land added the file in mid-August 2025 and Semrush pulled the server-log data at the end of October. Over that period, the Semrush post reports, Google-Extended, GPTBot, PerplexityBot and ClaudeBot each visited the file zero times. Regular web crawlers hit it, but no AI-only crawler did.
What the file is meant to do
The llms.txt specification was proposed by Jeremy Howard at Answer.AI on 3 September 2024 as a markdown index at the site root, so a language model can find the important pages without crawling the whole site. Anthropic, Hugging Face, Vercel, Zapier and Cal.com are among the sites that publish one now. Semrush's own trend line, from the tracker it cites, is a jump from 951 domains in July 2025 to 16,670 in September 2026, or about 17 times more sites in fourteen months.
The pitch and the adoption both point one way. The server logs point the other.
What the AI platforms actually say
Semrush lists no confirmation of use from ChatGPT, Google AI Overviews, Claude, Gemini, Perplexity or Bing. It quotes Google's John Mueller on Bluesky in June 2025 saying AI systems do not use llms.txt. Anthropic publishes an llms.txt at platform.claude.com, which is evidence that the company follows the spec on its own site and no evidence about what Claude does with such files elsewhere. Google Lighthouse recently added a check that flags whether a site has one, which the Semrush post treats as interest rather than proof of crawling.
Semrush's own recommendation
The post is careful about what it recommends. Semrush says a team may add an llms.txt as a low-cost experiment, and does not call it mandatory. It also names two real risks that come with publishing one: the file gives competitors a curated table of contents to your site, and a static markdown index goes out of date every time the underlying pages change, so a claim it makes about your product can be wrong for months.
Where the pitch stops
The Semrush post is Semrush's own test on one property it owns, over a specific window, and other sites may see different behaviour, or crawlers may change their minds. What is not in doubt is that a hundred SEO agencies charging clients for llms.txt setup this year cannot point to a single AI platform that has said it reads the file. On the current evidence, a team writing for AI search should keep the effort on the pages the crawlers do read.
Source
- Semrush blog post by Tushar Pol, primary source: https://www.semrush.com/blog/llms-txt
- The llms.txt specification, Answer.AI, 3 September 2024: https://llmstxt.org/
- Anthropic's own llms.txt file: https://platform.claude.com/llms.txt
This item was written by an AI system from the linked source. Reveneau is responsible for what it publishes.
Get AI News in your inbox
New developer tools, model and agent releases, and how teams are actually using them to release software. Short, and only when there is something worth reading.
