Common Crawl analyzed 584,107 llms.txt files, and 22.56 percent had no links

Common Crawl published a content analysis of 584,107 llms.txt files pulled from its July 2026 crawl, and the main finding is that the file the AI search community has been treating as a robots.txt for language models does not do what people are writing into it. The post, by Common Crawl Senior Research Engineer Malte Ostendorff, was published on 31 August 2026 and reached more readers this week through a Search Engine Journal write-up.

What Common Crawl actually looked at

Common Crawl added /llms.txt and /llms-full.txt to the list of URLs in its July 2026 crawl for a large sample of hosts, then filtered the 598,298 responses down to the 584,107 files that carried real content. Adoption in that sample was 11.72% for /llms.txt among hosts Common Crawl can fetch, well above the 2% the Web Almanac reported for 2025 across all sites. Common Crawl notes the two populations differ, so the numbers cannot be compared directly.

Many sites copy the format and leave out the useful content

Common Crawl reports that 49.90% of the corpus follows the full specification: an H1, a summary blockquote, and section headings of link bullets. But 22.56% carry no links at all, in a file that is supposed to publish a curated list of URLs. Only 32.94% of the links that do exist carry the short note the spec requires. 68.27% of the corpus is templated, with Wix alone producing 41.34% of the files. All in One SEO writes 73,136 llms.txt files with a median of 138 links and never includes a summary: the plugin treats the file as a sitemap. 2.54% of the corpus is a GoDaddy parked-domain template, a sales pitch aimed at a language model that still passes every structural test.

The finding a publisher should read carefully

Common Crawl found 1,570 files (0.27%) that name a specific crawler and either allow or deny it. Of those, 32 files claim to block CCBot. On 17 August 2026, Common Crawl fetched robots.txt for all 32 of those sites and reports that none of them blocks CCBot in robots.txt. Five allow it explicitly, eleven allow it with a handful of excluded paths, and fifteen do not restrict it at all. Common Crawl quotes proform.com as the example: an llms.txt section headed "AI Crawler Access (robots.txt status as of June 2026)" that lists CCBot under Blocked, while the actual robots.txt allows CCBot under User-agent: * with Allow: /. Common Crawl restates its position: "llms.txt is not an access-control mechanism, and nothing obliges any crawler to read it."

Common Crawl also reports on prompt injection in the corpus. Ten files match its strictest test, and the team read all ten by hand: four are genuine injections, one is a bug-bounty researcher's deliberate payload, one asks the reader to become a catgirl, one is a joke about poisoning crawler logs, and three are false positives on the phrase "you are now" or on the literal ChatML tokens shown in a technical blog. Every real injection was placed there deliberately by someone technical enough to be making a point.

For a team working on being cited by answer engines, the practical lesson is that any crawler rule written into llms.txt has no effect. Rules that a crawler will actually honor belong in robots.txt, and any opt-out written anywhere else has not been made. The llms.txt file is still useful as a curated list of pages worth reading, provided the links and their notes are real and the team maintaining it treats it as documentation.

Source

A Content Analysis of llms.txt Files from the July 2026 Crawl Archive, by Malte Ostendorff at Common Crawl, 31 August 2026. Reported by Search Engine Journal, 8 September 2026.