Making content only for AI agents repeats SEO's old mistakes, consultant warns

An SEO team right now is picking whether to publish a page only AI agents will read. Writing in Search Engine Journal, Chris Green argues that the pattern repeats the oldest mistake in search: creating one version of a brand for machines and another for people. Green is technical director and senior consultant at Torque Partnership.

Green walks through Google's own history. Panda cut low-quality pages in 2011. Penguin cut link spam in 2012. The Helpful Content system in 2022 rewarded pages written for people. The 2024 spam update called out scaled content, whether a human or an AI wrote it. Each move hit the same pattern: a site telling the engine one thing and the user another.

Where the parallel bites

Green names the practices he thinks fit this pattern. The llms.txt proposal points a model at useful parts of a site. Agent-specific markdown pages serve a cleaner version of content to agents. Some brands are now hosting standalone information endpoints that only agents fetch.

Green draws a line between two kinds of agent-friendly work. Making the same information available in a different format, as Cloudflare's Markdown for Agents does, is fine. Making different information available to different audiences is the problem. A page may read one way for a buyer and another way for an agent. Google already names that shape. It calls the shape cloaking and has fought it for years.

Why AI engines may not play along

Green raises a trust problem. An AI engine has no record of years of people trying to game it, the way Google has. If information exists only in an agent-only file, the engine cannot check it against a public page. The agent version and the public version may disagree. The engine then has no clear reason to pick one over the other.

He expects the engines to react. He argues their incentive is to serve answers people can check. A business of ever more fragmented, agent-only pages makes checking harder. Green writes that structured data, XML sitemaps, APIs and product feeds are not the problem. The problem is content made just to influence an AI answer. The matching public page a buyer can read does not exist.

For a team deciding whether to publish an AI-only page this quarter, Green's piece is a reason to stop. His conclusion is plain: the shortcut will not last. The safer path is to put the same information on the public site. A reader can then find it. An engine can then check it.

Source

• Why We Should Fear The Shadow Web We're Building For AI, Search Engine Journal