Duane Forrester says audits miss false AI descriptions of a company

Duane Forrester, founder of UnboundAnswers.com, published a column in Search Engine Journal on 3 September arguing that when an AI model has no real evidence about a company, it does not abstain. It produces a confident description of the nearest well-documented neighbour, and no content audit run against your own pages can find that failure.

The mechanism, and the research behind it

Forrester grounds the piece in three papers rather than one. He cites Mallen and colleagues on the finding that language models struggle specifically with less popular factual knowledge, and that scaling helps recall of popular facts far more than of the tail. He cites Longpre and colleagues at EMNLP 2021 on how models rely on memorised information over the text provided to them, framing the practitioner question as whether a model will hallucinate rather than read. And he cites Sciavolino and colleagues, also at EMNLP 2021, showing that dense retrievers underperform sparse methods on entity-rich questions and generalise reliably only to common entities.

The claim built on top of that research is that generic hallucination is random, but substitution is directional. Given a sparse entity and a dense neighbour, a model drifts toward the neighbour in the same direction every time, because the gradient pointing that way is a property of the training distribution.

Four forms he names

Forrester identifies four ways the substitution surfaces, each easy to misdiagnose. Silent analogy replaces your pricing model or your implementation timeline with a competitor's, or with a category average, and nothing in the answer shows that a substitution happened. Staleness presented as currency states a discontinued product or a departed executive in the present tense. Thin evidence handled with the confidence of thick evidence produces answers that read the same whether the model drew on one trade write-up or forty independent sources. Category knowledge applied to a specific company answers the industry question and attaches your name to it.

Why a content audit misses all four

Content audits inspect what exists: pages, structure, coverage, freshness. The failure Forrester describes lives in what does not exist, on questions your buyers ask that never touch your site. The natural instinct is to publish more, get retrieved, and let retrieval correct the weights. Forrester argues that instinct is weaker than it sounds, because the popularity gradient that thinned your presence in training reappears in what gets retrieved. Retrieval is a second application of the same bias.

His advice is to watch outputs rather than inputs, and to sample them a specific way. Asking a model to describe your company by name is the least informative test, because the name in the prompt pulls the thin material about you straight into context. The substitutions surface on category questions, on comparison questions naming only your competitor, and on capability questions where the model has to decide whether you belong in the answer at all. Run those repeatedly, and treat the results as a record of where third-party description of you is missing, rather than a list of pages to fix.

For anyone building AI features that describe real businesses, this changes what "correct" means at test time. A rubric that only checks named-brand prompts will pass with the questions where the model is least likely to fail. The harder tests are the ones where the brand is not mentioned and has to be produced. That is also the kind of test worth adding to an evaluation harness for retrieval-grounded features, because it separates recall on popular entities from recall on the tail your users actually care about.

Source

• Search Engine Journal: When AI Has Nothing On Your Company, It Describes Someone Else