Image: Search Engine Journal
Why it mattersA tracking tool that queries the API or Google sees about half of the sites ChatGPT cites, and about a quarter of the exact pages, so a report that only counts page matches understates how often the brand is picked.
A rank tracker that asks the OpenAI API what ChatGPT would cite finds roughly a quarter of the pages ChatGPT actually quotes, and about half of the sites behind them. The arXiv preprint "Understanding ChatGPT Citations", posted by Yunxuan Jiang and colleagues, recorded what ChatGPT's web application fetched during 80 questions run twenty times each, then compared those citations with what Google results and the OpenAI Responses API returned for the same prompts. Matt G. Southern at Search Engine Journal wrote up the paper.
How the test was run
The authors recorded the data flow their signed-in ChatGPT sessions received, so they saw every search ChatGPT performed, every page returned, and every page it picked as a source. They used 80 questions from Search Arena, with 40 in English and 40 in Chinese. Each question ran twenty times on GPT-5.3 Instant and GPT-5.5 Instant, across four request types: recommendation, fact lookup, information synthesis and analysis. Within three days of each ChatGPT run, the authors fetched up to five pages of Google organic results per question through SerpAPI, averaging 43 results, and sent twenty queries to gpt-5.3-chat-latest and gpt-5.5 through the Responses API with web search on.
The paper's own numbers
The authors report that Google results and the Responses API combined covered 28.3 percent of the pages GPT-5.3 Instant cited and 26.2 percent of the pages GPT-5.5 Instant cited. For sites, treating subdomains as separate, the same combination covered 50.3 percent and 48.7 percent. Google alone accounted for 9.4 and 6.4 percent of cited pages, and 26.1 and 22.8 percent of cited sites. The Responses API's own answers cited a tighter set, but the pages it quoted matched ChatGPT more often: GPT-5.5 Instant and the API agreed on 23.3 percent of pages. The paper calls a page cited if ChatGPT selected it at least once across the twenty runs for that question.
ChatGPT rewrites the prompt and the sources shift between runs
The paper also reports that 44.5 percent of the distinct pages a GPT-5.5 Instant question returned appeared in only one of its twenty runs, and 40.8 percent for GPT-5.3 Instant. The overlap was higher at the site level than at the page level. On Chinese-language questions, GPT-5.3 Instant used only English search terms in 67.0 percent of visible searches, and GPT-5.5 Instant did so in 69.6 percent. Pages that appeared in more runs for a question had a higher chance of being cited, though the authors note this does not prove that frequency drives selection. The paper is a preprint and arXiv does not peer review.
What this changes for a team tracking AI visibility: a tool that reports page-level matches between ChatGPT and the API is reporting a small share, by design, and a monitor that only runs a prompt once will miss most of what ChatGPT might cite for it. The authors suggest watching at both the site and the page level, running the same prompt multiple times, and checking whether a tracking tool pulls data from the API or from the ChatGPT application. They did not test any commercial tracker.
Source
The preprint is on arXiv at arxiv.org/abs/2610.15382. The write-up by Matt G. Southern is at Search Engine Journal.
This item was written by an AI system from the linked source. Reveneau is responsible for what it publishes.
Get AI News in your inbox
New developer tools, model and agent releases, and how teams are actually using them to release software. Short, and only when there is something worth reading.
