DecoyCount benchmark finds every frontier model below 50 percent on counting

A paper on arXiv on 26 October 2026 reports a new counting benchmark called DecoyCount. The authors are Qiuyi (Richard) Zhang, Andrew M. Dai and 14 others. Dai is the CEO of Elorian, a vision-reasoning startup. The benchmark was co-released with the AI data company Mercor.

The task sounds simple. The reader looks at a photo and counts the apples, or the chairs, or the people. The benchmark has 1,660 real-world photos. Each photo has at least one decoy. A decoy is something that looks like the target but should not be counted. A reflection of an apple is not an apple. A drawing of an apple is not an apple. The paper names seven kinds of decoy.

What the benchmark found

The paper reports accuracy for six named frontier models. Every one scored below 50 percent. GPT-6 Astra was the best at 43.17 percent. The others, as the paper gives them: Gemini-3.8-flash at 32.71 percent, Qwen3.8-max at 28.31 percent and Claude-Fable-5.1 at 25.22 percent. Each number is a three-run average, under the model's own reported settings. The paper's conclusion is that stronger general reasoning does not by itself produce aligned visual reasoning.

The paper also reports a reasoning-alignment gap. The authors give the model two prompts for the same photo. The first is a neutral question. The second gives a cue about the decoy to watch for. The gap is the accuracy change between the two. For the frontier models, the gap ranges from 11.8 to 22.1 percentage points. The models can count correctly when told what to ignore. They often fail to ignore the decoy on their own.

The caveat the paper is honest about

More capable models and larger thinking budgets did not close the gap in the authors' tests. The paper reports that they often made the gap wider. Agentic harnesses, where a model takes several turns and uses tools, also did not fix it on their own. A targeted vision tool, pointed at the decoy mechanism, did narrow the gap.

Where to read the vendor's own figure with care

Elorian's own model, Glimpse, is not in the paper. CEO Andrew Dai told Latent Space that Glimpse scored 50.48 percent on the 1,660 items in an internal test. That number is Elorian's own figure. It comes from a single run of its model without a harness. The six frontier numbers are three-run averages. The two measurements are not a published head-to-head.

Latent Space quotes Dai saying the evaluation images and text were withheld from Glimpse's training. Dai also says a contamination check found no overlap. Both claims are Elorian's own, with no outside audit cited.

What a team building a vision feature should take away

A product that counts things from a photo should not assume frontier accuracy. The paper's numbers come from a static benchmark with real photos and designed decoys. A product's own photos may be easier, or harder. The useful move is to copy the paper's method on a small set of the product's own images. Build a few decoy versions. Measure the gap between the neutral prompt and the cue-guided prompt. The gap size says how much of the error is perception and how much is prompt framing.

Source

DecoyCount: Evaluating Visual-Reasoning Misalignment in Counting, by Zhang et al., arXiv. Can Frontier Models See Past a Decoy? Elorian Tests Visual Reasoning, by Richard MacManus on Latent Space, has the Glimpse figure Dai supplied.