AI2 says frontier models hit 13 percent tracking long documents

Frontier AI models break down when a fact inside a long document keeps changing. The Allen Institute for AI released a benchmark to measure the gap. It is called StateBench.

AI2 posted the work on Hugging Face. Kyle Wiggers wrote the blog. A tech report, the dataset and the code are all linked from the post.

What the benchmark tests

StateBench asks a model to track one thing as it is updated across a long document. The example in the paper is a mystery novel. "John gave Mary the knife" changes the state of both people. A harder variant hides the knife one step further back: Edward put the knife in John's backpack, then John gave Mary the item in his backpack. The model has to follow the chain to answer who holds the knife.

AI2 splits the task into four levels. The easiest is r-trivial, where a value is assigned once and never changes. Aperiodic needs the model to find the latest value. Soluble has self-referential updates like x = x + 5. The hardest is general finite, with assignments by reference such as x = y and variable swaps.

What the frontier models scored

AI2 tested Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol and GPT-6 Luna. All four did well on the two easier levels. On the two harder levels, the post says accuracy fell sharply as the input grew longer. It dropped to 13 percent or less on the longest general finite problems, AI2 reports.

AI2 reports the drop as a group result across all four models. The post names the two levels where the models break. A reader can match that to their own product.

What a small, different model could do

AI2 also trained small models on the same task. It then tested them past their training length. One model, Gated DeltaNet, has more flexible memory updates. It stayed near perfect out to about 10 times its training length. AI2 trained it on 256 assignments. It held up on 64,000. The transformer models AI2 tested did not learn the hardest level at all, no matter how long AI2 trained them.

The unit is assignments, meaning the number of update steps. 64,000 assignments is a document with 64,000 updates to the variables being tracked.

How to try the same test on your own product

A long document and a shifting document are different problems. Chat histories, case files, codebases and contracts all carry facts that get overwritten. AI2's result says a production model may read the whole thing. It may still answer with the stale fact.

Write a small test on the product. Pick a document the system has to read. Change one fact three or four times across it. Ask for the final value. Measure how often the system gets it right as the document grows. The AI2 result shows what kind of failure to watch for. The test above measures it in the product's own setting.

Source

Hugging Face, with the tech report, the dataset and the code.