Ai2 found 16 AI benchmarks reduce to two underlying dimensions

The Allen Institute for AI published a method on 1 September called BenchMIRT, which applies multidimensional item response theory to language model benchmarks. Item response theory comes from educational testing, where it is used to work out what a question is actually measuring rather than just counting how many students got it right.

Ai2 applied it to benchmarks instead of students. The post states the scale of the run: "We trained BenchMIRT on benchmarking results from 100 LLMs across 16 benchmarks and more than 34K questions." Six of the benchmarks target general reasoning, including MMLU-Pro, GPQA, MATH and BBH, and ten come from a safety suite including HarmBench, StrongReject, WildJailbreak, BBQ, WMDP and XSTest.

The two findings

The first is about what the benchmarks measure. Ai2 reports that the method "independently recovered two dominant dimensions: safety and general reasoning." Sixteen separate benchmarks, given no labels in advance, sorted themselves into two underlying abilities.

The second is about how many questions are actually needed. Ai2 writes that "Keeping only 10% of the questions generally preserved nearly the same picture," and adds that "Keeping 50% of the questions often matched the full benchmark's measure even more closely."

On predictive accuracy the post reports that BenchMIRT "correctly predicted whether a model would answer a held-out question correctly 79% of the time," against 70% for a simpler approach.

These are Ai2's own results, published by Ai2, and no independent group has repeated them. Ai2 links a technical report, the datasets and the code, so the run is checkable in principle.

Cases that never disagree are one case

Few teams run MMLU-Pro. Almost every team building on models runs an eval suite of its own, and that is where this applies.

The finding to think about is that a set of tests written to cover different things can end up measuring the same small number of underlying abilities. An eval suite built up one case at a time, which is how they are all built, is likely to have this problem. Adding a case feels like adding coverage, and the honest question is whether the new case ever disagrees with cases already in the suite.

That question is answerable without item response theory. Take the pass and fail record of every case across every model or prompt version you have evaluated, and look for cases that always agree. Two cases that never disagree across a run of releases are one case, and one of them can be removed.

The subset result supports the same conclusion from the cost side. If a tenth of the questions give nearly the full result, an evaluation run is mostly repetition, and repetition is what makes a suite too slow to run on every change. A faster suite that runs on every pull request catches more than a thorough one that runs weekly.

The caution is that neither result says a benchmark is worthless. It says the benchmarks overlap, so a table of sixteen scores is reporting two things, repeated across sixteen scores.

Source

Allen Institute for AI, "BenchMIRT: What are LLM benchmarks actually measuring?"