Amazon Science raises LLM-as-a-judge accuracy by 9 to 14 points by modelling how the judges copy each other

Amazon Science published a post on 26 August 2026 showing that agreement across a panel of LLM judges is often a shared mistake, and reports 9 to 14 point accuracy gains on three benchmarks by treating the panel as a network of correlated voters. The work appears in a paper by Krishna Balasubramanian, Sasha Podkopaev and Shiva Prasad Kasiviswanathan at ICML 2026, titled Dependence-aware label aggregation for LLM-as-a-judge via Ising models.

What the paper measures

The setup is a common one: ten LLM judges score the same output, and an aggregation rule turns those ten labels into one. Two conditional-independence baselines are compared. The uniform majority vote weights every judge equally. The weighted majority vote weights each judge by past accuracy. Both assume each judge decides on its own.

Amazon's team argues that assumption is wrong in practice, and quotes the case directly: "If the eight agreeing judges are genuinely different sources of evidence, then agreement is a strong signal. But if they share a prompt template, a training lineage, a model family, or a common blind spot, they may be repeating the same mistake." The ten judges in the experiments were run at temperature zero, so repeated agreement between any two of them is a property of the models and their prompts.

The Ising fix

The paper models the panel as an Ising network. Each judge is a binary variable, each edge is a pairwise dependency between two judges, and the parameters cover both individual reliability and how often each pair agrees more than their solo accuracies would predict. The estimator learns these from the labels themselves and needs no reference answers, which is the setting most eval harnesses actually run in.

On three binary tasks with 10-judge panels, Amazon Science reports that dependence-aware aggregation beats the weighted majority vote on each one: relevance classification 0.912 against 0.820, toxicity detection 0.792 against 0.694, and summarisation assessment 0.806 against 0.737. The 9 to 14 accuracy point gain comes from swapping only the aggregation step; the judges and their labels are the same across both estimators.

Why this shows up in real evals

A team building a retrieval system that scores its retrieved passages with a panel of GPT-family judges is exactly the scenario the paper describes. The judges share tokenisation, training data, and often the same system prompt, so a passage that any one judge miscalls as relevant is the passage every judge is likely to miscall the same way. A ten-to-zero split then carries less information than it looks like it does: the dependence-aware estimator, once trained, can read those ten labels as a single opinion copied nine times when the measured pair correlations are strong enough.

The cost of adopting this is one extra parameter per judge pair, estimated from the same labels the harness already collects. In return, a panel of near-copies stops drowning out the one judge that saw the mistake.

Source

When LLM judges agree, should we believe them?, Krishna Balasubramanian and Sasha Podkopaev, Amazon Science, 26 August 2026. Paper: Dependence-aware label aggregation for LLM-as-a-judge via Ising models, Balasubramanian, Podkopaev, and Kasiviswanathan, ICML 2026.