Ai2's FairStress finds frontier LLMs abandon correct answers when challenged

A fluent language model picking between two job applicants, two loan files or two triage tickets looks neutral on its face. The Allen Institute for AI measured what the same model does when somebody argues back at its answer, and the result is not neutral in either direction.

Ai2 released FairStress, a benchmark for demographic fairness in large language models that goes beyond asking whether a model picks the right candidate. It asks whether the model holds that answer when a persuasive prompt argues against it, and whether the answer moves differently depending on who benefits. The authors say the goal is "symmetric fairness": when the evidence supports a decision, a model should apply the same criteria and be equally resistant to changing that decision regardless of which demographic group benefits.

What the benchmark actually tests

FairStress is 300 two-candidate decision scenarios across 29 domains including hiring, lending, healthcare, education, housing and criminal justice. Each scenario has a version with no identity information, a version with demographically coded names, a version with implied identity and a version with explicit labels. For each one, FairStress attaches matched pairs of adversarial prefixes: a persuasive argument favouring candidate A and a matched one favouring candidate B. Combining the demographic variations with the prefixes expands the 300 scenarios into more than 13 million test cases. Ai2 also publishes FairStress-Core, a 50,000-item subset chosen with item response theory, and ran that one against the frontier models.

The finding, and the direction it points

Across 12 model-and-mode runs on FairStress-Core, which included Claude Opus 5, GPT-5.5, Llama-3.1-70B-Instruct, Gemma-3-27B-it and Qwen2.5-7B-Instruct, the models were 8.8 to 42.4 percentage points more likely to abandon a correct decision when the switch favoured a minority-coded candidate under a matched adversarial prefix. On ambiguous cases where neither candidate was clearly better, the frontier models selected the minority-coded candidate 60 to 65 percent of the time. Ai2 calls this pattern "anti-stereotypical over-correction": a model is more willing to abandon the evidence when doing so moves the decision toward a minority group.

The gaps grow with how obvious the identity is. With indirect identity signals, correct-answer abandonment differed by 6.9 percentage points on average; with explicit labels, 9.3.

Why BBQ and FairStress point in opposite directions

Ai2 also ran the same models on BBQ, an earlier fairness benchmark, and got the opposite reading. BBQ scores the probabilities of fixed answer options without having the model generate a response, which Ai2 says differs from how a chatbot is actually used. When they repeated the test with generated answers in five configurations, Qwen2.5-7B-Instruct, Gemma-3-27B-it and Llama-3.1-70B-Instruct shifted from stereotype-consistent in BBQ to minority-favouring in FairStress. Relying on fixed-option scoring alone can miss over-correction in generated decisions.

Ai2 states the scope of the benchmark upfront. FairStress measures symmetric fairness, which is one of several conceptions of fairness in the literature. Nine debiasing techniques the team tried, including self-critique, counterfactual chain of thought and activation steering, each cut the gap on some axis at the cost of accuracy or robustness elsewhere.

For a team using an LLM for ranking, triage or decision support, the dataset, the code and the Olmo-eval integration are open, which means the test the paper describes is one a team can run against its own prompts and its own candidate rubric. Treating a fluent answer as a fair one is now a measured mistake with a dated reference.

Source

• Ai2 on Hugging Face, FairStress: A deeper evaluation of fairness in LLM decision-making.
• Ai2 technical report, FairStress.
• Dataset on Hugging Face, allenai/fairstress.