A benchmark of six LLM reliability layers found they sort records, not fix them

Most teams stack a second AI layer on top of a first one when the answers are wrong too often. A paper on InfoQ puts six of those layers on a benchmark. It reports that none of them recovered answers the first pass had missed. The authors, Pushpdeep Mishra and Wejdan Bagais, write: "Layers sort; they do not fix."

The test uses CADEC, the CSIRO Adverse Drug Event Corpus. It is 1,250 forum posts about drug side effects, with spans labelled and mapped to SNOMED CT codes. The baseline is a zero-shot open-weight model that tries to pull drug mentions out of the posts. It gets about one answer in three correct.

The authors then add, one at a time, six reliability layers. The first is a vocabulary check against SNOMED CT. The others are self-correction, voting across three samples, a smaller judge model at 3.2 billion parameters, refusal, and human review.

What each layer did

Voting changed about 27 codes per run. It broke as many correct answers as it fixed. The net change on F1 was between minus one and plus one.

Self-correction fired two or three times across 40 documents. It corrected nothing when it fired. The vocabulary check let 21 to 23 percent of records through, at 0.74 to 0.82 accuracy. The judge model sorted the records by trust. It did not restore the mentions the first model had already lost.

Refusal was the only move that lifted the accuracy of what the system released. It did so by holding back 77 to 79 percent of records. The authors call this the difference between accuracy and yield. An accuracy figure can climb while the system answers fewer questions. They write: "No layer recovers a mention the extractor missed."

Recognition beats generation

The paper also notes a separate finding on the first pass. When the model had to recall SNOMED codes from memory, it invented 13 to 18 percent of them. When it chose from a menu of candidate codes, the accuracy improved about tenfold.

The layers tested after that only sort what the model already produced. The biggest single gain came from changing how the model was asked.

The authors tested five models. The only one that varied across three identical runs at temperature zero was gpt-oss:20b, by four points of F1. The other four were llama3.1:8b, mistral:7b-instruct, qwen3:8b, and granite4:micro-h. The supervised baseline, CONORM, reached 0.72 F1. The zero-shot models reached 0.47 to 0.50.

A team that already runs a model-backed feature should read this as a reframing. The second layer does not fix the first. It chooses which of the first layer's answers to keep and which to withhold. Any answer the first pass missed stays missed. So the useful question before adding a layer is not how much more accurate it makes you. It is how much coverage you are willing to lose to raise the trust on what you keep.

Source

• Pushpdeep Mishra and Wejdan Bagais, LLM Reliability Layers Under Test: Accuracy, Yield, and the Limits of Verification, InfoQ.