When evals give false confidence

Every team that has run a real incident review has had this moment: the pipeline passed, the change was released, the customer found the bug. The suite was answering a different question from the one everyone thought it was answering.

Here are the six ways that happens.

1. The assertion that cannot fail

The most common by a wide margin. The check runs, the check passes, and there is no situation in which it would fail.

The mock returns exactly the value the assertion expects, so the real code path is never exercised. The setup fails without any error and the test asserts on an empty result. The assertion compares a variable to itself after a round trip. The try block catches and hides the error and the test only checks that nothing was thrown.

The sign: make it fail on purpose. Break the implementation the check is supposedly protecting and confirm the build fails. Do this for every check when it is written and spot-check a handful whenever you take over a suite. In an existing codebase this exercise usually finds something within the first ten minutes.

2. Flakiness

A check that fails one run in twenty does more damage than it prevents. The main damage comes from what the team learns: a failure might not mean anything, so re-run it. Once that habit exists, it applies to every check, including the one that was correct.

The sign: ask how often people re-run a build without changing anything. If the answer is more than rarely, you have this problem, and the fix is to remove the unreliable checks from the blocking tier today rather than to schedule a cleanup.

3. The suite that describes the implementation

Checks that assert internal structure rather than external promises break on every refactor. The team responds rationally by updating the tests to match the new structure, which means the suite is being rewritten to agree with whatever the code now does. At that point it only copies what the code does and has stopped being a check.

The sign: what happens to your suite during a refactor that changes no behaviour? If a large share of checks need editing, they were describing the code rather than the requirement.

4. Written by the same pass that wrote the code

If one run produces both implementation and tests, the tests encode the implementation's interpretation of the requirement, including its misunderstandings. They will pass. They prove internal consistency, which is not the property you wanted.

Anthropic's own guidance points to the fix: a fresh context or a separate reviewer so the thing doing the work is not the thing grading it, since a fresh context is not biased towards code it just wrote [1]. Our stronger version is to derive the check from the specification before the implementation exists.

The sign: read the check and ask whether it would still be the right check if the feature had been built a completely different way. If not, it came from the code.

5. Everything tests only normal use

The suite covers the flows people demo and none of the ones that break. No empty collections, no expired sessions, no duplicate submissions, no unauthorised caller, no dependency timing out.

This matters more for generated code because its failures tend to be of this kind. The most cited developer frustration in Stack Overflow's 2025 survey, at 66 percent, was output that is almost right but not quite [2], and almost right nearly always means correct in normal use and wrong in an unusual case.

The sign: count the negative cases in your suite. If fewer than a third of your checks assert that something is refused, rejected, or handled as an error, the unusual cases are not covered.

6. Nothing has failed in a month

A suite that never fails is either protecting perfect code or protecting nothing, and the passing results cannot tell you which. In a team releasing real changes, a healthy suite catches something regularly, and each catch is the system doing its job.

The sign: look at the last thirty days of pipeline history. If there are no genuine failures, go and break something to check that the connection still works. It is a strange thing to have to verify and it is worth verifying.

Why this is the real risk

Adding evals badly is a specific kind of expensive. You pay the cost of building and maintaining the suite, you lose the caution that a team without checks naturally has, and you gain a signal that says everything is fine. Teams in that state release faster with less scrutiny, which is the worst possible combination if the checks test nothing.

So audit for trust rather than volume. A suite of fifteen checks that have each caught something real is worth more than 2,000 that have never failed. And when your engineers say they do not fully believe the passing build, take that seriously, because they are usually right and they are describing the failure mode nobody else can see. The metric side of that is in metrics for AI code quality, and the pipeline practices that prevent most of this is in evals in CI for coding agents.