Metrics for AI code quality: what to watch instead of volume

A team we talk to often had a dashboard showing pull requests merged per week, and it had roughly tripled. Everyone knew things felt worse. Nothing on the dashboard could show it, because the dashboard measured the part of the job that a machine had taken over.

Here is what we watch instead.

Start with the two DORA numbers about failure

Google's DORA research has studied software delivery for over a decade, and its four key measures are the most validated set available: deployment frequency, lead time for changes, change failure rate, and failed deployment recovery time. The program's own view is that speed and stability are not a trade-off, and that the strongest teams do well on all of them at once [1].

Of the four, the two failure-side numbers are the ones that answer the question this guide is about.

Change failure rate. The share of changes to production that cause a degradation needing a fix, a rollback, or a patch. If your evals are doing their job, this falls even as your deployment frequency rises. If it climbs while volume climbs, you are releasing work that has to be redone and calling it progress.

Failed deployment recovery time. How long it takes to get back to working after a bad change. This is a measure of your ability to detect and reverse, which is exactly the ability an eval suite builds.

The reason to focus on these specifically is that the 2025 DORA report found AI adoption has a positive relationship with throughput and a negative relationship with delivery stability across nearly 5,000 respondents [2]. That is the pattern to watch for in your own numbers, and you cannot see it if you only report the throughput half.

Then add the two that are specific to this problem

Escaped defects. Count the issues found by a customer, or by production monitoring, rather than by your own checks. This is the closest thing to a direct measure of eval quality, and it is more honest than any coverage number because it counts what got through rather than what was attempted.

Track the ratio as well as the count: of the defects you found last month, what share did your suite catch first? A team improving its evals sees that share rise. A team with a large but weak suite sees no change and a lot of passing builds.

Rework rate. The share of changes that modify something released in the previous 30 days. Some rework is healthy iteration, so the number is not meant to be zero. A sharp rise usually means work is being merged before anyone knows whether it was right, which is the exact symptom of code being generated faster than it can be verified.

Two that sound useful and are not

Test coverage. It counts lines that executed, whether or not any behaviour was checked. It is possible to hold 90 percent coverage while proving nothing, and teams that aim for the number reliably reach it. Use it as a rough map of untouched areas, never as a target.

Eval pass rate. A suite passing at 100 percent tells you either that your code is correct or that your checks are weak, and the number cannot distinguish those. Watch what fails and how often instead, and be suspicious of a suite that has not failed in a month.

The measure nobody puts on a dashboard

Ask your engineers this question every quarter: when the pipeline passes, do you believe the change is safe?

The answer is the most predictive thing we know and it does not appear in any tool. A team that says yes will use the suite to block bad changes and let the agent work. A team that says no will re-run builds, review defensively, and rely on manual checking without saying so, which means the whole investment is producing formality instead of speed. If the answer is no, fix the trust problem before adding more checks. When evals give false confidence is about how that trust gets lost.

How to report it honestly

Two rules we hold ourselves to.

Never show throughput without stability next to it. A chart of changes released, with no change failure rate beside it, reports typing speed rather than engineering health, and typing is the part that got cheap.

Label a scoped target as a target. If a number on a page is a timeline you plan to hit rather than an average you measured, say so in the same sentence. We follow that rule on our own site because a company selling verification cannot be careless with its own claims, and it is worth following on an internal dashboard for the same reason.

For the business framing of these numbers, see the business case for evals. For measuring engineers rather than systems, our position is in measure engineers by outcomes, not output.