Measure delivery, not how much code was generated

The first question a board asks about AI adoption is what it achieved. The easiest numbers to produce are the worst ones to answer with: lines generated, pull requests opened, tokens spent, tools adopted. All of them measure activity, and activity was never the slowest step.

What to measure instead

DORA's current model uses five metrics, divided into two dimensions. Throughput is change lead time, deployment frequency, and failed deployment recovery time. Instability is change fail rate and deployment rework rate, where rework counts unplanned deployments that happen as a result of a production incident [1].

Those five are worth adopting as they are, for two reasons beyond their quality. They are defined by someone outside your company, so nobody can redefine one, without others noticing, when it looks bad. And they describe the delivery of working software rather than the production of code, which is the distinction this whole page exists to protect.

To those five we add one that DORA does not name and every AI-native team needs: rework rate at the change level, meaning the share of merged changes that come back for correction within two weeks. It is the clearest signal of whether generated work is actually finished, and it is usually the first number that moves when a team tightens its definition of done.

Why code-output metrics mislead

Three specific ways, all of which we have watched happen.

They measure the cheap part. Producing code is no longer the slowest step. Reporting how much of it exists is reporting on the resource you have plenty of.

They are easy to manipulate, and get manipulated without anyone deciding to. An engineer who knows pull request count is being watched will open more pull requests. Nobody has to be cynical for this to happen; people optimise for what is measured because that is what measurement does.

They draw attention away from the real limit. If review capacity is your limit, more generated code makes the queue worse, and a rising count of generated code looks like progress while delivery slows. That is the pattern behind our post on the review queue is the new bottleneck.

Do not trust how fast it feels

There is a specific and well-measured mistake here. METR ran a randomised controlled trial in 2025 with 16 experienced open-source developers across 246 real issues in repositories they knew well. The developers took 19 percent longer when allowed to use AI tools. They estimated afterwards that AI had made them 20 percent faster, and had forecast a 24 percent speed-up before starting [2].

That is a gap of 39 percentage points between measured and perceived, in the direction of optimism, among experienced engineers on familiar code. METR is explicit that this does not show AI fails to speed up most developers, and the setting matters: mature codebases the participants knew deeply [2].

The lesson for a leader is narrow and important. Do not base adoption decisions on how the team says it feels, in either direction. The engineer who says AI made them twice as fast and the one who says it wastes their time are both reporting a perception that the only controlled study we have found to be wrong by a wide margin.

Expect the two groups of numbers to move apart

When a team adopts AI seriously, the normal early pattern is throughput up and stability down. DORA's 2025 report of nearly 5,000 professionals found exactly that pattern: a positive relationship between AI adoption and delivery throughput and product performance, and a negative relationship with delivery stability [3].

Read that as a diagnosis rather than a warning. It means change volume rose faster than the control systems around it, so the next investment is in the controls: automated testing, release practice, feedback speed. Google's summary names those directly [3].

The failure is reporting only one dimension. A leader who reports throughput sees a triumph. A leader who reports incidents sees a disaster. Both are looking at the same team, and only the pair together tells you what to do next.

A reporting set that fits on one page

For a monthly or quarterly report, six numbers and one paragraph.

Change lead time and deployment frequency, for throughput. Change fail rate and failed deployment recovery time, for stability. Rework rate, for whether work is finished when it is called finished. And one product or business number that the engineering work was supposed to move, because a team can be excellent on all five and still be building the wrong thing.

The paragraph is what the numbers do not show: which constraint you are working on now, and what you expect to move next quarter. Numbers without that look like a list of scores rather than a management plan.

What not to do with these numbers

Two failure modes, both common enough to name.

Do not compare teams. Change fail rate on a payments service and on an internal dashboard are different numbers describing different levels of acceptable risk, and ranking them leads people to manipulate the numbers instead of improving. Compare a team to its own trend.

Do not report a single number as a verdict. Throughput without stability shows only how fast you are producing risk, and stability without throughput shows only how safely you are making no progress. Report the two together, and reporting one of them alone is the most common way these metrics get used to argue for something that was already decided.

Do not target individuals. The moment a metric is attached to a person's review, it stops measuring the system and starts measuring the person's response to being measured. Our post on measure engineers by outcomes, not output makes the argument, and it is more urgent now that individual output figures are easy to produce.

Getting the data without starting a platform project

The usual objection to all of this is that the team does not have the instrumentation to produce the numbers. In practice you can get every one of them from systems you already run, in an afternoon, and a simple version is enough to see a trend.

Deployment frequency comes from your deployment tooling or, failing that, from tags in version control. Change lead time is the gap between the first commit on a change and the deployment that carried it, both of which are already recorded. Change fail rate needs one convention, not a system: mark the deployments that required immediate intervention, in a field, at the time. Failed deployment recovery time is the gap between that mark and the deployment that fixed it. Rework at the change level needs one label applied during triage.

The only real requirement is that someone marks incidents and corrections consistently, which is a habit rather than a tool. Teams that wait for a platform to produce these numbers perfectly usually wait for a year and manage from single stories in the meantime, which is worse than managing on approximate figures that are honest about being approximate.

One caution: whatever you build, keep the definitions stable. A metric whose definition changes is a metric with no history, and history is the whole value here, since the comparison that matters is against your own past rather than against anyone else.

What a board actually asks

The question from outside engineering is rarely about lead time. It is some version of: is this working, and what did it give us.

Three sentences answer it honestly. Here is what we released and how often, from the throughput pair. Here is what it cost us in stability, from the fail rate and recovery time. Here is the constraint we are fixing next, and the number we expect to move.

That answer holds up under close checking because it does not claim more than the data supports, and because it treats a stability cost as a fact to manage rather than a failure to hide. The alternative, presenting throughput alone as evidence that the adoption succeeded, holds up exactly until the first bad quarter of incidents, at which point every number you presented becomes suspect at once.

Cost belongs next to delivery, not inside it

Token spend and tool licences are real costs and worth tracking, but they are inputs and they belong in the same conversation as headcount rather than in the delivery metrics. The useful framing is cost per delivered change over time, which is covered in budgeting agent capacity and cost. Mixing spend into a delivery scorecard makes it too easy to celebrate a cost reduction that came from releasing less.

Where this connects

These numbers tell you which part of the model to work on: rework leads you to the definition of done, change fail rate to the gates and release practice, recovery time to observability. The code-quality counterpart, measuring the software rather than the delivery, is metrics for AI code quality. The main guide shows how it all fits together.