Why is asking the team whether AI helps unreliable?

Because perceived speed and actual speed can disagree completely. In METR's July 2025 study, experienced developers took 19 percent longer with AI tools while believing afterwards that they had been about 20 percent faster, so the developers' own feeling failed to detect even the direction of the effect.

What should you measure instead of tool usage?

The four software delivery measures from the DORA research: how long a change takes to reach production, how often you deploy, how often a change fails in production, and how long recovery takes. They describe outcomes the business actually receives rather than activity inside an editor.

Why is measuring accepted suggestions or lines generated a mistake?

Those numbers report that the tool is being used and nothing about whether anything valuable arrived. They also invite the wrong behaviour, because a measure of volume rewards producing more code, and more code is a cost that has to be reviewed, maintained, and eventually understood by someone else.

Why do throughput and stability need to be read together?

Because AI adoption tends to move them in opposite directions, and reading either alone gives a flattering answer. The 2025 DORA report found AI adoption relating positively to delivery throughput and negatively to delivery stability, so a team can release far more while breaking production more often and call it a success.

How long before you can judge whether the tools are working?

Give it two quarters before drawing a conclusion. The first is mostly spent learning where the tools are unreliable and building the checks that make their output safe to accept, which is genuine progress that shows up as almost nothing on a chart, and programmes judged at ninety days are often cancelled just before they would have started working.

What is a fair way to compare before and after?

Compare the same team against its own earlier baseline on the same kind of work, rather than comparing one team against another. Teams differ in codebase age, domain difficulty, and review culture by more than any tool changes, so cross-team comparisons mostly measure those differences.

Does a rise in change failure rate mean the tools are bad?

Not on its own, since it usually means output grew faster than the verification around it. The useful response is to strengthen the automated checks and reduce change size before deciding the tooling is at fault, because the same rise appears whenever any team raises throughput without raising assurance.

What single measure best shows the review process is the constraint?

The time a change spends waiting for review, tracked separately from the time it spends being worked on. When AI writes code faster and the number of reviewers stays the same, the extra work waits for review, and that waiting stays hidden in any single total time figure.

Should individual developers be measured on AI usage?

No, and doing so reliably makes the number wrong. Once a usage figure is attached to a person's evaluation, it stops describing behaviour and starts describing what people believe is being watched, so the measure loses the information it was collected for.

How do you measure quality rather than only speed?

Watch what happens after release: defects found in production, how many changes need a follow-up fix within a week, and how much time goes to unplanned repair. Speed measures are easy to improve by taking less care, and these measures show when that happens.

What if the numbers show no improvement at all?

That is a real result and worth acting on rather than finding excuses for. It usually points to a constraint the tools never affected, most often review capacity or unclear requirements, and the honest response is to fix the actual constraint instead of buying more of something that cannot fix it.

How to measure whether AI is actually helping: Reveneau

The most useful study of AI-assisted development in the past two years had sixteen people in it.

METR ran it in July 2025 with experienced open-source developers, working on real tasks in repositories they already knew well. The result was that they took 19 percent longer with AI tools than without them.

The part that should change how you run your programme is this: before starting, the same developers expected AI to make them 24 percent faster. After finishing, having actually been slower, they still believed it had sped them up by about 20 percent.

Sixteen people is a small sample and the finding is specific to experienced developers in familiar code. Treat the 19 percent figure with caution. Take the gap seriously, because the gap is about measurement rather than about tools. The people doing the work were wrong about the direction of the effect, on their own tasks, immediately afterwards.

Which means the question "does the team feel more productive" cannot answer the question you are asking.

What the dashboards are actually telling you

Every vendor supplies a dashboard. Suggestions accepted, lines generated, sessions, active users, acceptance rate.

Each of those is a real number and none of them is evidence of benefit. They measure that the tool is being used. A tool can be used constantly and produce nothing the business receives, and worse, some of these measures invite the wrong behaviour: lines generated rewards producing more code, and more code is a liability that somebody has to review, run, maintain, and eventually understand.

There is a second reason to be careful with them. The 2025 DORA report, covering nearly 5,000 technology professionals, found 90 percent using AI at work and more than 80 percent believing it had increased their productivity. In the same research, AI adoption showed a positive relationship with delivery throughput and a continuing negative relationship with delivery stability. High confidence and mixed results exist together easily, which is exactly what the METR finding predicts.

Measure delivery instead

The four measures from the DORA research programme are the ones worth tracking, because they describe what reaches customers rather than what happens inside an editor.

How long a change takes to go from written to running in production. How often you deploy. How often a change causes a failure once it is there. How long it takes to recover when it does.

The rule is to read all four together. Throughput on its own always gives a flattering result: a team can release far more and break production more often, and if you only watch the first number the programme looks like a triumph. That combination is common enough that it deserves a name in your reporting, and the honest name is activity without progress.

We have written about the general version of this problem in measuring engineers by outcomes rather than output, and the team-level treatment is in measuring engineering productivity.

The measure most teams are missing

Split cycle time into two parts: time being worked on, and time waiting for review.

AI now writes code fast, while review takes as long as it always did. If the number of people who can competently review has not changed while output has, the extra work ends up waiting for review, and an end-to-end cycle time number hides it inside a single average that rises slowly for reasons nobody can name.

Tracking review wait separately turns a vague sense that things are slower into a specific claim you can act on. It is also the earliest warning available: the queue grows quietly for weeks before anyone calls it a problem, and by then the fastest-merging work is the least examined work.

Quality measures, because speed measures are easy to cheat

Any speed number can be improved by caring less. So track quality measures alongside them.

Defects found in production. Changes that need a follow-up fix within a week, which is the clearest sign of almost-right work reaching users. Time spent on unplanned repair as a share of total engineering time. Security findings by severity, tracked over time rather than once.

That last one deserves specific attention with generated code. Veracode's Spring 2026 testing across more than 150 models found generated code compiling and running correctly over 95 percent of the time and passing security checks in roughly 55 percent, with the failures concentrated in types of mistake that a person reading the code at normal speed does not see: cross-site scripting at 15 percent and log injection at 13 percent. Those defects will not show up in any speed measure, and they will not show up in review either. They show up in a scan, or in an incident.

How to compare fairly

Compare a team against its own earlier baseline, on similar work. Do not compare teams against each other.

Teams differ in codebase age, domain difficulty, deployment risk, and review culture by far more than any tool changes. A cross-team comparison mostly measures those differences and then attributes them to the tooling, which produces a confident conclusion that is about the wrong thing.

Capture the baseline before the rollout. This is the step everyone skips, and without it the entire evaluation reduces to arguing about whether things feel better. If the tools are already deployed, you can often reconstruct a usable baseline from deployment and incident history.

Then give it two quarters. The first quarter is mostly discovering where the tools are unreliable and building the verification that makes their output safe to accept. That is real progress and it looks like nothing on a chart, which is why programmes judged at ninety days get cancelled just before they would have started working.

Do not measure individuals

Attach a usage number to someone's performance review and it stops describing their work. It starts describing what they believe is being watched.

This is not a hypothetical risk with these tools, because usage is so easy to inflate and so weakly connected to value. Measure teams and systems. Use individual data for coaching if at all, never for evaluation.

When the answer is no

Sometimes the measures show no change, and that is a finding rather than a failure of the measurement.

It usually means the constraint was in a part of the work the tools could not affect. If the slowest step is review capacity, unclear requirements, or a slow and unreliable process for getting to production, faster code generation adds more work to that step instead of reducing it. That is the reason behind the DORA report's direct summary that AI makes a team's existing strengths and weaknesses larger, and we have written about the pattern in AI is an amplifier, not a fix.

The useful response is to fix the actual constraint. Usually that means verification that can be trusted, so reviewers stop checking things a machine could check: the approach is in how to write your first eval suite.

Thanks to the teams who let us look at their numbers, including the ones where the honest answer was that nothing had improved yet. Those were the more useful conversations, because every one of them ended up pointing at a constraint that had been there long before any tool arrived.

Sixteen developers were 19 percent slower and felt 20 percent faster. Whatever you decide to measure, make sure it could have caught that.

Related guide: Metrics for AI code quality.

Sources

METR, July 2025 randomised study of experienced open-source developers: 16 developers took 19 percent longer with AI tools, having expected a 24 percent speedup and believing afterwards they had been 20 percent faster.
2025 DORA report announcement, Google Cloud: nearly 5,000 respondents; 90 percent use AI at work, over 80 percent believe it raised their productivity, and adoption relates positively to throughput and negatively to delivery stability.
Veracode, Spring 2026 GenAI Code Security update: over 95 percent syntactic correctness against roughly 55 percent security pass rate, with cross-site scripting at 15 percent and log injection at 13 percent.
DORA research programme: the four software delivery measures.

How to measure whether AI is actually helping