Releasing AI-written changes: flags, canaries and rollback

The numbers behind releases changed and most release processes did not. If one in twenty changes causes a problem, twenty changes a week means one bad change a week. Eighty changes a week means four. Your customers experience the total, not the rate.

DORA's 2025 report of nearly 5,000 professionals found AI adoption associated with higher delivery throughput and lower delivery stability, and attributes the instability to change volume meeting weak control systems: automated testing, version control practice and feedback speed [1]. Progressive delivery is the control that sits closest to the customer.

Deploy is not release

The single most useful separation is between getting code onto the servers and turning it on for users. Once those are different events, deployment becomes routine and reversible, and the risky moment moves to a switch you control independently of your build pipeline.

That is what makes flags worth their cost. Not the ability to run experiments, though that is nice, but the ability to answer "make it stop" in seconds without a build, a queue, or a person who knows the deploy tooling being awake.

Flags need their own upkeep, and an AI-native team adds them fast. Every flag is a branch in behaviour, and a codebase full of old, unused flags is a codebase where nobody can say what runs for whom. Give each flag an owner and a removal date at creation, and treat removing a flag as part of the work rather than as cleanup that happens later. In practice the flag count is a useful measure of how much unfinished work is in progress.

Exposure in stages

The pattern is old and it works: internal users, then a small percentage, then a larger one, then everyone. What changes in an AI-native team is that you are running this cycle far more often, so it has to be cheap and mostly automatic.

Three details make the difference between real progressive delivery and something that only looks like it.

The stages must be small enough to matter. A first stage of ten percent is useless if the failure only shows up for the one percent of users with a particular kind of data. Where possible, stage by group of accounts rather than by random percentage: the accounts most likely to expose the failure first.

Someone must be watching the right signal. A canary release (a first release to a small group of users) with no comparison is just a slower release. You need the error rate, latency and the business metric for the users who have the change next to the same numbers for everyone else. That instrumentation is the subject of what to instrument when nobody typed the code.

Promotion has to be a decision, not a timer. "It has been an hour and nobody complained" is the most common reason teams give for widening a release, and it is close to worthless: the absence of a complaint measures how quickly your users report problems, not whether the change is working. Name the signal and the threshold before you start.

What a rollback plan actually contains

Most rollback plans are one sentence: revert the deploy. That works for stateless changes and fails for everything else, which is why it is worth writing down four specifics per risky change.

The trigger. What condition means we stop, stated as a number where possible. Deciding this in advance is what prevents the twenty-minute debate that happens while the error rate is climbing.

The mechanism. Flag off, deploy revert, or something else. If it is something else, that is the sentence worth writing.

The data question. Whether anything was written that will still be there after the code is gone. This is the part that turns a rollback into a project, and it is why data changes are sequenced separately in decomposing work for coding agents.

Who can start it. Ideally anyone on call, without needing a second approval. A rollback that requires finding the author is a rollback that takes forty minutes.

Then practise it. A plan that has never been run is an assumption. Running a real rollback on a quiet Tuesday tells you which of your four answers was wrong while it is cheap to find out. Our post on what a rollback plan looks like for AI-written features has the format we use.

Why AI-written changes need this more

Two reasons beyond volume.

The first is that the person who can diagnose the change quickly may not exist. When a human wrote the code, the fastest debugging path was often the author's memory. With generated code that knowledge is harder to find, so the value of being able to turn the change off without understanding it goes up. Under time pressure, rollback replaces understanding, and in an AI-native team you will use it more.

The second is that the pattern of failures is different. Generated code tends to be competent on the path it was asked about and weak on the paths nobody mentioned, which means failures gather in unusual inputs, unusual accounts, and unusual load. Those are exactly the cases a staged rollout finds and a full release does not, because at one percent exposure you get the unusual cases at a rate you can handle.

Release size and cadence

There is a temptation, when volume rises, to batch releases: fewer, larger deploys to reduce the number of risky moments. It reliably makes things worse. A large release fails in a way nobody can attribute, and the rollback takes everything with it, including the twelve unrelated changes that were fine.

Smaller and more frequent is the safer direction, and it is what makes the gates in the quality gates that should block a merge worth having. The pipeline's job is to make each individual release boring enough that doing it fifty times a week is not a source of anxiety.

What to measure about releases

DORA's current model splits five metrics into throughput and instability: change lead time, deployment frequency and failed deployment recovery time on one side, change fail rate and deployment rework rate on the other [2]. For release practice specifically, the two worth watching weekly are change fail rate and failed deployment recovery time, because they answer the two questions a leader actually has: how often does a release cause harm, and how fast is the damage stopped.

Deployment rework rate gets less attention and it is worth adding, since it counts unplanned deployments caused by a production incident. In a team where volume has risen, that number moving is often the earliest sign that the gates are letting something through. The full explanation is in measure delivery, not generated output.

Staging tells you less than you think

Progressive delivery is partly a response to a fact most teams already know and few act on: the pre-production environment does not represent production. It has less data, cleaner data, no real traffic patterns, different credentials, and usually a different scale of everything.

For generated code that gap is wider than it used to be, because the failures gather exactly where the environments differ. Code that is competent on the described path and weak on the undescribed one will pass a staging test built from the same description, then run into the account with four hundred thousand records, the user in a timezone nobody considered, or the burst of repeated retries that only happens when a real dependency is slow.

The conclusion is not to delete staging. It is to stop treating a passing staging run as evidence and start treating the first percent of production traffic as the real test. That change of view changes what you build: less investment in making staging lifelike, which is expensive and never finishes, and more in making exposure gradual and reversible, which is cheap and works.

Two things do still belong in staging. Anything destructive that you want to rehearse before doing it for real, including migrations and rollbacks. And anything requiring a third party you cannot safely exercise in production. For everything else, the production canary is both cheaper and more honest, which is the argument in our post on why your staging environment is lying to you.

Where this connects

Progressive delivery depends on knowing what a change can affect, which is set in what done means when an agent writes the code, and on being able to see what is happening, which is observability. When it goes wrong anyway, the process is incident response for agent-written code. The main guide has the wider model.