What does zero-downtime database migration mean?

It means changing a schema or moving data to a new datastore without taking the application offline or blocking reads and writes while the change happens. Users keep working normally through every stage, which is different from a maintenance-window migration where the system is intentionally unavailable for a fixed period.

What are the four stages of a zero-downtime migration?

Dual write, backfill, cutover, and cleanup. You start writing new data to both the old and new location at once, copy the historical data across in the background, switch reads and then writes over to the new location once the two are verified to match, and only then remove the old structure.

Why do you write to both the old and new database at the same time?

Because the backfill that copies historical records takes time, sometimes days on a large table, and during that window new data keeps arriving from live traffic. Writing to both places means nothing written during the migration is ever missing from either side, so the two datasets stay in sync while the slow copy runs behind them.

What is a backfill in a database migration?

A backfill is the background job that copies existing historical data from the old structure into the new one. It runs separately from live traffic, usually against a snapshot or in batches to avoid competing with real user queries for the same database resources, and it can take anywhere from minutes to days depending on table size.

How do you know it is safe to cut over to the new database?

By comparing the old and new datasets directly before flipping any traffic, row by row or through checksums on representative samples, and by running the new path in shadow mode, serving reads from the old system while quietly checking that the new system would have returned the same answer. Cutover only happens once that comparison holds for a sustained period.

Why do teams skip the cleanup stage of a migration?

Because by the time backfill and cutover are done, the immediate pressure is gone and the old table works fine sitting unused, so there is no urgent reason to remove it. Left alone, it becomes a second source of truth nobody trusts, and the next engineer who is not sure which table is current adds a defensive check against both, which is how technical debt compounds quietly.

Can AI write a database migration script safely?

AI can write correct migration code for the mechanical parts, the dual-write logic, the backfill job, the comparison queries, and it can do it quickly. What it cannot do on its own is decide what counts as an acceptable loss if the migration has to be reversed partway through, because that judgment depends on the business behind the data, not on the code.

What should a rollback plan for a database migration include?

A statement of exactly which data could be lost or duplicated if the migration is reversed at each stage, after dual write starts, mid-backfill, and immediately after cutover, plus a tested procedure for reversing it. Testing the procedure against a copy of production data is what separates a real rollback plan from a guess written down to look like one.

How long should the dual-write and backfill phases run before cutover?

Long enough that the comparison between old and new data has been clean through at least one full cycle of your busiest recurring load, whether that is a daily batch job, a weekly billing run, or a monthly close. Cutting over before you have seen your own worst-case traffic pattern behave correctly is the most common cause of a migration that passes every check and still breaks in production.

What is the biggest risk in the cutover stage specifically?

The moment reads or writes move to the new system, any bug that only shows up under real production concurrency becomes visible immediately and to every user at once, unlike the dual-write and backfill stages where a problem is contained to a background job. That is why cutover is usually staged, a small percentage of traffic first, watched closely, before the rest follows.

How to plan a database migration with zero downtime: Reveneau

We have scoped enough of these to know the moment a database migration actually gets risky. The schema change is rarely the hard part, and neither is the code. The hard part is the gap between "the new system has the data" and "the new system has been proven to have the same data as the old one, under real traffic, for long enough to trust it." Teams that skip proving that gap ship a migration that works in the demo and breaks three weeks later under a load pattern nobody tested.

The sequence that avoids downtime is not complicated. It has four stages: dual write, backfill, cutover, cleanup. Each one exists to remove a specific kind of risk, and skipping or rushing any of them is where the incidents come from.

Dual write: nothing new goes missing while you move the old

The moment you start a migration, you have two problems running at the same time. You need to copy years of historical data into the new structure, and you need every new record created during that copy to end up in both places. If you solve only the first problem, any row written after the copy started is missing from the new system the day you switch over.

The fix is to write every new record to both the old and new location from the start, before the historical copy even begins. Stripe's engineering team describes this as the first stage of their Data Movement Platform, which they use to move petabytes of financial data across more than 2,000 database shards without downtime. New writes go to the old location first, synchronously, so the request the user is waiting on does not get slower, then get propagated to the new location right after.

The lesson here is not specific to Stripe's scale. It is that dual write has to start before backfill, not after, or you spend the whole migration chasing data that arrived while your back was turned.

Backfill: move the history without touching live traffic

With new writes covered, the next job is copying everything that already existed before the migration started. This is usually the slowest stage, and it is where a migration can quietly hurt the system it is trying to improve, because a naive backfill job competes with real users for the same database connections and the same disk I/O.

The standard fix is to work from a snapshot rather than the live table, and to run the copy in small batches with pauses, rather than one long-running query that locks rows real users are trying to read. Stripe's platform uses a point-in-time snapshot plus distributed workers specifically so the backfill does not draw on the same production resources the application needs.

For a smaller system this does not require distributed infrastructure. It requires the same discipline at a smaller scale: batch the copy, throttle it, and monitor the production database's load while it runs so you notice a problem before your users do.

Cutover: prove the new system agrees with the old one, then switch

This is the stage most teams under-invest in, and it is the one an AI-assisted migration needs the most human judgment on.

Before any real traffic moves, compare the old and new datasets directly. Row counts matching is not enough. Check field values on a representative sample, or run a full checksum comparison if the table is small enough to make that practical. Then run the new path in shadow mode: keep serving real reads from the old system, but also query the new system in parallel and log whenever the two disagree, without anyone outside the team seeing the difference. Only once that shadow comparison has run clean through your system's busiest recurring load, a daily job, a weekly billing run, whatever creates your worst-case pattern, do you start moving real traffic.

Cutover itself should be staged, not instant. Move a small percentage of read traffic first and watch it. Then writes. Any concurrency bug that only appears under real production load becomes visible the moment cutover starts, and unlike a bug in the backfill job, everyone using the system sees it at once. This is exactly the kind of condition a staging environment cannot reproduce, because staging never has your production concurrency: we wrote about that gap in why your staging environment is lying to you.

Two examples show both directions this stage can go. Stripe's own account of the pattern ends in a system holding 99.999 percent uptime through migrations across its database fleet. GitHub's public account of its October 2024 incident is the other direction: a database migration triggered a DNS resolution failure that cascaded into a 19-hour-12-minute outage, with 100 percent of code search requests failing for about four hours and a quarter of Actions workflow runs delayed more than five minutes. GitHub's own report does not describe it as a comparison step being skipped, but the shape of the incident, a migration whose downstream effects on a dependent system were not fully understood before it ran, is exactly why the shadow-mode comparison step exists: it catches the case where the new path behaves differently under conditions the team did not think to test manually.

Cleanup: the stage that gets abandoned because nothing is visibly broken

Once cutover is done and the new system has been serving all traffic cleanly for a while, the old structure just sits there. It works. Nobody is calling it. There is no incident forcing anyone to finish the job.

That is exactly why it does not get finished. An old table left in place becomes a second source of truth that nobody fully trusts, and the next engineer who is unsure which one is current writes a defensive check against both, then a third engineer copies that pattern into a different part of the codebase. A migration that was clean at cutover slowly grows a shadow version of itself that outlives the reason it existed.

Set a firm date for removing the old structure before you start the migration, not after cutover succeeds. Removing dual writes, dropping the old table, and deleting the comparison job are part of the migration, not a follow-up task for whenever someone has time.

Where AI-assisted migrations still need a human-verified rollback plan

AI can write every mechanical part of this sequence well: the dual-write logic, the backfill batching, the comparison queries, the cutover script. That is genuinely useful, because these are exactly the kind of well-specified, checkable tasks an eval suite can verify before anything reaches production: does the backfill respect the batch size, does the comparison catch an intentionally introduced mismatch, does the cutover script fail closed if the new system returns an error.

What an eval suite cannot do is decide, on your behalf, what an acceptable loss looks like if the migration has to be reversed halfway through. That is a business judgment, not a code correctness question. If cutover has to be rolled back three hours in, are the records written only to the new system during that window recoverable, or gone. Does reversing dual write risk a duplicate charge, a lost order, a support ticket. Those answers depend on what the data means to your business, and nobody, human or AI, can infer that from the schema alone.

This is where the eval suite Reveneau runs against a migration plan is a useful check, and a rollback plan is where the team scoping the migration has to sign off directly rather than trusting that a passing test means the plan is safe. A rollback plan earns that name only once it has been executed against a copy of production data. Written down and never run, it is a paragraph that sounds reassuring right up until the day someone needs it to work.

The teams that get this right treat cutover as the one moment in the whole sequence where a human has to be the one who says go, because it is the one decision an automated check cannot fully make for you.

A migration is not finished when the new system has the data. It is finished when someone has proven, and can reverse, exactly what would happen if it had to stop halfway.

Sources

Stripe, How Stripe's document databases supported 99.999% uptime with zero-downtime data migrations: describes the dual-write, snapshot-based backfill, and shadow-comparison pattern Stripe's Data Movement Platform uses across more than 2,000 database shards, and states the 99.999 percent uptime figure.
GitHub, GitHub Availability Report: October 2024: describes the October 11, 2024 incident triggered by a database migration, its 19-hour-12-minute duration, and the specific impact percentages across Copilot, Actions, and code search.