What does zero-downtime database migration mean?

It means changing a schema or moving data to a new datastore without taking the application offline or blocking reads and writes while the change happens. Users keep working normally through every stage, which is different from a maintenance-window migration where the system is intentionally unavailable for a fixed period.

What are the four stages of a zero-downtime migration?

Dual write, backfill, cutover, and cleanup. You start writing new data to both the old and new location at once, copy the historical data across in the background, switch reads and then writes over to the new location once the two are verified to match, and only then remove the old structure.

Why do you write to both the old and new database at the same time?

Because the backfill that copies historical records takes time, sometimes days on a large table, and during that window new data keeps arriving from live traffic. Writing to both places means nothing written during the migration is ever missing from either side, so the two datasets stay in sync while the slow copy runs.

What is a backfill in a database migration?

A backfill is the background job that copies existing historical data from the old structure into the new one. It runs separately from live traffic, usually against a snapshot or in batches to avoid competing with real user queries for the same database resources, and it can take anywhere from minutes to days depending on table size.

How do you know it is safe to cut over to the new database?

By comparing the old and new datasets directly before moving any traffic, row by row or through checksums on representative samples, and by running the new path in shadow mode, serving reads from the old system while quietly checking that the new system would have returned the same answer. Cutover only happens once that comparison has stayed correct for a long period.

Why do teams skip the cleanup stage of a migration?

Because by the time backfill and cutover are done, the immediate pressure is gone and the old table works fine sitting unused, so there is no urgent reason to remove it. Left alone, it becomes a second copy of the data that nobody trusts, and the next engineer who is not sure which table is current adds a defensive check against both, which is how problems in the code quietly add up.

Can AI write a database migration script safely?

AI can write correct migration code for the routine parts, the dual-write logic, the backfill job, the comparison queries, and it can do it quickly. What it cannot do on its own is decide what counts as an acceptable loss if the migration has to be reversed partway through, because that judgment depends on the business behind the data, not on the code.

What should a rollback plan for a database migration include?

A statement of exactly which data could be lost or duplicated if the migration is reversed at each stage, after dual write starts, mid-backfill, and immediately after cutover, plus a tested procedure for reversing it. Testing the procedure against a copy of production data is what separates a real rollback plan from a guess written down to look like one.

How long should the dual-write and backfill phases run before cutover?

Long enough that the comparison between old and new data has shown no differences through at least one full cycle of your busiest recurring load, whether that is a daily batch job, a weekly billing run, or a monthly close. Cutting over before you have seen your own worst-case traffic pattern behave correctly is the most common cause of a migration that passes every check and still breaks in production.

What is the biggest risk in the cutover stage specifically?

The moment reads or writes move to the new system, any bug that only shows up under real production concurrency becomes visible immediately and to every user at once, unlike the dual-write and backfill stages where a problem is limited to a background job. That is why cutover is usually staged, a small percentage of traffic first, watched closely, before the rest follows.

How to plan a database migration with zero downtime: Reveneau

We have scoped enough of these to know the moment a database migration actually gets risky. The schema change is rarely the hard part, and neither is the code. The hard part is the gap between "the new system has the data" and "the new system has been proven to have the same data as the old one, under real traffic, for long enough to trust it." Teams that skip proving that gap release a migration that works in the demo and breaks three weeks later under a load pattern nobody tested.

The sequence that avoids downtime is not complicated. It has four stages: dual write, backfill, cutover, cleanup. Each one exists to remove a specific kind of risk, and skipping or rushing any of them is where the incidents come from.

Dual write: nothing new goes missing while you move the old

The moment you start a migration, you have two problems running at the same time. You need to copy years of historical data into the new structure, and you need every new record created during that copy to end up in both places. If you solve only the first problem, any row written after the copy started is missing from the new system the day you switch over.

The fix is to write every new record to both the old and new location from the start, before the historical copy even begins. Stripe's engineering team describes this as the first stage of their Data Movement Platform, which they use to move petabytes of financial data across more than 2,000 database shards without downtime. New writes go to the old location first, synchronously, so the request the user is waiting on does not get slower, then get propagated to the new location right after.

The lesson applies at any scale: dual write has to start before backfill, or you spend the whole migration looking for data that arrived while you were busy with the copy.

Backfill: move the history without touching live traffic

With new writes covered, the next job is copying everything that already existed before the migration started. This is usually the slowest stage, and it is where a migration can quietly hurt the system it is trying to improve, because a simple backfill job competes with real users for the same database connections and the same disk I/O.

The standard fix is to work from a snapshot rather than the live table, and to run the copy in small batches with pauses, rather than one long-running query that locks rows real users are trying to read. Stripe's platform uses a point-in-time snapshot plus distributed workers specifically so the backfill does not use the same production resources the application needs.

For a smaller system this does not require distributed infrastructure. It requires the same care at a smaller scale: copy in batches, limit how fast the copy runs, and monitor the production database's load while it runs so you notice a problem before your users do.

Cutover: prove the new system agrees with the old one, then switch

This is the stage most teams spend too little effort on, and it is the one an AI-assisted migration needs the most human judgment on.

Before any real traffic moves, compare the old and new datasets directly. Row counts matching is not enough. Check field values on a representative sample, or run a full checksum comparison if the table is small enough to make that practical. Then run the new path in shadow mode: keep serving real reads from the old system, but also query the new system in parallel and log whenever the two disagree, without anyone outside the team seeing the difference. Only once that shadow comparison has shown no differences through your system's busiest recurring load, a daily job, a weekly billing run, whatever creates your worst-case pattern, do you start moving real traffic.

Cutover itself should happen in stages. Move a small percentage of read traffic first and watch it. Then writes. Any concurrency bug that only appears under real production load becomes visible the moment cutover starts, and unlike a bug in the backfill job, everyone using the system sees it at once. This is exactly the kind of condition a staging environment cannot reproduce, because staging never has your production concurrency: we wrote about that gap in why your staging environment is lying to you.

Two examples show a good result and a bad one from this stage. Stripe's own account of the pattern ends in a system holding 99.999 percent uptime through migrations across its database fleet. GitHub's public account of its October 2024 incident is the bad result: a database migration triggered a DNS resolution failure that led to a 19-hour-12-minute outage, with 100 percent of code search requests failing for about four hours and a quarter of Actions workflow runs delayed more than five minutes. GitHub's own report does not describe it as a comparison step being skipped, but the pattern of the incident, a migration whose effects on a system that depended on it were not fully understood before it ran, is exactly why the shadow-mode comparison step exists: it catches the case where the new path behaves differently under conditions the team did not think to test manually.

Cleanup: the stage that gets abandoned because nothing is visibly broken

Once cutover is done and the new system has been serving all traffic cleanly for a while, the old structure just sits there. It works. Nobody is calling it. There is no incident forcing anyone to finish the job.

That is exactly why it does not get finished. An old table left in place becomes a second copy of the data that nobody fully trusts, and the next engineer who is unsure which one is current writes a defensive check against both, then a third engineer copies that pattern into a different part of the codebase. A migration that was clean at cutover slowly gains extra code that keeps both copies in use long after the reason for them has gone.

Set a firm date for removing the old structure before you start the migration, not after cutover succeeds. Removing dual writes, dropping the old table, and deleting the comparison job are part of the migration, not a follow-up task for whenever someone has time.

Where AI-assisted migrations still need a human-verified rollback plan

AI can write every mechanical part of this sequence well: the dual-write logic, the backfill batching, the comparison queries, the cutover script. That is genuinely useful, because these are exactly the kind of well-specified, checkable tasks an eval suite can verify before anything reaches production: does the backfill respect the batch size, does the comparison catch an intentionally introduced mismatch, does the cutover script stop safely if the new system returns an error.

What an eval suite cannot do is decide, on your behalf, what an acceptable loss looks like if the migration has to be reversed halfway through. That is a business judgment, not a code correctness question. If cutover has to be rolled back three hours in, are the records written only to the new system during that window recoverable, or gone. Does reversing dual write risk a duplicate charge, a lost order, a support ticket. Those answers depend on what the data means to your business, and nobody, human or AI, can infer that from the schema alone.

This is where the eval suite Reveneau runs against a migration plan is a useful check, and a rollback plan is where the team scoping the migration has to sign off directly rather than trusting that a passing test means the plan is safe. A rollback plan deserves that name only once it has been executed against a copy of production data. Written down and never run, it is a paragraph that sounds reassuring right up until the day someone needs it to work.

The teams that get this right treat cutover as the one moment in the whole sequence where a human has to be the one who approves the switch, because it is the one decision an automated check cannot fully make for you.

A migration is not finished when the new system has the data. It is finished when someone has proven, and can reverse, exactly what would happen if it had to stop halfway.

Sources

Stripe, How Stripe's document databases supported 99.999% uptime with zero-downtime data migrations: describes the dual-write, snapshot-based backfill, and shadow-comparison pattern Stripe's Data Movement Platform uses across more than 2,000 database shards, and states the 99.999 percent uptime figure.
GitHub, GitHub Availability Report: October 2024: describes the October 11, 2024 incident triggered by a database migration, its 19-hour-12-minute duration, and the specific impact percentages across Copilot, Actions, and code search.

How to plan a database migration with zero downtime