Why does code that passes staging still fail in production?

Because staging differs from production in five structural ways: data volume and structure, concurrency, elapsed time and accumulated state, the identity and permissions requests run under, and third-party behaviour. Each difference hides a class of defect, and a passing staging run only establishes that normal use works under favourable conditions.

What is the biggest difference between staging and production data?

Volume and history. Staging data is usually clean, small, and recently seeded, while production data includes rows created before the current validation rules existed, records in states the code no longer produces, and enough volume that a query which returns instantly on hundreds of records can lock a table on millions.

How do I catch concurrency bugs before production?

Write explicit checks that issue simultaneous conflicting requests rather than hoping to encounter the race by chance. Two concurrent accepts of the same invitation, two withdrawals against the same balance, and two writers to the same record are cheap to assert deliberately and effectively invisible to manual testing.

Why do permission bugs get through staging so reliably?

Because staging testing is usually done by whoever built the feature, using an account with broad permissions, since that is the fastest account to set up. Nobody tests the case of a user who lacks access, so the code that should deny them is never run.

Should we get rid of staging?

No. Staging is good at catching configuration errors, broken migrations, missing environment variables, and integration connections that fail completely. The change is to stop treating a passing staging run as evidence that a change is safe, and to move that judgment to explicit automated checks that reproduce the conditions staging cannot.

What is time-related state and why does it break things?

Long-running systems accumulate state that fresh environments do not have: caches populated weeks ago under different logic, sessions issued before a permission change, background jobs that have been failing silently, and scheduled work that only runs at month end. Staging is typically too new for any of it.

How does this apply to AI-generated code specifically?

Generated code is usually competent in normal cases and weakest at limits and unusual inputs, which are precisely the cases staging does not test. That makes a passing staging run an especially poor safety signal for generated changes, and it raises the value of explicit boundary assertions.

What should replace "it worked on staging" as a release check?

A set of automated checks that deliberately reproduce the hard conditions: large and messy fixtures, simultaneous conflicting requests, expired credentials, non-privileged identities, and third-party failure responses. Those run in the pipeline on every change rather than depending on whether someone happened to click the right sequence.

What is the cheapest first improvement?

Stop running tests as an administrator. Create a fixture user with the minimum realistic permissions and run the existing suite as that user. It takes an afternoon and it typically finds at least one access-control defect that had been passing every staging run for months.

Why your staging environment is lying to you: Reveneau

Q: How do you test third-party integrations realistically?

Assert behaviour against the failure modes rather than the sandbox's success cases: a timeout, a 429 with a retry-after header, a 500 on the second call of a two-call sequence, a response missing an optional field, and a duplicate webhook delivery. Those are the cases the real service produces and the sandbox does not.

"It worked on staging" is the most common sentence spoken during an incident, and it is almost always true. The code did work on staging. That is the problem worth understanding, because it means staging passed something it should have caught, and it will do so again next month unless someone names why.

Staging is a different system that resembles production in the properties that are easy to copy and differs in exactly the properties that cause failure. There are five of those differences, they are structural rather than accidental, and none of them is fixed by making staging bigger.

Difference one: the data is clean, small, and new

Production data has history. It contains rows written before the current validation rule existed, records in states the code no longer produces, a customer who somehow has two active subscriptions because of a race in 2024, and text fields containing characters nobody anticipated.

Staging data was seeded last Tuesday by a script that only produces valid records.

Two kinds of failure come from that gap. The first is volume: a query that returns in four milliseconds against 400 rows can lock a table against four million, and no amount of manual staging use will reveal it. The second is structure: code that assumes every user has a billing address is correct against seeded data and wrong against the 3,000 accounts created before that field became mandatory.

What to do instead of hoping: build fixtures that are deliberately difficult. A record missing an optional field that later became required. A string at the maximum length with a multi-byte character at the boundary. A collection with zero items, one item, and more items than the page size. Those fixtures cost an afternoon and they run on every change, which is the part manual staging can never do.

Difference two: staging has one user, production has all of them

Almost every race condition is invisible at one request per second and routine at a thousand.

The pattern is always the same. Read a value, decide something based on it, write. Between the read and the write, another request did the same thing. Two accepted invitations create two memberships. Two withdrawals against a balance of 100 both see 100. Two workers pick up the same job.

You will not find these by clicking through staging, because clicking is inherently sequential. You find them by asserting them deliberately: issue two simultaneous conflicting requests and assert that exactly one succeeds. That check is a handful of lines, it runs in the pipeline, and it either passes forever or it catches the defect on the day someone removes the lock.

The hard part is not writing the check. It is remembering that this class exists, which is why it belongs on a list of guarantees rather than in someone's memory. We covered how to write those sentences in how to write an acceptance test a machine can run.

Difference three: production has been running for a long time

Staging gets reset. Production accumulates.

By month six, production contains a cache populated three weeks ago under logic that has since changed, sessions issued before a permission was revoked, a background job that has failed silently every night since a dependency upgrade, a scheduled task that only runs on the last day of the month, and a feature flag someone enabled for one customer and forgot.

Staging is too new to have any of this, and resetting it regularly is usually considered good practice, which makes it newer still.

The checks that address this are the awkward ones: a token that expires mid-request, a cached value that is stale relative to the source, a job that has already run once for this record, a month-boundary calculation. They are unexciting and they are where the surprising incidents come from.

Difference four: everyone on staging is an administrator

This is the one that produces the most serious defects and it is entirely a consequence of convenience.

Whoever tests a feature on staging is usually whoever built it, using an account with broad permissions, because that account is the fastest to set up and the least likely to get in the way. So the code path for a user who should be denied is never exercised. Not once, on any change, in any release.

Then production has real users with real permission boundaries, and the first person to find the gap is either a customer or someone looking for it.

This matters more with generated code than it used to. Veracode's Spring 2026 research across more than 150 models found generated code compiling and running correctly over 95 percent of the time while being secure in only about 55 percent of generations. Access control is where that gap concentrates, and it is exactly the area an admin-account staging run does not test.

The cheapest single improvement available to most teams: create a fixture user with the minimum realistic permissions and re-run the existing suite as that user. One afternoon. It usually finds something.

Difference five: the sandbox is not the service

Staging talks to a payment sandbox that returns 200 in 40 milliseconds, every time, forever.

The real service rate-limits during a traffic spike, times out on the third call of a batch, returns a response format that is not in the documentation, delivers the same webhook twice, and occasionally succeeds while reporting failure, which is the worst one because a naive retry then charges the customer again.

Sandboxes copy normal use because that is what a sandbox is for. Everything expensive happens in the other cases.

Assert against failure explicitly: a timeout, a 429 with a retry-after header, a 500 on the second call of a two-call sequence, a response missing an optional field, and a duplicate delivery of an event you already processed. Five checks, all cheap, and each one corresponds to an incident that happens to somebody every week.

Why this is worse now than it was

None of the five differences is new. What changed is the volume of code passing through them.

Google's 2025 DORA report, across nearly 5,000 technology professionals, found AI adoption associated with higher delivery throughput and lower delivery stability at the same time. More changes reaching production and a higher share of them going wrong. If your safety mechanism for each change is a person exercising it on staging, then multiplying the number of changes multiplies neither the person's time nor their attention, and the share that gets a real look falls.

There is also a specific interaction with generated code. It tends to be strong in normal cases and weak at limits and unusual inputs, and those are precisely the cases staging does not test. So the thing staging is worst at checking is the thing generated code is most likely to get wrong. Our fuller take on that reading problem is in how to review AI-generated code.

What staging is still good for

This is not an argument for deleting it. Staging catches a real and useful class of problem, and it catches it cheaply:

Configuration that is wrong or missing in a deployed environment.
Migrations that fail against a database rather than against a test fixture.
Integrations that are simply not connected.
Anything that fails immediately and loudly on startup.
A human looking at the interface and noticing it is ugly or confusing, which no automated check will tell you.

That is a genuine list and it justifies keeping the environment. What it does not justify is treating a passing staging run as evidence that a change is safe, because staging answers "does normal use work under favourable conditions" and has been quietly used to answer a question it cannot answer.

The practical sequence

Take the three flows where a silent failure would cost the most. For each one, write down what must always be true, then write the check for each of the five conditions above that applies to it: difficult data, simultaneous requests, stale or expired state, a non-privileged caller, and a failing third party.

That is a day or two per flow, not a quarter. The suite grows afterwards inside normal work on one rule: every defect that reaches production becomes a permanent check, so the same failure cannot happen twice. The longer version of this method is in adding evals to an existing codebase, and the failure modes that make a suite stop meaning anything are in when evals give false confidence.

Thanks to the platform engineers who explained their incident timelines to us and were willing to say out loud which environment had approved each change. Every list looked the same, which is how we ended up with five differences instead of a longer, vaguer set.

Staging gives accurate results for a system your users never use. Production is where your users are.

Related guide: Evals in CI for coding agents.

Sources

Veracode, Spring 2026 GenAI Code Security update: across more than 150 models, generated code compiled and ran correctly in over 95 percent of cases while being secure in roughly 55 percent.
Google Cloud, Announcing the 2025 DORA report: nearly 5,000 respondents, positive relationship between AI adoption and delivery throughput alongside a negative relationship with delivery stability.

Why your staging environment is lying to you