Why flaky tests are worse when agents write them

A flaky test (one that passes or fails at random on the same code) used to cost everyone some attention. It fails, someone reruns it, it passes, work continues. Annoying, well understood, rarely urgent. Most teams keep a handful of them for years.

That arrangement stops working the moment something automated is reading the result.

An agent uses the suite to decide whether it has finished. That is the whole cycle: make a change, run the checks, interpret the outcome, act. An unreliable signal in that cycle produces confident action based on false information.

The two failure directions

The false failure. A test fails for reasons unconnected to the change. The agent reads this as evidence that its work is wrong, and starts fixing code that was already correct. It will keep going until the check happens to pass, and everything it altered along the way stays in the diff.

The result is a change nobody requested, in an area that had no defect, arriving with a fluent explanation of what was supposedly wrong. A reviewer skimming it sees a plausible commit message and a passing build. There is little in that picture to suggest the test was the unreliable part.

The false pass. A test passes for reasons unconnected to correctness. The agent concludes the work is done, and broken code moves forward with every check passing.

This is the worse direction, and the quieter one. A false failure at least produces visible, repeated changes that someone might question. A false pass generates a record showing the work was verified, and nobody looks again.

The third failure, which is about the instruction

There is a specific problem that only appears when you hand a suite to an agent and ask it to make the suite pass.

Deleting the assertion satisfies that instruction perfectly. So does wrapping the check in a retry until it eventually succeeds. So does making a comparison less strict until nothing can fail it. So does marking the test skipped.

Each of those is a completely correct solution to the sentence that was written. Each is a wrong solution to the problem that was meant. The cause is an instruction that named the wrong goal, and the agent reasoned correctly from it.

Two things follow.

Ask for the behaviour instead of a passing result. "This function must reject an expired token" is a goal that cannot be satisfied by deleting a check. "Make the suite pass" can.

And enforce the rule automatically. Treat the check files as protected, so a change to them fails the run or needs separate human approval. A rule that exists only as a sentence in a prompt is a preference, and anything optimising for a passing suite will eventually find the cheapest path to one. Related reading is in never let the model grade its own work and evals in CI for coding agents.

Finding the ones you do not know about

Most teams know about two or three flaky tests. Most teams have more.

The way to find them costs almost nothing. Run the entire suite repeatedly against completely unchanged code, twenty times or more, and record every check whose result varies. Nothing changed, so any variation is coming from the tests.

Do it overnight and read the report in the morning. The number is usually higher than anyone expected, and the useful part is that it converts a vague sense that the build is unreliable into a specific list with names on it.

The common causes are ordinary. Shared state between checks, so they pass alone and fail together. Real clocks and real dates, so something breaks at midnight or in a different time zone. Real network calls to things that are occasionally slow. Assumptions that operations complete in a fixed order when they actually run concurrently.

That last one deserves attention with generated code specifically, because concurrency bugs are exactly the type that looks fine when read at normal speed. Veracode's Spring 2026 testing across more than 150 models found generated code compiling and running correctly over 95 percent of the time while passing security checks in roughly 55 percent, with the weakest types being the ones spread across several individually reasonable lines: cross-site scripting at 15 percent and log injection at 13 percent. An ordering assumption has the same pattern. Every line is defensible and the relationship between them is wrong.

What to do with one once you find it

Move it out of the main suite the same day, away from the checks a merge must pass, so it stops sending unreliable signals, and put a deadline on repairing or deleting it.

The choice people avoid is deletion, and it is often correct. A deleted check leaves an honest gap that someone can see and make a decision about. A flaky check left running produces false information and, worse, teaches everyone that a red result probably means nothing.

That lesson spreads. Once a team cannot tell which failures are real, it starts treating all of them as meaningless, and the suite stops being something anyone makes decisions with. The genuine failure it eventually catches gets rerun twice and ignored along with the rest.

The check that cannot fail

There is a related problem worth testing for at the same time, because it produces the same false confidence from the opposite direction.

A test that always passes is simply wrong, and nobody sees it, because passing checks do not attract questions. Suites generated by reading an implementation are especially prone to this: they assert that the code does what it does, so they pass automatically and would keep passing against an empty function.

The check takes ten minutes. Break the behaviour on purpose and confirm the check fails. Do it in the two or three areas where a failure would actually matter: authentication and permissions, anything moving money, anything that writes or deletes customer data. If nothing fails, you have learned that the verification checks nothing, which is much better learned deliberately than during an incident. The full list of these is in when evals give false confidence.

Why this became urgent rather than annoying

None of the above is new. Flaky tests have been a known problem for as long as there have been test suites, and teams have tolerated them because the cost was limited: a rerun, a shrug, a note on the list of later work.

What changed is who reads the result. A human brings context to a failing build. They know that particular test is unreliable, they know it started after the migration, they know to rerun it. That context is doing real work, and it exists only in their memory.

An automated loop has none of it. It has the exit code. Every piece of judgment that people used to add to the suite's results has to be built into the suite, which means the suite has to be trustworthy in a way it never previously needed to be. The broader version of this argument is in how to write an acceptance test a machine can run.

Thanks to the teams who ran the twenty-times experiment and told us what came back. In each case the surprising part was how long the worst tests had been failing intermittently while everyone quietly worked around them.

A test that is sometimes right is a source of false information, and something is now reading it.

Related guide: What to check in an eval suite.

Sources

• Veracode, Spring 2026 GenAI Code Security update: across more than 150 models, over 95 percent syntactic correctness against roughly 55 percent security pass rate, with cross-site scripting at 15 percent and log injection at 13 percent.