Momentic says 9.6 percent of AI-written unit tests passed its audit

A coding agent that writes its own tests passes them with ease. A new audit says that is often the problem.

Jeff An wrote on the Stack Overflow blog on 13 November 2026 that his company audited 1,000 unit tests. The tests were written by AI coding agents. An is CTO and co-founder of Momentic. Momentic sells test automation for AI-generated code. The post is the company's case for its own practice.

What the audit found

Momentic put each test in one of four buckets, An writes. Only 9.6 percent were deemed useful. Another 31.6 percent were "low-consequence" and tested something trivial. A further 22.8 percent repeated a test already written elsewhere. The last 22.4 percent were "over-mocked" and tested the mock instead of the code.

An names three reasons the tests fail in the same ways. Agents prefer shallow checks that pass the first time. A test written by the same agent that wrote the code inherits the same blind spots. Long context also hurts judgement. Momentic says Claude Sonnet 4 drops from 89 percent to 38 percent accuracy as context grows. The context grew from 300 tokens to 113,000.

The Momentic alternative

An argues the fix is to split the writer from the grader. A separate agent, with no memory of how the code was built, writes verification tests. It runs the real app rather than mocks. It uses a different model provider where possible, for a second perspective.

Momentic says this second agent found 3 to 5 times more bugs than self-verification across the audited sample. The exact method and sample mix are not in the post. An does not name the products the tests were pulled from. He does not name the languages or the coding agents. The 9.6 percent figure is Momentic's own measurement on its own corpus. No outside team has reproduced it.

For a team that lets an agent write tests

An's claim has a straight consequence for a team that runs a coding agent inside its CI pipeline. A green test suite is weaker evidence than it used to be. The suite still runs. The number still goes up. But a share of the new tests pass because they ask the wrong question. The remedy An proposes needs no new product. Write the code with one agent. Write the tests with another. Give the second one no context from the first.

The harder part is reading the test suite itself. A test that imports every mock from the implementation is suspect. A test that checks a function returns the value a constant already carries is suspect. A test file written in the same session as the code it covers is suspect. An eval harness is still worth running on an AI-written feature. But a reader who trusts the test suite alone now trusts the same reasoning that wrote the bug.

Source

Your coding agent shouldn't grade its own homework, by Jeff An, Stack Overflow Blog.