Image: Trail of Bits
Why it mattersAn agent that writes a patch can quietly break something else; a second agent that checks only its proof, with 25 percent fewer false alarms, keeps review useful.
A security patch written by an AI agent can fix the bug and still break something else in the code. Trail of Bits said it reviewed 130 patches its own engineers had merged into open-source projects and found 21 of them introduced new defects, so it released two plugins to catch the same class of mistake in future runs.
The two plugins, patching-bugs and regression-hunt, come out of Patch the Planet, the joint initiative Trail of Bits runs with OpenAI to co-author security patches for open-source projects. They are written to run on top of OpenAI's Codex agent. The post was published by Anish Naik and Marcelo Morales.
What the review found
Trail of Bits said its engineers audited every patch they had authored and that upstream maintainers merged. Of 130 pull requests, 21 carried a regression a real user or attacker could reach: 16 functional, one performance, and four in build or test. None broke a security protection. The firm defines a regression as a new defect introduced by a change that breaks previously correct and supported behavior.
Those 21 confirmed regressions were spotted inside a set of 58 candidates in a single run, a precision of 36 percent in the firm's own words. Trail of Bits said the review was noisy and that patching at today's bug-discovery pace needs a better signal-to-noise ratio, which is the problem the plugins are built to address.
How regression-hunt scored
regression-hunt asks an agent to find possible regressions, confirm each one with a proof-of-concept run before and after the change, and then pass the results to an engineer to review. Trail of Bits said it ran the plugin against the full set of 21 confirmed regressions and compared it to a baseline Codex agent prompted to review the pull request for regressions.
The plugin produced 25 percent fewer false-positive findings than that baseline, with a 95 percent confidence interval of 14 percent to 38 percent. Trail of Bits said the stricter evidence requirement left some known regressions classed as borderline, where the plugin saw a possible issue but could not gather enough proof. The firm plans to work on raising that confirmation rate without giving up the precision gain.
The patching-bugs plugin is the second half, built from what Trail of Bits called thousands of Codex sessions by its own engineers. It handles reproducing the bug on a live deployment, writing the fix, and an adversarial check on it. Both plugins read two project files when present, THREAT_MODEL.md and PATCHING_GUIDELINES.md, so a project's own rules shape what each run looks for.
An engineering team shipping agent-written patches has to answer the same question Trail of Bits is answering here: how much of a human reviewer's time goes to picking real regressions out of the agent's own false alarms. Dropping a quarter of the false alarms is the kind of change that decides whether an unattended agent step is worth running at all.
Source
Your AI agent fixed the bug, but what else did it change? by Anish Naik and Marcelo Morales on the Trail of Bits blog.
This item was written by an AI system from the linked source. Reveneau is responsible for what it publishes.
Get AI News in your inbox
New developer tools, model and agent releases, and how teams are actually using them to release software. Short, and only when there is something worth reading.