Trail of Bits reanalyzes 1Password's AI patching study and finds agents block the exploit on 86 percent of fair trials

Trail of Bits published a reanalysis on 15 September 2026 of 1Password's August report on how well AI agents patch security vulnerabilities. The 1Password headline said models produced clean fixes only 26 percent of the time. Trail of Bits, writing with authors Anish Naik, Dan Guido, Benjamin Samuels and Marcelo Morales, argues that figure combined trials in which agents were told to apply the wrong fix with trials in which agents were blocked from compiling or testing their code, and that filtering to the fair trials tells a different story.

Where the 26 percent came from

Trail of Bits lists four design choices behind the headline. The sample was chosen for difficulty, with per-bug clean-fix rates ranging from 3 percent to 60 percent, so the average depends on which vulnerabilities went in. Two of the prompts explicitly told agents to apply the wrong fix, and those prompts account for 22 percent of the data. One evaluation mode prevented agents from building or running their code and accounts for 36 percent of the data. The two models tested, GPT-5.5 and Opus 4.8, ran at their default reasoning settings rather than at their highest available, and the report did not measure how effort affected the outcome.

Trail of Bits also flags grading problems. The stopping rule and the grading criteria disagreed: agents were told to stop when their patch defeated the supplied exploit, but the grader then evaluated code paths the exploit never exercised. Two automated reviewers assigned different outcomes to 36.8 percent of the same patches, and the headline averaged them. In one Linux bug the reference upstream fix itself contained an off-by-one error that 248 generated patches reproduced, and the automated grader flagged it in only 24 of them.

What the fair trials show

Trail of Bits reran the numbers on the trials where the agent could build and run code and was not instructed to apply the wrong fix, excluding runs the study classified as having consulted the upstream fix. Of 3,067 patches generated by 1Password's models under those conditions, 2,634 (86 percent) blocked the supplied exploit. Blocking the supplied exploit does not prove a complete repair, and Trail of Bits states so plainly, but the reading is a long way from 26 percent.

The human baseline it puts next to the number

Trail of Bits also published a baseline for how often human developers get a security fix right on the first try. Reviewing 2,265 vulnerabilities across 236 security assessments from 2024 to 2026, its engineers found that 283 first-fix submissions failed to fully resolve the reported issue: 12.5 percent, or one in eight, with a 95 percent confidence interval of 10.5 percent to 14.5 percent. The developers maintained the affected software, had detailed vulnerability reports, and knew Trail of Bits would review their patches.

Trail of Bits is releasing two agent skills alongside the post: post-patch-validation, meant to help an agent test its own fix, and review-walkthrough, meant to help an engineer review one. Both are pitched at teams running agent-assisted patching workflows rather than one-shot benchmarks.

What the reanalysis is worth to a team

The two vendors involved both have stakes in the number. 1Password published a report positioning AI patching as unreliable. Trail of Bits ships Patch the Planet, a joint agent-assisted patching initiative with OpenAI. A team considering an agent for security fixes should read the two figures alongside each other and look at the underlying method, since each set of authors kept the trials that suited their claim. Trail of Bits shows the trials it kept and the trials it dropped, which is what a benchmark reader can use.

Source

1Password's AI patching benchmark is misleading, Anish Naik, Dan Guido, Benjamin Samuels and Marcelo Morales, Trail of Bits, 15 September 2026.