AWS Continuum scored 89 percent on CyberGym-E2E, 37.8 percent on target match

A code security benchmark can grade an AI agent on the wrong thing. The headline score hides it.

AWS made Continuum for code vulnerabilities generally available on Thursday. The service finds bugs in a team's repositories. It checks each one against the live AWS environment. It opens pull requests with proposed fixes. The service is live in US East, Northern Virginia, and US West, Oregon. Amanda Caswell at The New Stack walked through the benchmark AWS cited.

The 89 percent score

AWS published its headline score on October 5. That was more than two weeks before the launch. In that run, Continuum passed 819 of 920 tasks on CyberGym-E2E. That is an 89.0 percent end-to-end success rate. The previous public high was 65.9 percent. AWS CEO Matt Garman wrote on X that Continuum scored higher than any other AI agent on the test.

How CyberGym-E2E works

CyberGym-E2E gives an agent 90 minutes to find and fix a vulnerability. The agent works on one of 139 open-source projects. It gets no bug description, proof of concept, crash log, or existing patch. All tasks involve memory-safety bugs in C or C++. AWS says that is a limit of the test.

Grading runs in four stages. Continuum found and reproduced a bug in 92.5 percent of tasks. It stopped the crash in 89.6 percent. It kept the project's tests passing in 89.0 percent. The fourth stage checks whether the patch also fixes the specific historical vulnerability the benchmark picked. The agent is never told which one that is. Continuum's patch addressed that bug in 37.8 percent of tasks. The previous public high was 26.2 percent.

The gap in the headline

A repository can hold more than one real bug. The four-stage design accepts that. The 89 percent score means Continuum found and patched a real flaw without breaking the project. The 37.8 percent score means the patch addressed the one the benchmark had in mind.

A buyer still cannot see one thing in these numbers. That is how well Continuum fixes a specific vulnerability a team hands it. That is the day-to-day job. AWS ran the tests itself. AWS says the agent followed the benchmark's rules, including internet-access limits. AWS also says it did not look up public fixes. AWS says a longer time limit raises the end-to-end score to 93.7 percent. The benchmark and testing tools are public. An independent run is possible. As of launch, the published results are AWS's own.

Corey Quinn, chief cloud economist at Duckbill Group, replied to Garman on X. He asked whether the GA was pulled forward from re:Invent given the state of the internet this week. He offered no evidence that AWS had changed its plan.

For a team evaluating an AI security agent, the gap between the two scores is a reminder. Ask what the agent was graded on. A score for finding and patching any real bug is one thing. A score for fixing the bug the team cares about is another.

Source

AWS Continuum for code vulnerabilities is now generally available, Esra Kayabali, Amazon Web Services. AWS says its AI security agent fixes 89% of bugs, but there's a catch, Amanda Caswell, The New Stack.