Reviewing one AI-written security report takes a maintainer eight hours

An open-source maintainer reads a security report that names a real-looking bug, opens the code it names, and finds nothing wrong there. The report was written by a chatbot and the hours spent reading it belong to a volunteer.

The New Stack reports that Akrites, a Linux Foundation initiative announced in June that provides a shared security incident response team for open-source projects, has reached a backlog of hundreds of thousands of AI-generated vulnerability reports. The outlet spoke with Christopher Robinson, Akrites CTO and chief security architect at the Open Source Security Foundation (OpenSSF), and Laura Guazzelli, open-source engineer at Microsoft, at the Open Source Summit Europe in Prague last week.

The number that sets the scope

Robinson tells The New Stack that reviewing a single AI-written security pull request can take an engineer up to eight hours. He describes the reports to the outlet as verbose, often wrong, overstating severity, and missing the data an engineer needs to write a patch. "Everybody with a $20 account is now a researcher," Robinson tells the outlet, describing what happened once a chatbot subscription was enough to produce a security report.

Different users running similar chatbots reach similar conclusions on the same code. "These models all kind of work similarly and they find similar bugs, so we're getting a ton of duplicative reports," Robinson tells The New Stack. The duplicates land in separate reports and each is read by a human before it can be dismissed.

Why a model can call a safe pattern unsafe

Guazzelli tells the outlet that a model can read a function on its own and miss the security checks the rest of the project applies to it. "The robots don't have the context," she says. "That's why they can't necessarily call something a false positive, because they don't understand that you have controls that are additionally applied somewhere else." A function that looks dangerous when read on its own may be harmless once the project's other security checks are counted in.

What Akrites is building

Akrites uses AI to deduplicate incoming reports and seconded engineers from member companies review the findings before they reach upstream maintainers. The initiative's founding members, named in the article, include Anthropic, OpenAI, Google, Microsoft, AWS, NVIDIA, Cisco, IBM, Red Hat, Citi and JPMorgan Chase. Each member sends engineering time alongside its funding, and many of the seconded engineers are already contributors to the upstream projects their time is directed at.

Robinson tells the outlet the quality of AI-generated reports has improved this year, but the better reports come from paid frontier models and the enterprise users of those models are not necessarily forwarding findings upstream. The same technology that produces the noise is also finding real bug chains that a human researcher would have taken months to notice.

For a company shipping code that depends on open-source libraries, Akrites changes the backlog but does not clear it. The security of a dependency now depends on whether a seconded engineer at a member company has reached the report that applies to it, so a mirror of the versions you ship is cheaper than finding out that an upstream fix never arrived.

Source

The New Stack, "Everybody with a $20 account is now a researcher"