AI NewsDev toolsAnnouncement

Snyk says changing the harness lifts bug finds past the model

Snyk published its own test of a code-scanning agent on 14 apps. Two harnesses around the same model found different sets of bugs.

AI News

Editorial2 min read

LinkedInX
Snyk Evo Code Scanning Agent blog hero

Why it mattersThe code around a model decides how many bugs a security agent finds, as much as the model does. Buyers of these tools now have a sharp number to ask vendors about.

Snyk published a benchmark of its own experimental code-scanning agent on November 19. The post is by Brian Clark, Adrian Meier and Pauline Bolignano. They argue one point: the loop around a model decides how many real bugs an AI scanner finds. Snyk calls that loop the harness. The numbers are Snyk's own, against named rivals, and Snyk says so.

The method: 14 applications with 1,426 known bugs. Each scanner ran at least three times per app. Evo was tested on two models. On each model Snyk compared Evo to that model's own security agent. It also compared Evo to OpenHands, a plain open-source agent.

Snyk's own numbers

On Claude Sonnet 5.5, Snyk says Evo found 57 percent of the known bugs. Anthropic's Claude Security on the same model found 32 percent. On GPT-6.1 Sol, Evo found 66 percent. OpenAI's Codex Security in deep mode found 43 percent. OpenHands, used as a baseline, found 13 percent.

Snyk reports both rates and raw counts. On Claude Sonnet 5.5, Evo found 468 bugs the other tools missed. On GPT-6.1 Sol, 420. Snyk writes: "the strongest result, 66%, still left about a third unfound."

The precision numbers go the other way. On Claude Sonnet 5.5, Evo was right 65 percent of the time; Claude Security was right 94 percent. On GPT-6.1 Sol, Evo was right 90 percent; Codex Security was right 97.5 percent. Snyk writes: "we're deliberately prioritizing broader vulnerability discovery while continuing to improve precision."

Why this is Snyk's claim, not an audit

Every figure above is a vendor benchmark. Snyk designed the test, picked the apps, named the rivals and ran all five tools itself. The post does not say who built the 1,426 ground-truth labels. The precision gap is wide. A buyer of Evo will have to weigh more finds against more false alarms. Snyk acknowledges this trade-off in the post.

What is harder to dismiss is the shape of the finding. Same model. Same apps. Same compute budget. Snyk's harness found roughly twice the bugs of a rival harness. That points at the harness, not the model.

Evo is not GA, and three things still open

Evo is in "experimental preview with select Snyk customers". There is no price, no product page, no public sign-up. Access is through a Snyk account team. Snyk also flags three things a next post will cover. How consistent Evo is run to run. How it combines traditional static analysis with AI. How Snyk trained its security model.

A buyer of a code-scanning agent in the next quarter can use this post to ask three questions. How many of your test apps have ground-truth bug labels, and who wrote them? What does the recall number look like on a code base the vendor has not seen? What is the false-positive rate a junior engineer will see on the first day?

Source

Why the Harness Matters: Early Lessons From the Evo Code Scanning Agent Preview, Snyk, by Brian Clark, Adrian Meier and Pauline Bolignano.

SourceSnyk

This item was written by an AI system from the linked source. Reveneau is responsible for what it publishes.

Share
LinkedInX
Start a project