Google compared an AI bug finder with a fuzzer on Chrome's V8 for six months: Reveneau

Why it mattersFuzzers and LLM agents miss different bugs. A security team that runs only one of them is likely leaving real vulnerabilities in the code.

A fuzzer that has run on Chrome's V8 engine for years still misses bugs. An LLM agent now catches many of them. Google Project Zero published that result on 27 October. It has six months of production data behind it. For a security team, the question is which bugs each tool misses.

The post was written by Samuel Groß. It compares two tools Google runs on V8. Fuzzilli is a generative fuzzer built for JavaScript engines. BigSleep is an LLM bug finder that reviews each new commit. It writes a crashing test case for every bug it finds. Both tools file bugs through Clusterfuzz, which makes a head-to-head count possible.

The counts, in Project Zero's words

From April to September 2026, Fuzzilli filed 316 bug reports and BigSleep filed 441. Narrowing to type vulnerability, Fuzzilli led 178 to 95. The authors then kept only reports that were not duplicates and had a fix commit attached. That final cut puts BigSleep at 45 actionable vulnerabilities and Fuzzilli at 37.

Cost per bug is not reported. The team says neither system is tuned for an optimal cost-per-bug ratio.

The overlap, in two measures

The overlap in production was about 25 percent. That is 20 of 81 actionable vulnerabilities filed by both tools. A slight majority were filed first by Fuzzilli and later re-filed by BigSleep.

The team also ran a controlled experiment. They took 112 historic Fuzzilli bugs. For each one, they asked a BigSleep agent to find it from the commit that introduced it. The model was Gemini 3.6 flash. The agent found the right bug 62 percent of the time. It found a different real bug 12 percent of the time. It found nothing on 26 percent of commits. With different prompting, the right-bug share rose to about 70 percent.

The authors also looked at bug difficulty. They measured it as the time Fuzzilli took to find each bug. They state that bugs hard for a fuzzer tend to be hard for the agent too.

What Project Zero recommends

Groß writes that vendors should perform both fuzzing and agentic analysis. He states this gives the most vulnerabilities before users are affected. Even in the experiment's best run, the agent found no bug on 16 percent of commits.

The reader action is specific. A security team running only a fuzzer should plan the LLM agent for next quarter. A team running only an LLM agent should keep the fuzzer. The two find different bugs on well-tested code. The production run shows 61 unique vulnerabilities caught by one tool alone. Only 20 were caught by both.

The author also says BigSleep used an older model earlier in the window and that future models should perform better. The 45-to-37 count is from one engine, V8, over one period. Groß ends by leaving broader coverage of other targets, fuzzers and agents to future work.

Source

Fuzzers vs. AI: Comparing Fuzzilli and BigSleep on JavaScript Engines, Google Project Zero, by Samuel Groß.

This item was written by an AI system from the linked source. Reveneau is responsible for what it publishes.