AI NewsOpen sourceAnnouncement

GitHub and Microsoft sent 99 AI bug reports, and maintainers fixed 55

GitHub and Microsoft started a pilot that scans open-source projects with an AI model, then has humans filter the output. 9,791 raw findings became 99 verified reports, and maintainers fixed 55.

AI News

Editorial2 min read

LinkedInX
Reveneau illustration of an AI vulnerability report being triaged by a human reviewer

Why it mattersOpen-source maintainers now get many low-quality AI bug reports. The number to copy from this pilot is one report sent per 100 raw findings.

Open-source maintainers are getting many low-quality AI-generated bug reports. GitHub and Microsoft ran a pilot to show what a usable version looks like. The two companies posted the results on the GitHub blog on 17 November. The pilot is called Codename MDASH. It scans a project with Microsoft's MAI Cyber model. A GitHub Security Lab triage agent filters the raw output. Only verified findings go to the maintainer as private vulnerability reports. Microsoft's FORGE Lab checks each finding before it leaves.

The headline numbers, from the blog post, are the shape of the filter. MAI Cyber flagged 9,791 possible issues across six projects. After deduplication and review, 99 private reports were sent to maintainers. Maintainers accepted enough of them to fix 55, with 14 fixes released by 15 November. That is close to one actioned report for every 100 raw findings a model produced.

The six projects

The pilot ran against AutoGPT, Caddy, Express, FastMCP, ImageMagick and Netty. All six are what GitHub calls alumni of its Secure Open Source Fund. These are projects most teams depend on without noticing. ImageMagick sits inside many content pipelines. Express is one of the most used Node.js web frameworks. Netty handles connections in a large share of Java network services. A critical bug in any of them reaches hundreds of thousands of codebases through dependency chains.

The authors are Ashley Wolf, GitHub's director of open source programs, and Gregg Cochran, a Microsoft staff program manager. They describe the goal in one line. Maintainers do not need more AI bug reports; they need fixes. The policy change behind that line is clear. An AI-sourced finding is held to the same bar as a human one before a maintainer sees it.

The ratio is the part worth copying

The number to keep is 99 reports from 9,791 flags. The rest of the industry has not been doing this. The GitHub post describes the baseline plainly: popular projects already receive many AI-written "security" reports that waste days of maintainer time per project. Publishing a scan is cheap. Reading one costs time that volunteer maintainers do not have. A pipeline that drops 99 of every 100 findings before a maintainer sees them is the first fix.

GitHub says it will carry the lessons into private vulnerability reporting. The two companies will keep working together on MAI Cyber 2. All Secure Open Source Fund alumni will be offered AI security scans with pre-verified findings. Any team sending AI scans to open-source projects today has a clear test. Does the sender filter the findings before the maintainer does? For a team building a scanning product, the question is simpler. Is the human triage tier in the price, or is the product just raw output?

Source

Maintainers don't need more AI bug reports. They need fixes., The GitHub Blog.

This item was written by an AI system from the linked source. Reveneau is responsible for what it publishes.

Share
LinkedInX
Start a project