GitHub swapped its secret scanner's LLM for a small classifier

A security feature that used to call a large language model for every candidate secret now runs a 150 million parameter classifier on GitHub's own infrastructure, and the figures the team is reporting are large. GitHub published a technical post on 13 October 2026 describing how its secret scanner was rebuilt around a fine-tuned ModernBERT model, with updates on 14 October, and the numbers come from a production A/B test against real customers.

GitHub reports an 11.5 percent improvement in the repository owner-weighted acceptance rate (its measure of whether an alert gets treated as a real finding rather than dismissed), an 86 percent cut in median scan execution time after a push, and a 98 percent reduction in inference cost compared with the LLM-based pipeline the new model replaces. The offline evaluation that preceded the A/B test showed 13 percent higher precision and 19 percent higher recall against the same baseline.

How the new pipeline is built

Secrets are rare in source code and most of the characters that look like a key are not one. GitHub's pipeline uses a cheap regex stage to produce candidates and then runs the ModernBERT classifier on each candidate alongside its context: the file path, the surrounding code, nearby variable names. The classifier is fine-tuned, exported to the ONNX format, and quantised to 8 bits so it fits inside a tight latency budget.

GitHub frames the design around what the feature is for. In the post the team writes: "The best security tool is the one developers trust enough to use. Higher precision means fewer distracting alerts. Higher recall means more real secrets are caught." Trust is the measurement that matters because push protection has to decide whether to block a push in real time, and a developer who has been blocked twice on nothing starts turning the feature off.

What a team using this sees

Push protection waits are shorter and alert queues are cleaner, which is what the 86 percent and 11.5 percent figures mean for a team that uses the product. GitHub's post names the second number as a precision gain in the eyes of the person who owns the repository, which is a stricter test than counting fixed false positives, because it includes alerts a developer dismissed as noise even when they were technically correct.

The post is also a design argument for AI engineering teams. The feature started on a general-purpose large model, and the team moved to a small, task-specific one once they had enough labelled examples to train it. The cost drop (98 percent) is the measurable part of that trade. The precision and recall numbers say the trade did not cost quality, in this task, on this data. Not every classification problem will give the same answer, but the pattern is specific enough to copy: start with the big model to prove the task is doable at all, label the production traffic the preview generates, then move the stable part of the pipeline to something you can afford to run at scale.

The feature is already in production, GitHub says, and no action is required from a repository owner to receive the gains.

Source

GitHub's model for secret detection: Finding more secrets with less noise, GitHub.