A multi-agent pentest tool matched the runner-up on one fifth the tokens

An AI engineer picking the mix of models for a multi-agent system has had little to compare against. On 3 November 2026, a new open-source project called sapper-v2 published a number that gives one. It took the same 85 flags as the runner-up at a Tencent security contest. It did the run 25 seconds faster. It used 1 billion tokens against the runner-up's 4.5 billion.

sapper-v2 is on GitHub as N1nEmAn/sapper-v2 under the MIT licence. It has 1,466 stars added over seven days. The repository calls the system "the Autonomous Pentest Legion". A bold red note warns it is for authorised security testing only.

What the contest measured

The event was the 3rd Tencent Cloud Intelligent Attack-Defense Challenge online qualifier. 634 teams entered. Each team's AI had to solve capture-the-flag security challenges for 8 hours with no human in the loop. The sapper-v2 README reports the result. Sapper scored 80.18 out of 100 and took 85 flags. The runner-up scored 79.29 and also took 85 flags, but spent 4.5 billion tokens to do it. Sapper used 1 billion.

The numbers are the team's own, taken from Tencent's scoreboard. No third party has audited the token totals. The 4.5-to-1 gap is sapper's own claim about its run. It is not a benchmark against any other tool.

What the architecture does

The README describes a mix of models by role. The commander and the generals run GLM-5.3. The soldiers and scouts run DeepSeek V4.1 Flash. The idea is to put the stronger model where decisions are made and the cheaper model where work gets done. The author says that is how 1 billion tokens beat 4.5 billion.

Three runtime windows. The commander reads the board, decides, and dispatches. It never attacks a target itself. The agent legion runs in tmux panes, one general or soldier per pane. A watchdog script relaunches a crashed commander with --continue, rotates API keys, and stops the whole run at the buzzer. The watchdog holds no strategy.

The README lists things the system invented on its own during the contest. A new role called a signalman, created by the commander to collect unsubmitted flags. Performance reviews for soldiers that had gone an hour without output. A rotation when a soldier's context hit about 260,000 tokens, which left a written playbook for the next soldier.

What a reader can take from this

Three honest limits. First, this was a contest with its own rules and its own targets. The 4.5-to-1 token ratio holds on those challenges. It has not been shown on a real engagement. Second, the project is pinned to authorised security testing. A team planning to use any part of it needs written permission from the system owner. Third, the GLM-5.3 and DeepSeek V4.1 Flash picks are specific to prices on 3 November 2026. A different month's prices may flip the economics.

Given those, the useful signal is the role split. A team building a long-running agent system can test the same shape. Put a stronger model on the planner and the reviewers. Put a cheaper model on the workers. Measure the token spend and the pass rate. The sapper number says the split pays off in one setting. Your own run will say if it pays off in yours.

Source

• N1nEmAn/sapper-v2, GitHub