NVlabs says SoL-Pi cuts a coding agent's tokens 45 to 49 percent

NVIDIA Labs released SoL-Pi on 2 September as an open-source extension for the Pi coding-agent harness, and posted its technical write-up at nvlabs.github.io/SoL-Pi. The repository had 1,234 stars ten days later. Every efficiency number below is what the authors report about their own tool.

Pi is a small coding-agent harness from Earendil. SoL-Pi installs on top of an unmodified Pi release, uses Pi's public extension APIs, and includes four mechanisms turned off by default.

What the four mechanisms change

Action Fusion watches for the pattern where an edit is followed by a validation command in the next turn. It runs the two together and returns one observation, saving one call to the model. NVlabs writes that in base Pi rollouts, adjacent edit-then-command pairs made up 12.3 percent of cross-turn transitions.

ObservationPack archives large tool results locally, then leaves a handle and a short excerpt in the context window. Later requests recall exact pages only when needed, so a big file is not sent with every later turn.

Evidence-Preserving Reducer hands long build and test logs to a cheaper agent first, which produces a receipt. Every quoted line in the receipt is verified against the archived original before the newest, most capable model sees it. Delegation does not require trusting a summary.

Online Context Compact waits for a subtask to finish, then reconsiders compaction. It runs only when the expected future savings are large enough to cover the cost of the rewrite.

The numbers the authors report

On EdgeBench, a 51-task suite of long-running work that NVlabs used as a held-out test set, SoL-Pi retains about 94 percent of Pi's average score. Against Pi, it uses 45 to 49 percent fewer tokens and about a third less cost. Against the model-native harnesses for Codex and Claude Code, it uses 35 to 64 percent fewer tokens at 50 to 54 percent lower API-equivalent cost.

The Terminal-Bench 4 result is more mixed and worth reading in full. On 63 tasks, Codex solved 18 for $272.35, Pi solved 18 for $286.45, and SoL-Pi solved 15 for $211.12. Per solved task that comes out to $15.13 for Codex, $15.91 for Pi and $14.07 for SoL-Pi. The saving is real, and three tasks that Pi and Codex finished, SoL-Pi did not.

For a researcher running one long problem, NVlabs estimates savings of $8.75 to $13.50 per hour compared with the native Codex and Claude Code harnesses, and $4.36 to $5.71 per hour compared with Pi. The choice of backend model sets the range.

The caveats the authors keep in the write-up

All comparisons ran at the highest reasoning-effort setting NVlabs tested. Costs are API-equivalent estimates that price recorded token use at each model's Standard rates, not subscription invoices. Preparation time and requests without returned usage are outside the plotted totals.

The team also ran a swarm test on Anthropic's kernel-optimization take-home. One Sol coordinator with 20 Luna workers running SoL-Pi reached 1,127 cycles at $60.11 in API-equivalent cost, against 1,333 cycles at $39.20 for a single Sol and 1,366 cycles for 20 stock-Pi workers at $82.12. The authors state on the page that this is one nonrandomized trial per condition and that the SoL-Pi run paused three times for network repairs, so a causal estimate is not available.

A team weighing the extension can start with the two local mechanisms. Action Fusion and ObservationPack add no model calls and cannot stop a running task, and NVlabs turns them on in the default configuration. The reducer and the compactor add model calls of their own, so leaving them off until the local ones are proven is the safer order.

Source

• SoL-Pi: Scaling Auto-Research Loops for Efficient Agent Harnesses, NVIDIA Labs blog and technical report
• NVlabs/SoL-Pi on GitHub