
Image: Runta
Why it mattersA team that switches models and keeps the old harness can pay more and finish fewer tasks. The pairing matters.
Picking a coding harness and sticking with it costs money when the model under it changes. A new benchmark from Runta tested 22 harnesses with five open models. The best harness was a different one every time.
Shilin Zhu at Runta published FrontierHarness Eval v1.1 on November 17. The run used 83 billion tokens across 27,280 agent trials. That is 76 times the size of v1.0. The setup is 22 harnesses, 5 models, 62 tasks, 4 attempts per task.
What the benchmark found
The best-value harness for each model is different. Runta reports these pairings as the top pick by cost per passed task.
- GLM-5.3: Codex CLI at $0.803 per pass
- GLM-5.3 Flash: Pi
- DeepSeek V4.1 Flash 1017: ZCode
- Muse: Oh My Pi
- Kimi-K3: Alma
The ranking shifts are large. Codex CLI is first with GLM-5.3 and sixteenth with Kimi-K3. DeepSeek Harness: Creator is first with Muse and twenty-first with GLM Flash. OpenCode v1 is first with DeepSeek and thirteenth with Muse. The median harness moves 11 of 22 places from one model to the next.
With a fixed model, cost ranges 1.8 to 5.0 times across harnesses. DeepSeek showed the widest range. All five differences exceeded what chance would produce, at p=0.003.
Cost does not buy pass rate either. The rank correlation between cost per task and pass rate was between -0.39 and +0.22 across models. The harness with the best pass rate ranked fifth to twentieth in cost per task.
What drives the cost
Zhu tested four possible drivers. Steps per pass correlated +0.51 with cost, with p=0.015. Prompt size, tool count and cache hit rate did not correlate. The explanation is simple. Every step sends the whole context again.
The five best-value winners share three habits. One shell tool handles 61 to 93 percent of tool calls. The context is appended, not rewritten, so the cache hits 91 to 94 percent on passed runs. No subagents. Little summarisation.
Most of the money went on runs that did not succeed. Runta reports 64.5 percent of inference costs went to trials that did not pass: 52.7 percent on failures and 11.8 percent on timeouts. Among failures, 68 to 82 percent ended with a confident wrong answer. The harness did not say it had run out of time.
What Runta does not say
The five models are open-weight. Claude, GPT and Gemini are not in this run. Zhu says that explicitly. All runs went through Baseten, so cache pricing is one provider's. Runta provided the runtime, with a container deleted after each trial for isolation. A different inference provider could rank the harnesses differently.
The bootstrap stability is uneven. The Codex CLI pick for GLM-5.3 held in 99.6 percent of 4,000 resamples. The Kimi-K3 pick of Alma held in 25 percent. Pi with GLM Flash held in 42 percent. The headline pairings are not all equally confident, and Runta publishes the percentages next to them.
The practical note Zhu ends on is the one for a team: when you change the model, treat the harness as a variable, not a setting. Test the candidate harnesses on a private task set that is large enough to see past the noise.
Source
Shilin Zhu, "Introducing FrontierHarness Eval v1.1," Runta. Baseten served the models; Runta provided the runtime.
This item was written by an AI system from the linked source. Reveneau is responsible for what it publishes.
Get AI News in your inbox
New developer tools, model and agent releases, and how teams are actually using them to release software. Short, and only when there is something worth reading.


