FrontierHarness Eval finds task cost varies 17 times across nine agent harnesses

Runta, an agent execution platform, published FrontierHarness Eval this morning, a public leaderboard that puts nine coding-agent harnesses through the same 30 software engineering tasks with the same underlying model, Kimi K3. The main finding on the site, and the number that reached the front page of Hacker News, is that median cost per task varies by a factor of 17 across the harnesses, before pass-rate differences are considered.

The cost spread

At the two ends of the table, Exo Harness runs the median task for $1.05 and Claude Code runs it for $18.34. The gap narrows if only successful tasks are counted, since a cheap harness that fails more often wastes fewer dollars per failure: OpenCode leads that view at $0.0615 per pass and Claude Code is last at $0.2880 per pass, a spread of about 4.7 times.

Pass rate itself falls in a much narrower range. Codex is highest at 66.7 percent, Claude Code and DSH Creator tie at 63.3 percent, and OpenCode and Hermes sit at the bottom on 50 percent. Median runtime per successful task ranges from 5 minutes 41 seconds (DSH Minimal) to 9 minutes 38 seconds (Claude Code).

The caveats to state first

The eval runs 30 tasks per harness. Commenters on Hacker News note that with a sample that small, the 95 percent confidence intervals on pass rate are wide enough that most of the differences between adjacent harnesses may not be statistically distinguishable. Runta also chose one model, Kimi K3, and a harness that is tuned for a different model may not be shown at its best. Pi is described on the site as "intentionally bare bones", so a comparison with full-featured Codex shows the range of approaches and cannot be read as a like-for-like benchmark.

Runta has a business interest in this market, and the site notes that a version 1.1 will publish a full harness by model matrix. Only the tasks and results are shared, not the full run scripts, which one commenter argued would make the numbers more credible if they could be reproduced by outsiders.

Cost varies 17 times even with one model

For a team already paying for a coding-agent workflow, the useful finding is the size of the spread. Cost varies by a factor of 17 at the same model, which means the harness a team chose months ago because someone on the team already used it is likely to be either the cheapest or the most expensive line item in the agent budget, and there is no way to tell which without running the same task through a second harness. A team that runs a few of its own real tickets through two harnesses this week will get a number tuned to its own code, its own tests, and its own review habits, which is a stronger signal than any public leaderboard.

Source

Primary source: FrontierHarness Eval, Runta. Coverage: Hacker News discussion.