HarnessTax benchmark from UC Berkeley finds Claude Code costs twice what Pi costs at the same success rate

Melissa Z. Pan and five co-authors from UC Berkeley's Sky Lab and Arena published HarnessTax on 16 September, a study that runs the same seven models through three coding-agent wrappers, Claude Code, Codex CLI, and Pi, on SWE-bench Lite and Terminal-Bench 2.0. The finding they lead with: the same model often solves the same share of tasks in each wrapper, and pays up to twice as much in one of them.

The cost gap, in the paper's own numbers

The paper reports that Claude Code costs 2.0 times as much as Pi and 1.6 times as much as Codex on SWE-bench Lite, and 1.5 times as much as Pi on Terminal-Bench 2.0, using the geometric mean of cost ratios across shared models. The success-rate gap over the same runs stays inside two percentage points on SWE-bench Lite and five on Terminal-Bench 2.0.

The paper gives one worked case. Anthropic's Fable 5 solves 97.8% of attempts in Claude Code, 96.7% in Codex, and 96.7% in Pi, and the average cost per attempt is $1.33 in Claude Code against $0.67 in Pi. The two wrappers take similar numbers of turns, 15.3 and 15.4 on average, so the higher Claude Code bill goes to more spending inside each turn. The paper reports that Claude Code's mean initial context on the first model call is over ten times Pi's, with a longer instruction block and a larger tool schema.

An alternative wrapper wins in nine of twelve comparisons

The paper reports that across the six OpenAI and Anthropic models it evaluates, and both benchmarks, an alternative wrapper reaches the highest observed success rate in nine of twelve comparisons. It gives two examples. Sonnet 4.6 solves 68.9% of attempts in Codex against 66.7% in Claude Code on SWE-bench Lite, at a similar cost. GPT-5.6 Sol reaches 83.3% in Pi against 78.9% in Codex on Terminal-Bench 2.0, at $0.42 against $0.76 per attempt.

The scope, in the authors' own words

The paper runs 30 randomly sampled tasks per benchmark, three attempts per pair, with 95% confidence intervals from 10,000 bootstrap resamples. It caps each attempt at 100 turns and uses each wrapper's high-effort setting. Its authors state that the two benchmarks are open source and the models may have seen them during training, and that results may look different on other workloads. Pi, the wrapper that comes out cheapest, exposes only four tools (read, write, edit, bash) and ships with no built-in sandboxing, so a team choosing Pi is choosing that trade too.

For a team already paying for a coding-agent workflow, the useful test is the smallest one: run a handful of real tickets through a second wrapper with the same model and price-tag both. The paper's own point is that the wrapper is a bigger cost lever than most teams' routines treat it as, and the honest check runs on the team's own code and its own tests, on the workload the bill is coming from.

Source

Primary source: HarnessTax: How Much Does the Harness Matter for Coding Agents?, by Melissa Z. Pan, Shuo Yang, Negar Arabzadeh, Wei-Lin Chiang, Ion Stoica and Matei Zaharia. Discussion: Hacker News.