With one model, coding harnesses used 3,500 to 292,000 tokens per task

The New Stack collected three benchmarking efforts on coding-agent harnesses and reported the results on 27 August. The main number: with the model held fixed and the tasks held fixed, tokens per solved task ranged from roughly 3,500 for Aider in architect mode to 292,000 for OpenClaw.

The article is by Janakiram MSV. Every figure below is from it, and the caveats are the article's own.

What was measured

Three separate efforts, with different strengths.

A June benchmark ran twelve configurations through OpenRouter on the same tasks, so every harness used the same API and the same model. It covered Aider, Claude Code, Codex, Goose, Hermes, Kilo, Kimi Code, Nanobot, OpenClaw, Opencode and Qwen Code, counting Aider's architect mode separately. The suite ran on DeepSeek V4 Flash and then on Nvidia's Nemotron 3 Ultra.

Composio, in August, compared eight harnesses on 30 enterprise workflows spanning Airtable, Gmail, Google Calendar, Google Sheets, GitHub, Slack and PostHog, each with a 900 second time limit. A programmatic verifier graded the outcome rather than a model acting as judge. Of 240 runs, 129 workflows completed successfully. Cost per successful task ran from $0.028 for Pi Agent to $0.195 for Claude Code, and DeepAgents matched Claude Code's pass rate exactly at a quarter of the cost per success.

Artificial Analysis maintains a continuous index across 326 tasks, averaging three attempts each.

Where the tokens go

The June benchmark named the cause and called it the "startup tax". Before a prompt does any work, the harness sends its own system prompt, tool descriptions and environment setup. That measured around 700 tokens for Aider in architect mode and around 26,000 for OpenClaw.

Paid once, a fixed cost of 26,000 tokens would not matter. It is not paid once. Each turn resends the context, so the article calculates that a harness that resends those 26,000 tokens on each of fifteen turns spends roughly 390,000 input tokens on this setup text alone.

This startup cost multiplied by turn count predicted tokens per solved task with an R-squared of 0.99 across both models. Growth rate did not explain the gap: every harness added a few hundred tokens per turn.

The caching result, and why it needs care

Composio found Claude Code drew 1.5 percent of its input tokens from cache, against roughly 70 percent for Codex and 57 percent for OMP. Fresh input costs about five times cached input, so Codex came out cheaper per solved task than a harness using half as many raw tokens.

Read the attribution before repeating that. The June benchmark's author blamed Claude Code's near-zero cache share on the way its requests reached the model, not on its prompts: it was the only harness talking to OpenRouter through the Anthropic-style messages endpoint, and the gateway's translation of that format appeared to reduce cache hits. That is a finding about a gateway, not about the product.

The June run also made a single pass per harness, task and model, so it carries no variance estimates, and agent runs are stochastic. Composio disclosed that Pi used a different reasoning setting and that Prime Agent produced only 24 gradable runs out of 30.

What to do with it

Measure your own two numbers: the fixed system prompt size your harness sends, and the average turn count of a real session. Their product is most of your bill, and both are things you can change without changing model.

Then check your cache hit rate, especially if you route through a gateway.

Source

• Aider, Claude Code, and OpenClaw ran an identical model. Token use varied 70-fold., The New Stack, 27 August 2026