Autoprompt cut Terminal-Bench failures 45 percent but took three times longer

Autoprompt is a skill for coding agents, published on GitHub under the account Spielewoy, that rewrites a task prompt before the agent starts work. The repository was created on 17 August, carries an MIT licence, and has 971 stars and 64 forks as of 2 September.

What makes it worth reporting is that the author published the benchmark run behind the main claim, including the cost.

What was measured

The author reports a run on Terminal-Bench 2.1, a benchmark of command-line tasks. OpenCode on its own solved 60 of the 89 tasks. OpenCode with Autoprompt enabled solved 73 of 89.

That moves the failure count from 29 to 16, which the repository describes as "The measured result was 29 to 16 failures (45% fewer) in this run."

The repository states the cost in the same place. The improvement came at about three times the time and twice the tokens. It also carries the caveat that "for very small tasks, this may differ heavily."

How much the number can be trusted

The author's own framing is careful, and the limits are worth stating plainly. This is one run, on one benchmark, against one agent, and the comparison was made by the person who wrote the skill. Nobody independent has repeated it.

Thirteen tasks out of 89 is also a small enough difference that a rerun could change it, and the repository does not report multiple runs or any measure of variance. What the number establishes is that a measurement was taken and published with its cost attached, which is more than most projects of this kind offer.

The interesting part is the pattern of the trade-off. Two thirds of a benchmark solved for one unit of cost, three quarters solved for roughly two units: most teams running agents have noticed this trade-off without ever measuring it.

Once it is written down, the question becomes which tasks should use the skill. A task where a wrong answer is caught immediately by a failing test is cheap to get wrong, and paying double to raise the success rate is poor value. A task where a wrong answer reaches review, or reaches production, is expensive to get wrong, and double the tokens costs little compared with the mistake.

The practical step is to run something like this measurement on your own work rather than trusting a public benchmark. Terminal-Bench tasks are self-contained command-line problems, and a team's real backlog rarely looks like that. The method here, meaning run the same task set twice and record both the success count and the token spend, still applies to other work even when the specific number does not.

Source

Spielewoy/autoprompt-skill on GitHub