AI NewsProductivityAnnouncement
Cline published a 47-task coding agent benchmark with pass-at-k and pass-to-the-k scores
Cline ran 47 real coding tasks against 11 models and published the pass-at-1, pass-at-3 and pass-to-the-3 scores. The top single-try score was 51.1 percent.

Image: Cline
Why it mattersA single-run pass rate hides how often the same model fails the same task on a retry. Ask the agent vendor for pass-to-the-k before you buy.
A coding agent can pass a test once and fail the same test on the next run. Cline-Bench puts a number on that gap. On a 47-task set, Gemini 4 Argon led the single-try score at 51.1 percent. Opus 5.5 solved 42.6 percent of the tasks on every one of three tries. Gemini 4 Carbon solved 66.0 percent on at least one of three tries. It dropped to 29.8 percent on all three.
Cline published the benchmark on November 19, authored by Runhui Wang. The post says each task carries a user request, a runnable starting repository, and a test suite. A solution has to pass those tests. Cline ran 11 models through it.
What the three scores answer
Pass-at-1 is the single-try score. It answers whether the agent can finish the task once. Pass-at-3 asks whether the agent can finish it on at least one of three tries. Pass-to-the-3 asks whether the agent can finish it on every one of three tries. That last number is the one to watch when the agent writes code with no human review.
Gemini 4 Argon scored 51.1 percent at pass-at-1, 63.8 percent at pass-at-3 and 40.4 percent at pass-to-the-3. Opus 5.5 scored 48.9 percent, 59.6 percent and 42.6 percent. Gemini 4 Carbon scored 48.9 percent, 66.0 percent and 29.8 percent. The gap between pass-at-3 and pass-to-the-3 is the share of tasks the model solves sometimes, not always.
How the grader works
Cline says it runs the agent in one environment and the grader in a second. The verifier restores the protected tests before it runs the agent's patch. A faithfulness review checks that each task matches the user's own request. A fairness review runs a known-wrong patch against the tests. That catches tests that pass work they should not.
The method is Cline's answer to grader cheating. If the agent can read the tests, it can write code that passes them without doing the real work. A separate environment, with a clean restore, is how Cline says it stops that.
What it costs to run
Cline reports an estimated mean cost per task. Opus 5.5 ran at $13.50. GPT 6 Astra ran at $8.64. Gemini 4 Carbon ran at $7.72. Gemini 4 Argon ran at $4.92. Sonnet 5.5 ran at $3.80. DeepSeek V4.1 Flash 1017 ran at $0.10. The spread covers a factor of over a hundred on one benchmark.
Cline's figures are from its own test on 47 tasks, picked and verified by Cline, run by Cline. A buyer choosing a coding agent on these numbers alone is choosing on one vendor's benchmark. The method it describes is the part another team can apply to its own tests. Ask for the pass-to-the-k number before you pick the agent that touches production code.
Source
Cline-Bench: Faithful and Fair Evaluation of Agents on Real User Requests, by Runhui Wang, Cline.
This item was written by an AI system from the linked source. Reveneau is responsible for what it publishes.
Get AI News in your inbox
New developer tools, model and agent releases, and how teams are actually using them to release software. Short, and only when there is something worth reading.


