Image: Hugging Face
Why it mattersA buyer who picks a coding agent on its top-line score may miss a built-in behaviour that breaks real work. Test the agent on tasks that outlive one session.
A coding agent can pass four out of five public tests and still fail a developer's work. DeepSeek's own open-source coding agent is tied for first on Joel Niklaus's harness leaderboard across five scored evaluations. On Terminal-Bench it drops to sixth, because of one behaviour the agent does by design.
Niklaus is a harness optimization researcher at Hugging Face. He added DeepSeek Harness to his leaderboard this week. The leaderboard compares ten agent harnesses running the same models. A difference in score points at the harness, not the model. The New Stack wrote up the finding.
What the agent does
DeepSeek Harness runs a short-lived job registry inside the agent process. DeepSeek's own repository documentation describes it as deliberately short-lived. "Choose it when jobs should live in the harness process and die with it," the docs say. Jobs that must outlive a restart or run across processes need a different backend.
Niklaus confirms the same behaviour in his leaderboard's changelog. In the one-task mode the benchmark uses, the agent's process manager removes background services when the session ends.
Three Terminal-Bench tasks ask the agent to start a service that keeps running after the agent exits. The task's grader then calls that service. On those three tasks, DeepSeek Harness passed two out of 36 attempts across DeepSeek-V4.1-Flash and GLM-5.3-Flash. OpenCode, Pi and Nous Research's Hermes Agent each passed all 36. The service answered the agent's own checks and disappeared before the grader could reach it.
What that cost in the ranking
The three Terminal-Bench tasks knock between 1.5 and 1.7 percentage points off DeepSeek Harness's score, by Niklaus's figures. On DeepSeek-V4.1-Flash at high reasoning effort, that is more than the 1.3-point gap to the leader. The changelog records the behaviour as a harness issue, not an infrastructure fault. The Method page lists it. Niklaus did not re-run the trials. The published scores show what the agent did.
Niklaus launched the Harness Leaderboard this month. He had run 28,849 trials across its original nine harnesses on three open models: DeepSeek-V4.1-Flash, GLM-5.3-Flash and Flint. The leaderboard's main page shows each harness's scores on five evaluations. Each cell carries the number of trials behind it. The New Stack said it had asked DeepSeek for comment. It had not received a response by publication time.
For a team using DeepSeek Harness on real work, the answer is in DeepSeek's own documentation. If a job must outlive the session, pick a different backend.
There is a wider note here on how to buy a coding agent. A headline leaderboard score is averaged across many tasks. The task a team runs every week may sit inside the one evaluation where the agent breaks. A single default setting can cost more than the whole gap to the leader.
Source
Harness Leaderboard by Joel Niklaus (changelog), covered on The New Stack by Amanda Caswell.
This item was written by an AI system from the linked source. Reveneau is responsible for what it publishes.
Get AI News in your inbox
New developer tools, model and agent releases, and how teams are actually using them to release software. Short, and only when there is something worth reading.
