A new study measures coding agent harness parts one at a time across 176 settings, and says which parts matter for which model

A paper posted to arXiv on Wednesday takes apart the harness that coding agents run inside and evaluates each part on its own. Run-Ze Fan and eight other researchers vary three components, planning, action space and context management, one at a time, across 176 matched settings and four models, on SWE-Bench Verified and Terminal-Bench 2.1. The paper's central point is that harness design is usually reported as one number for the whole system, and that number hides which part is doing the work.

What the study varies, and holds

The execution loop is held fixed while three components change: planning, action space and context management. The paper covers five context-management strategies, four context-window budgets, and targeted ablations of planning and action space. That gives 176 matched settings across the four models. The authors say a matched setting is a direct comparison, meaning any change in the score can be traced to the one component they changed, so the model and the task are held constant.

What each component earned

Context management. The value grows as the context-window budget shrinks. The authors say most of the benefit comes from preventing the run from failing when the context overflows. Better summarisation quality contributes little on top of that. Among the five context-management strategies tested, staging rule-based elision before LLM-based summarisation was the most efficient overall. Making the elided content recoverable, so the model could pull earlier text back in, added machinery that the models rarely used and did not improve accuracy.

Planning. For weaker models, planning raises accuracy. For stronger models, it saves cost with almost no change in accuracy. So the same "add a planner" advice pays off differently depending on which model runs behind it, and a team moving from a weaker to a stronger model can drop planning without losing accuracy, only cost.

Action space. Predefined tools help models with weaker bash proficiency. Models that are bash-capable perform well with a bash-only interface, at a lower cost, especially on command-line tasks. Extra tools cost tokens too, in the room they take up for their definitions and calls.

The authors close with a note on trajectories: context management extends the execution trajectory without changing much about how the agent behaves, planning changes where the trajectory stops, and the action space changes the granularity at which the agent writes code. That is useful when a team is debugging why one harness produces the same answer as another with three times the token count.

The paper's argument, backed by its 176 settings, is that a good harness for a small open-weight model looks different from a good harness for a frontier model, and both differ again at a lower context budget. Copying a harness across models is how teams end up paying for planning they no longer need, or tool definitions their model was already going to skip. The full paper is 43 pages.

Source

An Empirical Study of Harness Design for Coding Agents by Run-Ze Fan, Zihao Zhang, Simin Ma, Yebowen Hu, Shouju Wang, Kaiqiang Song, Fei Liu, Hamed Zamani and Xiaoyang Wang. Posted on arXiv on 17 September 2026.