The best model in a new benchmark steered a coding agent through a full task 24.69% of the time

LoopArena, a benchmark from the DreamX team at AMAP, measures something most evaluations skip: how well a model directs another agent through a long piece of work. The repository is Apache-2.0 licensed, has 87 stars, was created on 31 August and last received a push on 3 September. The accompanying paper is on arXiv.

The setup separates two jobs that usually get scored together. One model is the Controller, which reviews progress and decides what happens next. A separate fixed coding agent, the Worker, does the actual editing. Only the Worker can change the repository. The Controller has no coding tools and no access to the evaluator's private information, so the score reflects supervision rather than coding skill.

What was measured

The benchmark runs in three settings of increasing scope. Type I gives the model a frozen snapshot and asks it to choose the best next instruction from four candidates that were already validated by execution, scored over 90 cases. Type II guides the worker through one slice of a real task, and Type III through the complete task, each over 27 cases. Type II and Type III are paired one to one across 11 SCBench and 16 BeyondSWE tasks.

Five models were scored as Controller, all using Qwen3.7-Plus as the shared Worker and Reporter. The authors report these results for version 0.1.0:

GPT-5.5 led on all three settings, at 87.78% contract accuracy on Type I, 51.85% on Type II and 24.69% on the full task. Qwen3.7-Plus scored 72.22%, 48.15% and 23.46%. Claude Opus 4.8 scored 76.67%, 48.15% and 20.99%. DeepSeek-V4-Flash-0731 scored 77.78%, 45.68% and 19.75%. GLM 5.2 scored 74.44%, 37.04% and 16.05%.

The gap the authors point at

Two details are worth more than the ranking. The first is the size of the drop from choosing a good next step to actually finishing: the same model that picks the right instruction 87.78% of the time completes a full task under a quarter of the time. Good local decisions do not add up to a finished task.

The second is a warning about shortcuts. The authors report that fixed control scores 46.91% against 39.51% for no control on Type II, but on Type III both score 18.52%. A control strategy that looked clearly better on the condensed version showed no advantage at all on the complete one. They also report that Type II costs 64.4% less to run on average and produces a similar ordering, with a Spearman correlation of 0.9747, so it is useful as a cheap proxy for ranking while still being the wrong place to draw conclusions about whether control helps.

Supervising and coding are separate skills to test

Teams increasingly run structures where one model plans and reviews while another writes code. This is the first public number many will have seen on how well the supervising half performs, and it is low. The strongest result on full tasks leaves roughly three quarters of them unfinished.

The practical reading is that the two roles need separate evaluation. The model your team trusts to write code has not been tested on whether it can tell when the work is going wrong and say what to do about it. Those are different skills, and on this benchmark the ordering is not the same as on coding leaderboards.

Source

AMAP-ML/LoopArena on GitHub