StarSkirmish v0.2 puts top LLMs at 7 percent against human-written bots

A model that writes a bot that beats other LLM-written bots can still win only about one game in fourteen against a careful human engineer. The second StarSkirmish run makes that gap visible: it asks the top large language models to write a StarCraft Brood War bot in one hour, and the one that ranks first wins about 45 percent of its matches overall and 7 percent against the three top human-written bots in the field.

Writing on the StarSkirmish site, Kai McPheeters describes a benchmark that tests long-horizon reasoning and agentic coding, since each model must learn from a round of practice games before writing its final C++ bot. Each model built five bots. Each bot played six games against each of 26 opponents on three maps: Heartbreak Ridge, Benzene and Destination.

The scores, in StarSkirmish's words

GPT-6 Astra came first at about 45 percent, StarSkirmish says, followed by GPT-6.1 Sol at about 42 percent, Gemini 4 Argon at about 39 percent, Claude Opus 5.5 at about 38 percent and Claude Fable 5.1 at about 37 percent. Muse Wonder Full sits at about 25 percent at the bottom of the chart. Scores for Claude Sonnet 5.5, Claude Haiku 5.5 and GPT-6 Luna are not labelled in the body of the post.

Those are overall win rates against all 26 opponents in the field. Against the three top human-written bots in that field, Stardust, PurpleWave and BananaBrain, the ranking shifts. GPT-6 Astra, the top model overall, wins 7 percent against those three. StarSkirmish does not break out the other LLMs against the same subset.

What changed between v0.1 and v0.2

StarSkirmish moved from a shared, third-party harness to one supplied by each lab, with every model set to maximum reasoning effort. McPheeters is explicit that this changes what the benchmark measures. v0.1 asked which model was best at writing a bot inside a common framework. v0.2 asks which lab, model and harness together produce the best bot, and the comparison between the two runs is no longer apples to apples.

The gap between the top LLM score and the top human bots is also the first number here that holds across runs. v0.1 had Astra beating Stardust in a run that was later rolled back after a cheating finding, so a score against those three remains the field's ceiling for the moment.

The benchmark is one person's test on three maps against a known field of opponents, so a lab building its own eval off this would want to add maps and double the game count before quoting the ranking. The useful number for a buyer is the 7 percent against the top humans. The ranking among LLM bots says which lab did best inside this harness today; the gap to Stardust, PurpleWave and BananaBrain says how far the state of the art is from a strong hand-tuned bot, and that is the number that moves in a buyer's work.

Source

StarSkirmish Bench v0.2, Kai McPheeters on StarSkirmish.