IBM Research measures a 24-point consistency gap on AppWorld and halves it with automatic guidelines

IBM Research published a post on 15 September 2026 on measuring and closing the gap between an agent's average success rate and the fraction of tasks it succeeds on every time. On the AppWorld test_normal benchmark, a ReAct agent using GPT-4.1 succeeds on 77.4 percent of runs across five repetitions, and succeeds on all five runs for only 53.0 percent of tasks. IBM calls the difference the consistency gap, and reports halving it from 24.4 points to 12.0 points using guidelines its open ALTK-Evolve library writes automatically.

The metric almost nobody reports

Agent leaderboards report Mean@k, the average pass rate across k runs. IBM argues the useful number for anyone shipping an agent is Pass^k, the fraction of tasks where the agent succeeds on every one of k runs. Pass^k is the pessimistic mirror of the familiar Pass@k (at least one success): it demands that every attempt succeed. Pass^k is always less than or equal to Mean@k.

For the AppWorld benchmark, IBM records Mean@5 of 77.4 percent and Pass^5 of 53.0 percent for a ReAct agent on GPT-4.1. On the hard subset, the gap reaches 30 points. The ReAct agent ran at temperature 0.0, so the variance is not ordinary sampling. IBM writes that the source is the shape of the model's token distribution at each decision step: sharp distributions resolve the same way run after run, and flat ones can reorder under small platform-side perturbations like GPU floating-point non-associativity and request batching. A trajectory chains dozens of decisions, so the per-step flip rate compounds.

The diagnostic and the fix

IBM's tool, the Consistency Analyzer, replays each decision step in one recorded trajectory with k completions drawn from the same context (default k=5). It does not re-run the whole task, needs no ground truth, and does not require model logits. The output is a per-step consistency score that flags decision points where the model was one token-sample away from a different action.

Every flagged step becomes a candidate guideline in the existing ALTK-Evolve format, which the agent retrieves at inference time. IBM shows a real example generated by GPT-4.1 from an AppWorld task about counting bucket-list items in a note: the guideline tells the agent to use a line-anchored regex against checkbox markers rather than a plain substring count, because note titles often repeat the marker in a legend line. That is a targeted patch to a flat step in one trajectory.

What the intervention buys

IBM reports that the consistency guidelines halved the gap from 24.4 points to 12.0 points. Same-task Pass^5 rose 16.0 points, and similar-task Pass^5 rose 13.0 points, so the guidelines generalise past the exact trajectory they came from. IBM writes that average accuracy did not fall, which matters because a naive fix for flakiness is to trade probability mass for determinism at the cost of the top-line score.

The full methodology and evaluation are in a companion technical report on arXiv linked from the post. The library is open on GitHub as IBM/altk-evolve.

Where a team can use it

A demo failure that never reproduces in testing usually has this shape: the flakiness sits in the decision distribution and never touches the code. A team building a production agent can capture one representative trace, run the Consistency Analyzer, and see which steps in it are near-ties. Repair work then targets those steps. Reporting Pass^k alongside Mean@k in an internal eval is a change any team can make this week, and it will surface the tasks that a bigger model will not fix.

Source

Your Agent Aced the Task. Will It Do It Again?, Evelyn Duesterwald, Lilian Ngweta, and colleagues, IBM Research on Hugging Face, 15 September 2026. Library: IBM/altk-evolve.