IBM's prompt-only method beats a fine-tuned 30B agent on τ²-bench retail

A prompt-only method ranks ahead of a model fine-tuned on the same data at the top of a customer-service agent leaderboard. IBM Research reports that stock GPT-4.1, with 292 learned guidelines appended to its prompt, reaches Pass^4 of 67.5 percent on τ²-bench retail. Pass^4 is the share of tasks the agent gets right on every one of four repeated runs, so it is the strict version of the benchmark score. RAFT-30B-A3B, a model fine-tuned on synthetic τ²-bench retail data, reaches 61.4 percent on the same split, so the guideline method is 6.1 points ahead. No weights were touched, and no human annotation was collected.

The agent writes its own playbook from its training-task attempts

ALTK-Evolve is an open IBM library that runs the agent on training-task attempts, then uses the same model to turn successes and failures into short behavioural rules. Rules carry a support count, the number of distinct training tasks each one was extracted from. The agent reads them as a plain Markdown list appended to the domain policy before every conversation on the held-out test set.

IBM reports two variants on τ²-bench. The sup3 core keeps rules with support of three or more, which comes out to 31 for retail and 14 for airline. The lossless set keeps every rule, 292 for retail and 101 for airline. Both run on the benchmark's stock ReAct scaffold.

Retail first with GPT-4.1, airline first with GPT-5.5

On IBM's October 12, 2026 leaderboard snapshot, retail lossless with GPT-4.1 reaches Pass^4 of 67.5 percent and ranks first. Airline is harder. Lossless with GPT-5.5 reaches Pass^4 of 82.0 percent, also first and ahead of every frontier model, which range from 66 to 72 percent. The sup3 core with GPT-4.1 reaches 44 percent on airline, eighth place, ahead of GPT-5.2 without reasoning and close to Claude Sonnet 4.5.

The gap between the two retail numbers is the usable finding. A prompt-level method on a mid-tier model reaches a higher Pass^4 than both a fine-tuned specialist and every frontier model that was tested without the method, on the same benchmark, with no weight updates.

65 percent more tokens per retail conversation, with shorter conversations

IBM gives per-conversation cost at API list prices. Retail baseline runs at 0.084 US dollars, lossless at 0.139 dollars, a 65 percent increase. Airline baseline runs at 0.081 dollars, lossless at 0.107 dollars, a 32 percent increase. The guideline block is the extra token cost; the agent reads the whole playbook into its prompt on every turn.

Conversation length falls. IBM reports lossless trimming retail from 10.59 to 10.16 assistant turns and airline from 9.15 to 8.82. The agent reaches the policy-compliant answer sooner when it already knows the rules the benchmark will test.

The guidelines read as a plain operations playbook

The learned guidelines read as the kind of policy a human operations lead would write after a week of listening to calls. From IBM's retail example: authenticate the user before any order action; confirm every change with a structured summary and require explicit approval before executing it; break refunds into itemised components with timing; take one action per turn; and parse a combined handle like daiki_johnson_9523 into first and last name before looking the user up.

Source

ALTK-Evolve: How learned memory tops the τ²-bench leaderboard on customer service tasks without a frontier model, Ashwath Vaithinathan Aravindan, Vatche Isahagian, Vinod Muthusamy and colleagues, IBM Research on Hugging Face. Library: AgentToolkit/altk-evolve. Benchmark: sierra-research/tau2-bench.