Sierra open-sourced Hyper-tau-bench, a test of AI agents that build other agents, and the top model passed 23.9 percent of the tasks

Sierra, the enterprise agent company co-founded by OpenAI board chairman Bret Taylor, open-sourced a new benchmark on 8 September called Hyper-tau-bench. It measures how well an AI developer agent, working on its own, can build a customer-service agent for a simulated business. Sierra published the accompanying paper on 4 September.

What the benchmark asks

Sierra gives a developer agent the raw material of a business: documents, transcripts, an API, and a codebase. The agent has to build a customer-service agent that meets model and cost limits. Sierra then runs the finished agent against unseen customer conversations in airline, retail, telecom, and banking, with tasks like canceling a flight or disputing a fee. It passes when the agent gives the right information and makes the correct change in the underlying systems.

The benchmark carries 53 build tasks. Banking accounts for 35 of them and is the hardest: its policy corpus holds 2,969 individual facts, and one task can depend on 580.

The scores

Sierra ran six combinations of model and coding harness. On the public leaderboard, Claude Opus 5 in Claude Code came first at 23.9 percent, followed by GPT-5.6 Sol in Codex at 22 percent, GPT-5.6 Terra in Codex at 18 percent, Kimi K3 in OpenCode at 17.9 percent, Kimi K3 in Kimi Code at 16.1 percent, and Claude Sonnet 5 in Claude Code at 14.9 percent. None of the six broke 25 percent.

The overall numbers hide sharp differences by domain. Claude Opus 5 reached 72.8 percent on retail, 55.9 percent on airline, and 48.2 percent on telecom, then fell to 5.9 percent on banking. GPT-5.6 Sol did better on banking at 9 percent.

A "Human + AI reference" bar on the chart sits at 82.2 percent. Sierra flags this as an oracle rather than a measure of average human performance: those reference builds were done by an author working with a frontier model and with access to the ground-truth requirements the autonomous agents had to discover on their own.

Where the agents lose ground

Sierra also looked at what the developer agents actually did. Four patterns come up in the paper. They stopped researching too soon: in banking, agents opened fewer than 80 of about 1,700 available files. They almost never asked the business for missing information, with such questions making up 0.3 percent of tool calls; on tasks where the reference build scored 95 to 100 percent, builds that asked no questions scored 5 percent, one question 15, two questions 25. They handled the finished agent's cost budget badly, with most builds spending 45 percent of the allowed amount. And they rarely tried more than one design: 92 percent used a single loop of one model deciding whether to answer or call a tool.

One data point from a telecom experiment shows how much design choice mattered. Adding a single sentence to the developer agent's instructions, suggesting a different architecture, lifted its score from 31 percent to 67 percent.

Sierra researchers Ben Shi and Keshav Dhandhania write in the paper that these are familiar engineering mistakes: "The failures mirror ones human agent developers see." The agents also tended to pick familiar tools, with 96 percent of Codex builds choosing an OpenAI model to power the finished agent and 13 percent of Kimi builds doing the same.

Anyone building an agent to build agents now has a public, reproducible test with a low ceiling and room to move.

Source

Sierra research blog: Hyper-tau-bench, paper, leaderboard, and code. Reporting by The New Stack.