AI NewsProductivityAnnouncement
Vercel: 60 percent pass rate is 8 percent across five tries
Vercel shows a coffee-ordering agent that passes 60 percent of single tries. Across five tries in a row, the pass rate falls to 8 percent.

Why it mattersA single passing demo is not proof an agent works. The second number, five in a row, tells you if users can rely on it.
A team watches a coding agent order a coffee from its own command-line tool. The order goes through. The team cannot say the agent works. The same agent will fail on another run with the same input, because the model picks words with some randomness. Vercel put a number on this and gave a way to measure it.
The post is called "Agents are users, so your software needs evals". The authors are George Fahmy and Yury Selivanov at Vercel, with Amelia Charles. Their main point is that an agent is a user of your software. It finds your product, picks a tool, calls an interface and either finishes the job or does not. Each step can fail, and one passing trial does not prove the next one will pass.
Two numbers that disagree
The post walks through a worked example called SF Beans. It is a coffee-ordering command-line tool. The agent is Claude Code with Opus 5.5. Three of five trials passed, so the success rate is 60 percent. The post then derives two more numbers from those five trials.
Pass@5 asks if the agent gets it right at least once in five tries. For this example that is one minus 0.4 to the fifth, about 99 percent. Almost certain. Pass^5 asks if the agent gets it right five times in a row. That is 0.6 to the fifth, about 8 percent. Almost never.
The two numbers answer different questions. Pass@5 fits a retry flow. The user runs the agent, it fails, they run it again. Pass^5 fits a background job. The agent runs five tasks and the user comes back later to check. The same agent, on the same tool, is good enough for one and not good enough for the other.
A pyramid for cost
The post groups tests by the number of model calls each one makes. Deterministic tests need zero and run fast. One-turn evals need one call. They check a single decision, for example whether the agent picks the right tool. End-to-end evals need two or more calls across a whole task. Each call costs money. Vercel reports 56 cents per successful trial in the SF Beans example. The number is 78 cents per success once the failed trials are counted too. Running this five times across three models is not free.
For a team shipping a tool
The post ends with what to do. Run the agent against the thing it is meant to use. Vary the request so one eval does not cover every path. Run each eval many times, not once. Pick the metric that matches how users call the tool. Pass@k if the user retries on a failure. Pass^k if the user will not be watching.
The example tool and the numbers are Vercel's own. The post names Harbor from the Terminal-Bench team and Vercel's Sandbox as one way to run many trials at once. The method works without them.
Source
This item was written by an AI system from the linked source. Reveneau is responsible for what it publishes.
Get AI News in your inbox
New developer tools, model and agent releases, and how teams are actually using them to release software. Short, and only when there is something worth reading.
/filters:no_upscale()/articles/llm-reliability-layers-limits-accuracy/en/resources/258figure-1-1793885323099.jpg)

