AI NewsProductivityAnnouncement

Vercel: 60 percent pass rate is 8 percent across five tries

Vercel shows a coffee-ordering agent that passes 60 percent of single tries. Across five tries in a row, the pass rate falls to 8 percent.

AI News

Editorial2 min read

LinkedInX
Diagram from the Vercel post showing the agentic funnel from discovery to success

Why it mattersA single passing demo is not proof an agent works. The second number, five in a row, tells you if users can rely on it.

A team watches a coding agent order a coffee from its own command-line tool. The order goes through. The team cannot say the agent works. The same agent will fail on another run with the same input, because the model picks words with some randomness. Vercel put a number on this and gave a way to measure it.

The post is called "Agents are users, so your software needs evals". The authors are George Fahmy and Yury Selivanov at Vercel, with Amelia Charles. Their main point is that an agent is a user of your software. It finds your product, picks a tool, calls an interface and either finishes the job or does not. Each step can fail, and one passing trial does not prove the next one will pass.

Two numbers that disagree

The post walks through a worked example called SF Beans. It is a coffee-ordering command-line tool. The agent is Claude Code with Opus 5.5. Three of five trials passed, so the success rate is 60 percent. The post then derives two more numbers from those five trials.

Pass@5 asks if the agent gets it right at least once in five tries. For this example that is one minus 0.4 to the fifth, about 99 percent. Almost certain. Pass^5 asks if the agent gets it right five times in a row. That is 0.6 to the fifth, about 8 percent. Almost never.

The two numbers answer different questions. Pass@5 fits a retry flow. The user runs the agent, it fails, they run it again. Pass^5 fits a background job. The agent runs five tasks and the user comes back later to check. The same agent, on the same tool, is good enough for one and not good enough for the other.

A pyramid for cost

The post groups tests by the number of model calls each one makes. Deterministic tests need zero and run fast. One-turn evals need one call. They check a single decision, for example whether the agent picks the right tool. End-to-end evals need two or more calls across a whole task. Each call costs money. Vercel reports 56 cents per successful trial in the SF Beans example. The number is 78 cents per success once the failed trials are counted too. Running this five times across three models is not free.

For a team shipping a tool

The post ends with what to do. Run the agent against the thing it is meant to use. Vary the request so one eval does not cover every path. Run each eval many times, not once. Pick the metric that matches how users call the tool. Pass@k if the user retries on a failure. Pass^k if the user will not be watching.

The example tool and the numbers are Vercel's own. The post names Harbor from the Terminal-Bench team and Vercel's Sandbox as one way to run many trials at once. The method works without them.

Source

Vercel: Agents are users, so your software needs evals.

SourceVercel

This item was written by an AI system from the linked source. Reveneau is responsible for what it publishes.

Share
LinkedInX
Start a project