Cost, time and step limits as agent evals

HAL is a leaderboard (a public ranking table) of AI agents run by a team at Princeton University. As read on 30 September 2026, it listed two 2025 runs on a benchmark (a public set of tasks with a fixed scoring method) named AssistantBench. Both rows used the same agent software, Browser-Use. With the o3 model at its medium setting (April 2025) the agent scored 38.8 percent, and the cost shown beside the score was 15.15 US dollars. With GPT-5 at its medium setting (August 2025) it scored 35.2 percent at 41.69 US dollars [1]. The second row cost 2.75 times as much as the first (41.69 divided by 15.15) and scored lower.

Two rows show one comparison and no trend. The site says its leaderboard is paused, so the figures are a record of those 2025 runs and say nothing about current models. A report that listed accuracy alone would hide which run was the costly one. This page is about putting cost, time and step counts into the eval itself, each with a limit that fails a run when it is exceeded.

Why accuracy alone is the wrong score for an agent

The main evidence is a July 2024 paper titled "AI Agents That Matter", by Sayash Kapoor, Arvind Narayanan and three co-authors at Princeton University. They ran published coding agents five times each on HumanEval, a benchmark of 164 programming problems, and recorded the cost of each agent beside its accuracy [2]. Three of their results:

• For accuracy the authors describe as similar, cost differed by "almost two orders of magnitude" [2]. Two orders of magnitude is a factor of one hundred.
• Two complex agents, Reflexion and LDB, cost over 50 percent more than a simple comparison method the authors call the warming strategy, and a third, LATS, cost over 50 times more [2].
• The authors found "no significant accuracy difference between our warming strategy and the best-performing agent architecture" [2].

Their conclusion is that "useful agent evaluations must control for cost", because accuracy "can be improved by scientifically meaningless methods such as retrying" [2]. The models were from the GPT-3.5 and GPT-4 period and the result comes from one benchmark, so the exact ratios belong to that test. The reasoning applies to any agent: a score can be raised by spending more, so a score reported without its cost cannot be compared with another.

A second study, the HAL paper of October 2025, covered 21,730 agent runs across 9 models and 9 benchmarks, at a total cost the authors give as 40,000 US dollars and mark as approximate. In 21 of 36 combinations of model, agent and benchmark, a higher reasoning effort setting, which lets the model do more computation before it answers, produced "equal or lower accuracy" [3].

What to measure on every run

Record five things for each run of each task.

1. Model usage cost. Models are billed by the token, a word piece the model reads or writes. Cost is the tokens used multiplied by the provider's price, added up over every model call in the run.
2. Wall-clock time. The elapsed time from the first request to the final result.
3. Number of steps. One step is one turn of the agent's loop: the model chooses an action, the action runs, the model reads the result.
4. Number of tool calls. A step can contain more than one call, and a tool can have its own charge.
5. Whether the task passed.

Anthropic's guide to agent evals lists similar quantities as things a fixed set of tasks lets a team track: "latency, token usage, cost per task, and error rates" [4]. Latency is the waiting time for a response. The Berkeley Function Calling Leaderboard publishes an estimated cost in US dollars and latency in seconds beside each model's accuracy [5].

Report the spread as well as the average. A February 2026 paper on agent reliability ran each task 5 times and found "high variance in token and compute usage across runs" [6]. Run each task several times, as agent reliability across repeated runs describes, and record the middle value and the largest value across the runs.

A limit turns a measurement into a pass or a fail

A measurement becomes an eval when it has a threshold. Set one limit for each measure, and count a run that exceeds it as failed even when the result is correct.

Limit	What it protects against	How to set it
Step limit	A loop, and runs that take many needless steps	From the step counts of passing runs, plus a margin
Tool-call limit	Repeated calls to a service that charges per call or restricts how many it accepts	For each tool, from passing runs
Cost limit per task	A run that costs more than the task is worth	From the value of the task to the business
Time limit	A user or another system left waiting	From how long the user will wait

Published tests set limits of this type. tau-bench stopped each task at 30 agent actions [7]. The OSWorld benchmark, which tests agents on real desktop applications, set "the maximum steps of interaction to 15 and the maximum time limits to 30 minutes for all tasks" [8]. The authors of SWE-Agent, as cited in "AI Agents That Matter", set a cost limit of 4 US dollars per task [2]. METR runs its agents "up to a token and time limit" [9].

Researchers chose those values for a benchmark. No source used for this page recommends a step or cost limit for a product as a number, so any figure for your product comes from your own runs. The method:

1. Run the task set with repeats, with only a high safety limit in place.
2. Look at the passing runs alone. Note the largest step count, cost and time among them.
3. Set each limit above that largest value by a margin you choose and write down.
4. Run again, and read every run that now fails on a limit alone.

Here is an invented illustration. Suppose the passing runs of a refund task took between 6 and 11 steps across 40 runs. A step limit of 15 allows an unusual valid route and still ends a run that has stopped making progress.

How a step limit stops a loop

A loop is an agent repeating the same step: it calls a tool, gets a result it cannot use, and makes the identical call again. Each repeat costs tokens and time, and with no stopping rule the run continues until something outside the agent ends it.

Anthropic's guide to building agents names the control: "it's also common to include stopping conditions (such as a maximum number of iterations) to maintain control" [10]. An iteration is one turn of the loop. The same guide says the autonomous nature of agents "means higher costs, and the potential for compounding errors" [10], and a loop produces both at once.

In an eval, use the limit in three ways:

• End the run. When the step count reaches the limit, the run stops and is recorded as failed, with the reason "step limit reached".
• Detect the repeat earlier. A code check on the recorded steps can flag the same tool called with the same arguments several times in a row, before the limit is reached. A tool error is one situation where an agent can start repeating itself, which is why error handling is one of the five checks in tool-call evals.
• Check what happens at the limit. An agent that reaches its limit should stop, say what it completed, and hand the task to a person. Write a case for that behaviour.

Limits work beside other controls, which are covered in permissions and confirmation steps for AI agents.

How much an agent run costs

Cost depends on the task, the model and the date, so the only figure you can rely on is one you measure. Published figures show the range.

• tau-bench, 2024. With a gpt-4o agent and a gpt-4 simulated user on the retail tasks, the authors reported 0.38 US dollars per task for the agent and 0.23 for the simulated user, and put the cost of one trial of every task at 200 dollars, a figure the paper marks as approximate [7]. Both figures are repeated as the paper states them: 0.61 dollars multiplied by the 115 retail tasks is 70.15 dollars, and the passage does not explain the difference.
• SWE-Agent, 2024. With a limit of 4 US dollars per task, running the agent on the entire benchmark "could cost over USD 8,000 for a single evaluation run", as cited in "AI Agents That Matter" [2].
• HAL, 2025. 21,730 runs for a total the authors give as 40,000 US dollars and mark as approximate [3]. Dividing that total by the count gives 1.84 dollars per run. That is an estimate, because the total is approximate, and it is an average across unlike benchmarks.

One pass of an eval set costs the number of cases, times the repeats for each case, times the cost of one run. As an illustration with an invented case count: 50 cases, 5 repeats each, at the 0.61 dollars per task reported for tau-bench, comes to 50 x 5 x 0.61 = 152.50 dollars for one pass. The cost of the product itself is covered in what it costs and how long it takes to release an AI agent, and the budget for evals in what AI evals cost and who should own them.

How long a task an agent can finish: METR's time horizon

A time limit depends on how long a task an agent can complete at all. METR, a research organisation, measures this with a figure it calls the time horizon. Its paper defines the 50 percent version as "the time humans typically take to complete tasks that AI models can complete with 50% success rate" [11]. METR's page adds that the figure is "a measure of the difficulty of a task, rather than the time an AI spends to complete the task" [9]. A two-hour horizon means the agent succeeds half the time on tasks that take a skilled person two hours.

METR's figures differ between its publications, so each is given with its source and date [11] [12] [13] [9].

Source	Date	What it states
Paper, abstract	First version 18 March 2025, read in its fourth version of 10 July 2026	Claude 3.7 Sonnet has a 50 percent horizon of 50 minutes, and the horizon has doubled every seven months from 2019 on. The abstract marks both figures as approximate.
Paper, body	Same paper	Doubling time of 207 days for the 50 percent horizon over 2019 to 2025, and 204 days for the 80 percent horizon
Blog post	19 March 2025	The same model's horizon is one hour, which the post marks as approximate
Time Horizon 1.1	29 January 2026	Task set grew from 170 to 228 tasks. Doubling time 196.5 days over the whole period, and 130.8 days from 2023 on (interval 107 to 161)
Data file	Read 30 September 2026	Doubling time 187.778 days over the whole period, and 128.744 days from 2023 on (interval 104.428 to 158.012)

Three facts from these sources matter more than any single value.

• Higher reliability means a shorter horizon. The paper reports that models' 80 percent horizons are 4 to 6 times shorter than their 50 percent horizons [11].
• Success falls as tasks get longer. In March 2025 METR reported that the models of that date succeeded on almost all tasks that take a person less than 4 minutes, and on fewer than 10 percent of tasks longer than a threshold that METR gives as 4 hours and marks as approximate [12].
• The estimates are wide and they change. METR's data file, as read on 30 September 2026, gives the model listed as gpt_5_4, released on 5 March 2026, a 50 percent estimate of 341.74 with an interval of 186.58 to 768.78, and an 80 percent estimate of 53.88. The file prints no unit, and the scale matches minutes. METR's January 2026 post says its confidence intervals are still wide [13], and its page says measurements above 16 hours "are unreliable with our current task suite" [9].

METR's March 2025 post describes its tasks as multi-step software and reasoning tasks [12], so a horizon for another type of work has to be measured on that work. For your limits, the practical reading is this: if a task takes a skilled person several hours, plan for a lower pass rate than on short tasks, test it with repeats, and set the time and step limits from passing runs. Agent benchmarks explained covers the benchmark itself.

Put the same limits into the released agent

The released agent should stop at the same step, cost and time limits that the eval used, so that the eval describes how the product behaves. A production run that reaches a limit is also a candidate for a new eval case, as turning production traces into eval cases describes. The rest of the method is in the AI agent evals guide. A related post covers what founders get wrong about AI agents.

How Reveneau applies this

At Reveneau, all code is written by AI, and every change must pass a large eval suite written from the specification before the code. For an agent, that suite records cost, time, steps and tool calls for every run, and each task has a written limit for each of them. A run that returns the right result over its limit fails. We report cost and time beside the pass rate, so that a change which raises accuracy by spending more is visible as such. The run time of the eval suite matters as well: we grade the judged checks with Jev, TypeSafe AI's decision model, and on Reveneau's own suite the run is ten times faster than with its previous language-model grader.

We use AI instead of hiring more engineers, so a build takes a small team, and that saving goes into the client's price. Reveneau, as a company, takes responsibility for the whole project through production and after release. To set limits for an agent you are planning, see AI development at Reveneau or contact us.