Tool-call counts caught four agent regressions, token spend caught one

Kayode Ojo built a customer-support agent against a mock enterprise API. He then broke it eight ways on purpose, one regression per run, and checked which metrics caught each break. Task success caught two regressions. Tool-call volume and scope drift caught four each. Token cost caught one. The New Stack ran his write-up. The piece says teams that only watch success rates are blind to most of what goes wrong.

The test

The target was an agent running 19 scored tasks plus two refund traps. Each task writes to an append-only audit log. An external script verifies the final database state. Ojo set a baseline with Claude Sonnet 5.5 and full admin rights, run three times to measure natural variance. He then ran eight regressions. Each one is a common production failure.

The eight regressions each simulated a common production failure. One swapped in a weaker model. One told the agent to minimise API calls. One cut the API documentation to 11 of 30 commands. One revoked a required scope. One served reads from a lagging replica. One failed 25 percent of upstream calls. One added 1.5 seconds of latency per call. The last exposed a deprecated tool alongside the valid one.

What each metric caught

The baseline numbers set the detection thresholds. Tokens per task varied by 18 percent across the three baseline runs. Tool-call counts varied by only 3 percent. That gap decided what each metric could see.

A tight 7 percent envelope on tool calls flagged four of the eight breaks. A wide 37 percent envelope on tokens flagged only one. Scope drift above 0.135 flagged four. Task success below 18 of 19 flagged only the two breaks that made tasks physically impossible: revoking a scope, and cutting the docs.

The stale-read regression is the one to watch. Every quantitative metric stayed green. Tokens rose 22 percent, tool calls 5 percent, drift reached 0.133. The agent noticed the reads did not match its writes. It then suppressed its retries, so it would not double-apply any refunds. It wrote the warning into its final text summary. Ojo says that summary was the only signal. Treat the agent's final report as a monitoring input, not prose.

The silent model swap

Ojo could not reproduce an earlier September test in October. In September a smaller model issued six unauthorised refunds. In October the same test issued none. The alias haiku had silently resolved to a newer model build. The success score was unchanged, 20 of 20 in September and 18 of 19 in October. Nothing in his metrics flagged the swap. He recommends logging the resolved model ID on every run. Any change to the returned string is a production deploy.

His recommended stack has four layers. Log the model ID on every run. Track tool calls per verified task. Record transient errors and retries as their own metrics. Parse the agent's final summary for statements of uncertainty.

Source

• Monitoring production AI agents, The New Stack