Grafana borrows SLOs from reliability engineering to measure AI agent behaviour

A response from an AI agent can come back in 800 milliseconds, cost almost nothing, pass every technical check, and still be confidently wrong. Grafana's Ivana Huckova proposed on the company blog that the right tool for that gap is already in every reliability engineer's hands: a service-level objective and an error budget. Set the target, measure the agent against it, and let the remaining budget decide whether the team ships the next prompt change or stops and fixes quality first.

What the SLO measures

The post names five behavioural dimensions Grafana scores conversations on: groundedness in the supplied context, fulfillment of what the user asked for, toxicity, prompt injection resistance, and personal information leakage, plus the usual latency and cost. Deterministic checks like regex catch the easy cases such as a social security number in a reply; a model-based judge grades the subtler ones such as whether the answer was actually grounded. Treating quality as several dimensions, instead of one score, lets a team set a separate target for the thing that matters on its application. An assistant can be well grounded but expensive, or helpful but occasionally toxic, and the team decides which dimension gets the tightest objective.

The example target, and the 5 percent

Huckova's worked example sets a target of 95 percent of conversations judged fulfilled over a 30-day window. The 5 percent gap becomes an error budget. When the budget is full, the team has permission to experiment with prompts, model swaps and new features. When it runs low, the team eases off new work and spends attention on quality until the budget recovers. That is the same discipline teams already apply to request failures and latency; what moves is the signal being counted, from an HTTP 500 to a conversation a judge marked as not fulfilled.

Where the pieces run

Grafana Assistant inside Grafana Cloud is the tool the post uses to create evaluations: a team describes what good looks like in natural language and the Assistant turns that into evaluation code, then runs it against real conversations and recommends an SLO target based on the scores it finds. Agent Observability is the production side, where the deterministic checks and the model-based judges sit next to each other and feed one dashboard. The post was published on 24 September 2026 and InfoQ covered it on 16 October.

A reader who already runs SLOs on an API now has a template for the agent in front of that API. The number 95 percent is not special; the discipline is: write down what fulfilled means for this application, pick the window, decide which dimension carries the tightest budget, and watch what the budget does to the shipping cadence. For a team building any user-facing model-backed feature, that is a shorter path to a usable production signal than inventing a bespoke quality metric from scratch, and it does not require the team to adopt Grafana to apply it.

Source

Ivana Huckova, "What if your agent's hallucinations had a budget? How to start using SLOs for agent behavior", Grafana Labs. Coverage: Craig Risi, "Grafana applies SLOs and error budgets to AI agent behaviour", InfoQ.