Google open-sources AQuA, a reference agent for diagnosing production agent failures

A coding team releases an agent, writes evals for it, and watches the dashboard show green. A real user then picks a hotel, says no beef, and gets a trip with a steak dinner. The eval never caught it. Google just released a reference agent for exactly that problem.

Google published AQuA, the Ambient Quality Agent, on 13 November. It is a reference implementation under the google/adk-recipes repository on GitHub. The repository uses the Apache 2.0 licence and had 10,614 stars at release. AQuA runs as a sidecar in Google Cloud. It reads conversations from an existing agent's telemetry.

What it does, in five stages

InfoQ, reporting on Google's release, describes a five-stage pipeline: sampling, reviewing, clustering, verifying, and tracking. AQuA samples production conversations. It reviews each one with a large model against the agent's own configuration. It then clusters the findings. It verifies each cluster against an immutable snapshot of the agent taken at deployment. It tracks the resulting issues over time. The project's README confirms it reads conversations from BigQuery through Google's ADK analytics plugin and Cloud Telemetry datasets.

The output is a list of insights. Each insight has a status of NEW, RECURRING or RESOLVED, and a history of when it last occurred. Developers can open a dashboard to see the failing traces and the tool calls that led to them. A chat interface sits next to the dashboard for root-cause analysis.

The numbers Google gave

Google ran AQuA on a travel assistant that had 32 production sessions. InfoQ reports the result. AQuA found 42 findings grouped into 9 candidate clusters, rejected 3 as unsupported, and verified 6 as real issues. Two were worth acting on. Seat selection was being bypassed in 15 of 32 sessions. Dietary preferences were being ignored in 7 of 32. The team changed the prompt. Seat bypasses fell from 15 to 2. Dietary omissions fell from 7 to 0. The number of sessions passing every check rose from 5 to 13.

Google's own statement, quoted by InfoQ, draws a line between a model's guess and a verified cause. Google says an insight is its citations, not a self-reported score. AQuA holds each cluster to evidence from a session before it lets a finding through.

What this changes

Eval-driven development catches regressions a vendor can predict before release. It cannot catch an agent that returns a successful response while quietly ignoring a user's requirement. The trace looks correct on the outside. AQuA is Google's first open reference for finding that class of failure from production traffic.

An AI engineering lead reading this can start in one of two places. The first is to adopt AQuA directly if the agent already runs on Google ADK with BigQuery telemetry. The second is to copy the shape onto an agent built on a different stack. That shape is the five-stage pipeline and the citation-first design. The repo is the first public blueprint for this work. A team that writes its own version now has something to measure its design against.

Source

google/adk-recipes/core/python/ambient-quality-agent, Google. Reporting via Google Open Sources Ambient Quality Agent (AQuA) to Diagnose Production AI Agent Failures, Leela Kumili, InfoQ.