AI NewsProductivityAnnouncement

Datadog adds live traffic A/B tests for AI agents

Datadog can now split live traffic between two versions of an AI agent and compare quality, cost and errors.

AI News

Editorial2 min read

LinkedInX
Datadog illustration of Online Experiments comparing two AI agent variants running against production traffic

Image: Datadog

Why it mattersTeams that improved an agent on a fixed test set often find the change acts differently on real user requests. Live A/B tests close that gap.

A team can improve an AI agent on a fixed test set and still miss what real users do. Datadog says it now has an answer. The company added Online Experiments to its Agent Observability product. A team runs two versions of the same agent in production. Traffic gets split between them at random. Response quality, cost, latency and error counts come back for each side.

Alex Barksdale, Yakov Shapiro, Demetri Pananos and Miguel Tulla Lizardi wrote the post on Datadog's engineering blog. Datadog describes the gap it is trying to close. Offline evaluation tests an agent against a dataset the team wrote. The dataset can only hold what the team thought of. Live users bring requests the team never anticipated. The agent behaves differently on those.

How it works

Datadog's Feature Flags product handles the traffic split. A team wires both versions into the application. The team then chooses what share of users sees the change. The service can raise the share over time without a redeploy.

Scores come from two sources. Agent Observability runs its evaluators against both variants on the same traffic. That gives response quality, accuracy and completeness. The Experiments product adds the operational numbers. A team sees cost, token use, latency and workflow errors on the same chart. When the output of one variant looks suspicious, the team can pull up the trace for that single run. The trace shows which tools the agent called, what arguments it passed, and what each tool returned.

What the dogfood run actually found

Datadog ran the system on its own investigation agent. The two variants were GPT-5.6 Luna and GPT-6 Luna. Traffic went to each at 50 percent. A model-based judge scored each investigation from one to five on accuracy, completeness and whether the conclusion matched the evidence.

The result Datadog reports is a 44.3 percent cut in estimated cost for GPT-6 Luna. Quality scores stayed close. The control scored 3.84, the variant scored 3.86. The newer model also showed increases in execution time and workflow errors. The confidence intervals were too wide to call a regression. The traces told a sharper story. GPT-6 Luna passed invalid tool parameters more often than its predecessor and usually recovered on retry. The quality score did not pick up the retries. Datadog used that finding to tighten tool handling before changing the production model. The failed calls then became new regression cases for the offline suite.

Datadog names its competition in the same post. LangSmith and Langfuse do tracing and evaluation. LaunchDarkly does feature rollouts. Datadog says its Online Experiments puts the three jobs in one tool. The traces, metrics and feature flags are already tied together.

Source

Datadog blog: Compare AI agent changes with Online Experiments in Agent Observability.

SourceDatadog

This item was written by an AI system from the linked source. Reveneau is responsible for what it publishes.

Share
LinkedInX
Start a project