Turning production traces into eval cases

Take an invented furniture shop whose support agent tells a customer, "Your refund has been issued." Three days later the customer writes again: no money has arrived. Someone opens the record of that conversation and finds that the agent called the order lookup tool, made no call to the refund tool, and still reported success. The shop and the incident are an illustration.

That record is a trace. The shop can correct the agent's instructions and stop there, and the same fault can return with the next change. Or it can turn the trace into a test case that runs before every release. This page is the method for the second choice. It is part of the AI agent evals guide.

What a trace has to record

Anthropic's eval guide defines the record this way: "A transcript (also called a trace or trajectory) is the complete record of a trial, including outputs, tool calls, reasoning, intermediate results, and any other interactions" [1]. A trial is one attempt at one task. The page on why testing an agent differs from testing one model answer explains the term in full.

To be reusable as a test, a trace needs six things:

1. The input that started the run.
2. Every model call, in order.
3. Every tool call, with its arguments and the result the tool returned.
4. The final reply to the user.
5. The versions in use: which model, which instructions, which tools.
6. Time, cost and step count, which cost, time and step limits turns into checks.

Item 5 is easy to leave out. Without it, nobody can say whether a failure belongs to the current version or to one that was replaced last month.

A standard format: OpenTelemetry's conventions for generative AI

OpenTelemetry is an open standard, run as a Cloud Native Computing Foundation project, for recording what software did while it ran. Each recorded step is called a span. Its semantic conventions are agreed names for the things recorded, so that a trace written by one tool can be read by another.

One set of these conventions covers generative AI. As read on 30 September 2026, it covers model calls, agent runs, tool executions, errors and measurements of token use (a token is the unit of text a model reads and writes) [3]. The agent page defines named steps that include invoke agent, plan and execute tool, and the events page defines an event for recording an eval result [3].

Three facts about its condition matter before you depend on it:

• The status line of the conventions reads "Development" [3]. We could not open the page that defines that label, so we make no statement about what it guarantees.
• The conventions moved in 2026 to their own GitHub repository (a shared store of files). The page on the OpenTelemetry website that used to hold them is now titled "Moved: Generative AI semantic conventions" and says it "is no longer maintained in this repository" [3].
• In the new repository's introduction file, the heading "Schema URL" has one word under it: "TODO" [3].

Our position: use the conventions, because a shared format lets you change tracing tools and keep your old traces readable. Keep the mapping from your own field names to the convention's names in one file, so that a renamed field is a one-line change.

The six steps from a production run to a permanent case

Step	Who does it	What is stored
1. Record	The application, automatically	One trace per run, with message content only where you chose to record it
2. Sample	A written rule, plus a person choosing extra runs	A review list, with the reason each trace was chosen
3. Review	A person who knows the subject area	A label per trace (failure or acceptable) and the reason
4. Write the case	The reviewer and an engineer	Starting state, input, expected outcome, link to the source trace
5. Remove personal data	An engineer, checked by the reviewer	The case, with invented values in place of real ones
6. Re-run	The test system, on every change	Pass or fail per case, per version of the agent

Which traces to sample

Nobody can read every trace, so a rule picks them. The vendors' documentation starts from runs that went badly. LangChain's LangSmith documentation says to "filter the most interesting traces, such as traces that were tagged with poor user feedback, and add them to a dataset", and describes automation rules that add a trace when it has, for example, a low feedback score [5]. (A dataset here is a stored set of test cases.) Langfuse's documentation describes the same practice: "A common workflow is to select production traces where the application did not perform as expected" [6].

Add a second rule: a random sample of runs nobody complained about. Users report the failures they can see, and an agent that wrote a wrong delivery date into a record would get no complaint until the delivery failed. A fixed number of random traces each week, read by a person, finds failures that no user reported. Choosing what to look for in those traces is the subject of error analysis before metrics.

The set can start small. Anthropic's guide says: "20-50 simple tasks drawn from real failures is a great start" [1].

How to write one confirmed failure as a case

A reviewer who knows the subject area reads the trace and decides whether it shows a failure. If it does, four things are written down.

The starting state. What the order and the customer record looked like before the run. The case runs in a test environment rebuilt to this state each time, as test environments for agents explains.

The input. The user's message or messages, with personal data replaced.

The expected outcome. State it as a fact about the world after the run: a refund record exists for this order, for this amount. A correct agent can word its reply in many ways, so the agent's production wording stays out of the pass condition. Anthropic's test for a good task applies: "A good task is one where two domain experts would independently reach the same pass/fail verdict" [1]. Langfuse's documentation gives the job to a reviewer: "you let an expert add the expected output" [6]. The choice between checking the result and checking the steps is covered in outcome evals vs trajectory evals.

A link back to the source trace, so a later reader can see where the case came from.

Then run the new case several times against the version of the agent that produced the failure. It should fail at least once. A case that passes every time on that version is testing something else and needs rewriting before it is kept.

What four tools' own documentation says

Each row below is a vendor describing its own product, as read on 30 September 2026. We have made no comparison of how well these features work.

Tool	What its documentation says	Detail to know
LangSmith (LangChain) [5]	"A common pattern for constructing datasets is to convert notable traces from your application into dataset examples."	Reviewers can change the inputs, outputs and reference outputs (the expected answers) before a trace is added
Langfuse [6]	An "Add to dataset" control on any recorded step of a production trace	A dataset item keeps the identifier of its source trace
Braintrust [7]	"Build datasets from production logs, user feedback, manual curation"	A record's input "can hold a reference to a logged trace" in place of a copied value
Arize Phoenix [8]	Any recorded step, or group of steps, can be added to a dataset from the interface	The reviewer can "make any changes you might need to make before saving the example"

The Braintrust detail raises a design question. A case that refers to a stored trace depends on that trace still existing. Delete a source trace in a test and see what happens to the case.

Personal data: remove it before the trace becomes a case

A production trace holds what the customer typed. The OpenTelemetry conventions say so directly: "Model instructions, user messages, and model outputs are considered sensitive and are often large in size" [4]. They state the rule for tools that record traces: such tools "SHOULD NOT capture them by default" and "SHOULD provide an option for users to opt in" [4], meaning to switch the recording on by choice. The fields that hold messages have the requirement level "Opt-In" [4]. This text has the same Development status as the other conventions.

Two vendors document ways to limit recording at the source. LangSmith's documentation describes hiding inputs and outputs completely, or rule-based masking, which replaces the sensitive parts with a placeholder [9]. Langfuse's documentation says to "redact sensitive information before trace data leaves your application" [10]. Both pages describe features and make no claim about how complete the masking is. Test it with traces that contain invented personal data, and count what remains unmasked.

What the law says, and the limit of what we can tell you

We found no regulator that speaks about AI traces by name. What follows rests on OpenTelemetry's wording above and on general data protection guidance from the Information Commissioner's Office (ICO), the UK regulator. It covers UK law only, and a lawyer should confirm what applies to you.

The ICO's guidance on principle (c), data minimisation, says personal data must be "adequate, relevant and limited to what is necessary in relation to the purposes for which they are processed", and explains the last part as "you do not hold more than you need for that purpose" [11]. The page shows a notice that the guidance is under review because of the Data (Use and Access) Act [11]. The guidance makes no mention of traces. Our inference is that a trace holding a customer's name or message is personal data, so the general rule applies to it.

The purpose of an eval case is to test behaviour, and the behaviour rarely depends on the real name. Replace each real value with an invented one of the same shape: a different name, a made-up order number in the same format.

How long to keep traces

The ICO's guidance on principle (e), storage limitation, states: "You must not keep personal data for longer than you need it" [12]. It adds a duty to write the periods down: "You need a policy setting standard retention periods wherever possible" [12]. The guidance gives no number of days, and this page gives none either.

Applied to this method, again as our inference, which a lawyer should confirm: raw traces and eval cases are two stores with two purposes, so they can have two periods. Raw traces serve sampling and review, and that purpose ends once a trace has been read and labelled. Cases serve testing for as long as the agent exists, which is the reason step 5 removes personal data from them. The ICO's minimisation guidance also says people "have right to get you to delete any data that is not necessary for your purpose" [11], so keep a list of every place a trace is copied to.

Re-run the set on every change

A case is useful when it runs on every change to the instructions, the model or the tools. OpenAI's documentation states the move in one sentence: "Once you know what 'good' looks like, move from individual traces to repeatable datasets and eval runs" [2]. Braintrust's documentation describes datasets as "versioned collections of test cases" [7]. When a case is corrected, record which version of the set each result came from.

Checks on the final state are written in code. Checks that need judgment, such as whether the reply told the customer the truth, need a grader. Grading agent traces with Jev describes one way to do that, and the post never let the model grade its own work argues for keeping the grader separate from the model being graded. Repeat each case, as agent reliability across repeated runs describes.

A set built from production describes failures that already happened, and a feature that has not been released has no traces. New features need cases written from the specification before the code.

How Reveneau applies this

At Reveneau all code is written by AI, and every change must pass a large eval suite, written from the specification before the code, before release. Cases taken from production traces are the second source for that suite: the specification supplies the cases that can be foreseen, and production supplies the ones nobody foresaw. For every agent we build, we record traces with versions attached, we ask every client for a reviewer who knows the subject area, and we add each confirmed failure to the suite as a permanent case with personal data replaced.

The judged checks in the suite are graded by Jev, TypeSafe AI's decision model, and on our own suite the run is ten times faster than with our previous language-model grader. We use AI instead of hiring more engineers, so a build takes a small team and that saving goes into the client's price.

Reveneau, as a company, takes responsibility for the whole project through production and after release. Under this method, a failure found in production becomes a check that every later change must pass. Our AI development service describes how to start.