Tracely freezes a failed agent run as a pull-request test

Most teams that ship an AI agent still write its tests by hand. Tracely, a new MIT-licensed observability and regression-testing platform, records every production run through OpenTelemetry, then takes each failing run and freezes it as a test that will fail against the code that produced the failure and must pass once the fix lands. The repository is at 928 stars seven days after it was opened on 11 October 2026.

What the platform does

Tracely accepts OTLP traces over HTTP from any language, with a Python SDK that auto-instruments provider calls so an existing agent needs almost no span code added. The author, Julien Wuthrich, promotes agent-specific fields such as agent.id, conversation.id, turn and step to indexed columns in ClickHouse, and stores the raw payloads in S3-compatible blob storage so the full input to each tool call and model call is kept with the span. Evaluators run online as traces arrive, graded by either a language-model judge at the conversation, run or span level, or by a deterministic rule such as a regular expression that fails if an API key appears in an answer.

From a failing trace to a test

When a run fails, Tracely clusters it with other failures that share a structural or semantic shape, so thirty-one broken production runs collapse into one issue with the occurrence count beside it. A failing run can be promoted into a hermetic test case that bundles the exact tool outputs and model answers from the live trace as recorded fixtures. The platform calls these fail-to-pass contracts: the case must fail against the code that produced the trace, and must pass again before the fix is merged.

tracely gate is the command the CI runs, and the project states that it replays each case against the stored fixtures with no API key and no model call. GitHub Actions is the first integration: the gate posts a commit status and keeps a comment on the pull request current, exiting non-zero when any case fails. Three modes are shipped: Tracely drives the agent endpoint under test, Tracely gates traces the team's own CI already emits, or Tracely replays the fixtures entirely offline against a code change.

How to adopt it

Tracely self-hosts through a one-click Railway deployment that stands up the API, worker, UI and the data stores (ClickHouse for traces and scores, Postgres with pgvector for the registry, Redis and MinIO for queues and blobs). A local Docker Compose variant exists for development. The current provider list is OpenAI, Anthropic, Google and Mistral, with LangGraph and LiteLLM both supported directly and anything else reachable through OTLP. The project also publishes an MCP server endpoint so a coding agent can read traces and write new evaluators in the same workspace. The repository is open under MIT, with 139 forks and 19 open issues at the time of writing.

For a team already running an agent in production, Tracely is a short path from an incident to a test that catches the same incident the next time. The hand-written eval dataset problem does not go away, but it stops being the first line of defence: production itself writes the hard cases, and the replay stays cheap because the fixtures are stored, not regenerated. A pull request that reintroduces a known failure is blocked by a test that costs nothing to run.

Source

jwuthri-cloud/Tracely-ai on GitHub.