Open-source llm observability and evaluation
Tools for seeing what an AI application actually did: traces of every model call and tool use, evaluations that score answers, versioned prompts and datasets, and monitoring once it is live.
An open-core LLM engineering platform, MIT outside its ee folders and part of ClickHouse since January 2026, for tracing, prompt management, evaluations, datasets and a playground, with Python and TypeScript SDKs.
Comet's Apache-2.0 LLM observability and evaluation platform, one licence across the whole repository, for agent tracing, LLM-as-a-judge evaluations, prompt management, production monitoring, CI tests and an MCP server.
Arize's AI observability platform built on OpenTelemetry for tracing, evaluations, datasets, experiments and prompt management, under the Elastic License 2.0, which forbids offering it as a hosted service.
Questions people ask
What is the best open-source alternative to LangSmith?
Langfuse for the most widely used option, with an MIT core, tracing, prompt management and evaluations. Opik for a single Apache-2.0 licence across the whole repository, plus CI tests and production monitoring. Phoenix for OpenTelemetry-based tracing across Python, JavaScript, Java and Go, under the Elastic License.
Which of these can I self-host for free?
All three, with different licences. Langfuse's MIT core runs without the enterprise folders, by its own licence text. Opik is Apache-2.0 throughout. Phoenix is Elastic License 2.0, which allows self-hosting for your own use but forbids offering it to others as a hosted service.
Do they work with frameworks other than LangChain?
Yes. Langfuse lists OpenAI, LlamaIndex, Haystack, LiteLLM, the Vercel AI SDK and Mastra among its integrations; Opik's table has 64 entries including CrewAI and Spring AI; Phoenix is built on OpenTelemetry and describes itself as vendor, language and framework agnostic.
Can these evaluate answers automatically?
Yes. All three run evaluations with a model as judge, covering things like hallucination and retrieval quality, alongside code-based checks and human labelling. Opik adds a PyTest integration so evaluations can fail a build, and Langfuse links traces to a playground for iterating on a bad result.
Running one of these inside your own environment, with your own data and your own security rules, is the kind of work Reveneau does. Read how a forward deployed engagement works.