Openlayer ships jevals, a Python library that runs agent evals and guardrails in one API call using Jev

Openlayer, the AI observability company, released jevals on 20 September, a Python library that runs eval suites and runtime guardrails against agent traces in a single API request. It is MIT licensed, alpha, and has 42 stars on GitHub as of this evening. The pitch is that a team currently paying an LLM to grade every eval separately can send the same work to TypeSafe's Jev decision model, which returns calibrated probabilities for many typed questions in one forward pass.

The one number worth reading

The README ships with its own benchmark table, so treat every figure below as Openlayer's, not an independent measurement. Running Ragas against GPT-4.1-mini takes six LLM calls plus embeddings per sample, costs $2.60 per thousand samples, and finishes twenty samples in 22 to 35 seconds. Running jevals against Jev takes one request per sample, costs $0.03 per thousand, and finishes twenty samples in 0.8 seconds. Openlayer measures Jev's own median latency at 244 ms and p95 at 371 ms.

The comparison is honest about what changes: same samples, same task, different judge. The README makes a cost and latency claim, not a claim that Jev grades more accurately than GPT-4.1-mini. The request count and the token bill both drop, which is a scaling problem for any team running evals on every trace in production.

What ships in the box

Jevals carries 37 built-in evals across three namespaces: jevals.agent (12 checks such as tool-call correctness and step completion), jevals.security (13 checks such as prompt injection and PII leak), and jevals.quality (10 checks such as instruction following and answer relevance). Evals are defined in YAML rather than code, which is the library's second bet: a gate file is diffable in review and does not need a Python round-trip when the rule changes.

Adapters ship for the OpenAI Agents SDK, LangGraph and the Claude Agent SDK, so a trace produced by any of the three can be passed to jevals without a custom serialiser. An MCP server is included, which puts the same evals in an editor session on Cursor or Claude Code.

Where the trust actually sits

The design forces one useful question: how much do you trust a Jev score compared to a Claude or GPT-4.1-mini score. Jev is a classifier, not a generator, so it does not write a rationale, and the library returns a probability rather than a paragraph. The README claims 83 to 87 percent accuracy on Banking77 and CLINC150 for Jev's classification base, from TypeSafe's own JevBench. Those are intent-classification benchmarks and not eval-grading benchmarks, so they do not tell you whether Jev grades "did this answer follow the instruction" as well as an LLM does on your workload.

Two smaller Openlayer benchmarks in the README use Kev-4B and Laya on a Mac and cost $0 per thousand because they run locally, at about one second for twenty samples with Laya. Local eval is the harder version of the same bet: no vendor call means no vendor bill, and no data leaves the machine.

The library is a week old and openly labelled alpha; the version number is 0.0.x. Do not migrate a production pipeline to it this week. The reason to look at it now is to see whether the cost math holds on a workload you already understand.

Source

• openlayer-ai/jevals on GitHub