Jev vs a fine-tuned classifier or a small model

A hosted decision model is new. A classifier you train yourself is not, and for a team with labelled data it is often still the right tool. This page is the honest comparison: the cases where training your own wins outright, the cases where Jev wins, the one figure that lets you compare them, and what both have in common once they are live.

What each one is

A fine-tuned classifier is a model you train, or adapt from a base model, on your own labelled examples. You collect a few thousand tickets with their correct queues, you train, and you get a model that predicts the queue for a new ticket. It runs where you put it, it costs what your hardware costs, and it knows only what your labels taught it. A small model, in the sense this page uses, is the same idea with a general-purpose base that has been adapted rather than trained from nothing.

Jev is a hosted model that answers questions you write in words, with no training on your side. You describe the categories in the request, and the model returns a probability per category [1]. It runs on TypeSafe's servers, costs $0.042 per million input tokens, and its primary training language is English [2].

Both return probabilities over a fixed set of answers. The differences are in where the knowledge comes from, where the model runs, and what it costs to change.

When to train your own

Five conditions each favour training your own, and any one of them is enough.

You have a large labelled set for a stable task. If you have tens of thousands of labelled examples and the categories have not changed in a year, that data is an asset. A classifier trained on it will encode your exact distribution, your exact vocabulary and your exact edge cases. Jev cannot use your labels except as examples in the state, and the state is capped at 32k tokens [2].

The data may not leave your network. Jev is a hosted API, and every request sends the state to TypeSafe. If the text is regulated, contractually confined, or simply sensitive, the question is settled before you look at accuracy. This applies to any hosted model, and the compliance hubs on this site, starting with financial software compliance, cover what regulated data can and cannot be sent where.

The system must work offline or at the edge. A device in the field, a plant floor, a vehicle. A hosted model is unreachable there, and a small classifier that fits on the device is the only option.

The text is mostly outside English. TypeSafe names English as the primary training language for jev-1.13.0 [2]. A classifier trained on your own French or Japanese tickets knows those languages exactly as well as your labels do. Jev's accuracy on them is something you would have to measure, and the vendor gives you no figure to start from.

The decision is numeric. If the output is a number to be compared with another number, a classifier or a regression model built for that is the tool, and TypeSafe's jaggedness page rules Jev out directly: it does not count reliably, cannot reliably compare two numeric values, and reads dates as text [3]. Where Jev is weak goes through each.

When Jev wins

Four conditions favour Jev, and they are the mirror of the list above.

The categories change often. A trained classifier has to be retrained every time a category is added, renamed or split, and retraining needs new labels. With Jev, a new category is a new option in the request, described in words, live the same afternoon. For a product whose taxonomy is still moving, that is the deciding difference.

Labelled data is scarce. A new product, a new queue, a new kind of check. You have fifty examples, and fifty examples train a poor classifier. Jev needs none, because the category is described rather than learned, and fifty examples are enough to check its calibration and set a threshold.

You need a new question today. An eval check for a criterion that was written this morning, a guardrail for a failure that appeared yesterday. Writing a question takes minutes. Training a classifier takes a labelling effort first.

The question is a judgment rather than a pattern. Whether a response follows an instruction, whether a diff stays within a plan, whether a trace completed the goal. These are questions about the relation between two pieces of text, and a classifier trained on surface patterns handles them badly, because the pattern is different every time. LangChain's test of Jev as an agent evaluator is this kind of question, and on five cases with 100 repetitions each it found Jev agreed with the human reference 100 percent of the time on the pass question, while calling the result observational and early [4].

The one figure that lets you compare

The jevals README from Openlayer cites JevBench at 83 to 87 percent on Banking77 and CLINC150 [5]. Both are public intent-classification benchmarks: Banking77 is customer queries labelled with one of 77 banking intents, and CLINC150 is queries across 150 intents in ten domains. They are the kind of task a fine-tuned classifier is built for, so the figure is the fair comparison point.

Three things to hold alongside it. The figure is TypeSafe's own, reported in a third party's README, and TypeSafe wrote the evaluation. The benchmarks are intent classification, which says something about routing and nothing about eval grading, guardrails or scope checks; Openlayer's README makes that distinction itself [5]. And a fine-tuned classifier's accuracy on the same benchmarks depends on the base model and the training, so there is no single number to put beside 83 to 87. The published literature on those benchmarks is outside this page's fact base, so we do not quote a figure for the trained side, and a team that needs the comparison should train on its own data and measure both.

What the figure does tell you is the shape of the trade. On a standard intent task with a fixed taxonomy, Jev is in the mid-80s with no training. A classifier trained on that taxonomy with enough labels may beat that, and it will cost you the labelling and the retraining every time the taxonomy moves.

What both have in common once live

Whichever you choose, three obligations arrive with it, and teams that switch from one to the other are often surprised that they do not go away.

Versioning. A trained classifier changes when you retrain it. Jev changes when TypeSafe releases a version: the current model is jev-1.13.0 and the jev-latest alias moves with each release [2]. In both cases, a threshold set against the old version may be wrong against the new one. Pin the version, and treat a model change as a change to be tested.

Re-labelling. A threshold is set from labelled data, and labelled data ages. The categories drift, the language of the inputs drifts, and the cases the model gets wrong today are different from the ones it got wrong at launch. Both kinds of model need a steady supply of fresh labels, and the cheapest source is the uncertain band: the cases the model itself flagged as unsure, labelled by a person, kept for the next check. Calibrated probabilities and confidence, explained describes the re-check.

Monitoring. The distribution of probabilities on live traffic is the earliest warning that something changed. A classifier that starts returning more 0.5s is seeing inputs it was not trained on. Jev returning more answers in the middle band is telling you the same thing about its questions. Either way, the fix is to look at the flagged cases, and Jev in production covers the monitoring for the hosted case.

The hosted dependence, stated plainly

Choosing Jev means depending on a vendor for a decision inside your product. The API has documented error codes, including 429 for the rate limit and 529 for "TypeSafe is temporarily overloaded", and the docs say to use exponential backoff [6]. A path with a user waiting needs a ceiling on that backoff and a safe default when it is reached. A trained classifier on your own hardware has no such dependence and its own failure modes, which are yours to operate.

Price is part of the dependence. TypeSafe has said it cannot prove the $0.042 launch price is unsubsidised [7]. A team building on Jev should carry a line in its cost model for the price going up, and should know at what multiple the decision to use it would change. A classifier's cost is hardware and people, and it moves for different reasons.

How Reveneau decided

Reveneau grades the judgment checks in its eval suite with Jev, and the reasons match the list above. The checks are judgments about the relation between two texts, a criterion and a diff, a plan and a trace, an instruction and an output, which is the case where a pattern classifier is weakest. The criteria change with every specification, so a trained model would be retraining constantly. Labelled data for a new criterion is scarce on the day the criterion is written. And the text is English. Every condition pointed the same way. The deterministic checks stay deterministic, the uncertain band goes to a person and the label is kept, and on our own suite the run is ten times faster than with the previous grader.

If our checks had been a fixed taxonomy over sensitive text in three languages, we would have trained. The choice is the list, applied honestly. How Reveneau uses Jev has the setup, Jev vs a language model is the other comparison, and the pillar, Jev and System One models, holds both. For the wider question of adapting a model against using one as is, RAG vs fine-tuning for product teams is the general guide.