When to move back to a language model or to rules

A feature that was a good fit for a decision model in the pilot can stop being one, and the reasons are ordinary: the product changed, the market changed, the model changed, or the task was never the shape it looked. This page lists the five signals, says where each one points, and says how a swappable design keeps the move to a week. It ends with the statement the whole guide rests on: a decision model is one tool among rules, language models and people, and moving between them is normal.

Signal 1: the option set keeps changing

A Choice question offers up to 255 options, and the options are written in advance [1]. If the list of queues, categories or labels changes more than once a quarter, every change means rewriting the question, re-running the labelled set and retuning the threshold, and the labelled set itself has to be relabelled for the new options. At some rate of change the maintenance costs more than the decision saves.

Where it points: to a language model, if the options are open-ended and the task is closer to "describe this" than "pick one of these"; or to rules, if the options change because a person is maintaining a list that could be a table. Before moving, check whether the changing part can be split off: often the top-level routing is stable and only a sub-category churns, and the sub-category can go to a language model while the routing stays.

Signal 2: users need a reason

The model gives none, because it is "not trained to generate text" [2]. A feature can launch without anyone asking why and then acquire the requirement: a customer escalation process that wants the reason for a routing, an appeals flow for a moderation block, a regulator's question about a denial. Auditability when the model gives no written reason covers the hand-off for cases where a reason is required by rule; this signal is the softer version, where users simply want one.

Where it points: to a language model writing the reason from the same state, with the decision model's probabilities stored beside it, or to a person for the decisions with consequence. Most often the move is partial: the decision stays with the decision model and a language model is added to explain it on request, labelled as generated. The full move, letting the language model decide as well, is only needed when the reason and the decision have to come from the same place.

Signal 3: agreement drops after a version change and does not recover

A new model version is run against the calibration set before adoption, following measuring a decision model in production. If agreement falls, the first response is to retune the questions and thresholds against the new version's probability distribution, once. If it falls and stays fallen after that retune, the new version behaves differently on your task, and the old version may not be available for long. At writing there is a single version, jev-1.13.0 [3], so this signal is untested; the procedure exists so the first version change is measured.

Where it points: to whichever replacement scores best on the labelled set. That might be a different decision model from another vendor, a language model asked the same questions, or rules if the labelled set shows the task was simpler than it looked. The labelled set is what makes this a measurement rather than a debate.

Signal 4: the task turned out to need counting or dates

TypeSafe's jaggedness page for jev-1.13 says the model "does not count reliably", cannot reliably judge whether two numbers are near each other, and "reads dates as text, not as ordered quantities" [2]. A feature can be scoped around those limits in the pilot and then grow a requirement that hits one: the routing that now depends on how many items are in the order, the moderation that now depends on whether a date is in the past.

Where it points: to code, first. Counting, comparison and date ordering are deterministic and cheap, and TypeSafe's own guidance is to keep deterministic rules in code [4]. Extract the number or the date with whatever reads text, then count, compare or order in code, and send the decision model only the text judgment. If the extraction itself is the hard part, a language model does the extraction and the decision model still does the judgment. A full move is rarely needed for this signal.

Signal 5: English-only is a problem

English is the primary training language [3], and no accuracy figure is published for others. A product that launched in one market and opens in another can find the labelled set built for that market misses its agreement cutoff.

Where it points: to a labelled set in the new language first, because the model may do well enough and the only honest way to know is to measure. If it misses the cutoff, the fallback path for that market becomes the primary path: a language model with published multilingual performance, or a person, until a decision model with a published figure for that language exists. Keep the English market where it is. Route by language in code.

How a swappable design makes the move cheap

The three-part design from early-access risk and vendor lock-in is what turns each of the five signals into a week of work.

The questions are plain text in your repository, so a language model can be given them as a prompt on day one, and a rules engine can be given the same option list. The thresholds are numbers in the same repository, so the replacement's escalation logic starts from the same table and is retuned against the labelled set rather than invented. The labelled set, grown monthly from escalations, is the instrument: run the replacement against it, compare agreement per action with the current model, and move when the replacement meets the cutoff. The fallback path already exists in code and has already executed in a test, so in most cases the move is promoting the fallback to the primary and demoting the decision model to the fallback, then removing it when the numbers hold.

A week is the estimate for a feature built this way. Without the questions in the repository, the week starts by rewriting them from memory. Without the labelled set, there is no way to know when the replacement is good enough, and the move becomes a matter of opinion. Without the fallback in code, the move is a build.

Most moves are partial

Each of the five signals points at part of a feature. The changing sub-category, the explanation on request, the counting step, the second language. In each case the decision model keeps the part that still fits the four tests in which features are System One tasks, and the other tool takes the part that does not. A feature that is half decision model, a quarter rules and a quarter language model is a normal shape, and it is the shape TypeSafe's own design guidance describes when it says to keep control flow in code and to use the model for the judgment in the middle [4].

Reveneau's eval suite is that shape. Deterministic checks are code and never go to a model. Rubric-graded checks go to Jev as Noul or Score questions with the rubric in the criteria. Uncertain grades go to a person, whose label is kept. We moved the grader to Jev this month because the grader was swappable, and the suite runs ten times faster than it did with a language model grading it, on our own suite. On any of the five signals above, the same swappability moves it back, and the labelled cases from the uncertain band are what would tell us whether the move was right.

The honest statement

A decision model answers questions whose answers you already listed, at a speed and a price that make it worth checking things you could not afford to check before. That is a narrow job and it is a good one. Rules answer questions whose logic you can write out completely. Language models answer questions whose answers are open, and they write. People answer questions with consequence and give reasons. A product uses all four, and the boundaries between them move as the product learns what its tasks are.

The pillar guide has the sequence this page closes. How to choose what not to automate is the older version of the same judgment, and how to decide if a feature needs a human in the loop covers the person's share. Jev in production has the engineering side of the swap. Moving back is a measured decision, made against a labelled set, on a stated signal. It is the same discipline that made the move in.