Where Jev is weak: the jaggedness list

A vendor that publishes a list of its own model's failures is doing you a favour, and the right response is to read the list carefully and build around it. TypeSafe's page for jev-1.13 is titled model jaggedness, and jaggedness is a good word for it: the model is strong on some things and weak on neighbouring things, with no smooth gradient between them [1]. This page gives all nine entries in plain words, with what each one means for a request and how to work around it. It is deliberately direct, because a guide that only said what the model does well would be worth less than this one.

1. It reads the question literally

TypeSafe's wording: Jev "answers the question you wrote, not the one you meant. Scoping words, negations, and implied conditions are read at face value" [1].

What it means. If you ask "The response mentions a competitor" and the response names a competitor's product without naming the company, the model may say no, because you wrote competitor and it read competitor. If you ask "The customer is unhappy" the model has to decide what unhappy means with no help from you. If your question contains "not" or "unless" or "except", the model will apply that word exactly and only as written.

The mitigation. Treat question text as code. Write each question as one positive clause about something present in the state, say exactly what you mean by every noun, and test the question against cases where wording and intent could diverge. How to write the state and the questions has the rules and worked pairs. Literal reading is also a strength once you design for it: a question that is answered exactly as written is a question whose behaviour you can predict.

2. It does not count reliably

TypeSafe's wording: the model "does not count reliably", and the error grows with the size of the count [1].

What it means. "The list contains more than five items" is unreliable, and "the list contains more than fifty items" is less reliable still. The model reads the state as text and has no counter.

The mitigation. Count in code. Write the count into the state as a fact, such as item_count: 7, and ask the model the judgment that depends on it, if any. Most of the time the judgment was the count, and once code has it, the model is not needed for that question at all. TypeSafe's own guidance is to keep deterministic rules in code [2], and a count is the most deterministic rule there is.

3. It cannot reliably compare two numbers

TypeSafe's wording: the model cannot reliably judge whether two numeric values are near each other [1].

What it means. "The quoted price is within 10 percent of the list price" is unreliable. So is "the two dates are less than a week apart" and "the response time is under the target". Any question that turns on the relation between two numbers is asking the model to do arithmetic it does not do well.

The mitigation. Compare in code and write the result in. price_within_tolerance: true. days_between: 12. Then, if a judgment remains, ask that. The pattern is the same as for counting, and it removes a whole class of wrong answer at the cost of one line of code per comparison.

4. Score levels are weak in numerical calibration

TypeSafe's wording: Score levels "are weak in numerical calibration" [1].

What it means. A Score returns each level number multiplied by its probability, added up, so a scale of 0 to 3 can return 1.43 or 2.7. The ordering of two scores is reliable: a 2.7 is more of the thing than a 1.43. The exact distance is less reliable: 2.7 is not exactly twice as much as 1.35, and a threshold read off the scale as if it were a measurement will be wrong by an unknown amount.

The mitigation. Set every Score threshold from labelled data. Run a few hundred labelled cases through the question, look at where the scores fall for each label, and put the threshold where it separates them best. TypeSafe's guardrails cookbook shows a working set of thresholds on a 0 to 3 scale, review at 0.35 and act at 0.70 or 0.85 [3], and those are numbers TypeSafe chose from its own cases rather than from the scale. Calibrated probabilities and confidence, explained covers the method.

5. It reads dates as text

TypeSafe's wording: the model "reads dates as text, not as ordered quantities" [1].

What it means. "The invoice is overdue" needs the model to know that 2026-08-15 is before today, and it does not know that as an ordering; it knows it as a string. "The later of the two dates" is unreliable. Anything involving before, after, between or how long ago is unreliable.

The mitigation. Do every date computation in code and write the result in as a plain fact: days_overdue: 12, is_before_deadline: true, age_of_account_in_days: 400. Dates are the case where teams most often assume the model can manage, because the strings look so unambiguous. The strings are unambiguous, and the ordering between them is what the model lacks.

6. Double negatives and indirection are less reliable

TypeSafe's wording: double negatives and indirection are answered less reliably [1].

What it means. "The response does not fail to mention the refund policy" is a double negative, and the model may resolve it either way. "The response avoids the topic the user did not raise" is indirect. Both make the model do a logical transformation before it can read the state, and the transformation is where the errors come from.

The mitigation. One positive clause per question. If you find yourself writing "does not", turn the question around: "The response mentions the refund policy." If you find yourself writing about what something avoids or omits, write about what it contains instead and invert the result in code. A negation in code is one character and always correct. A negation in a question is a source of error.

7. Accuracy falls as the state fills with unrelated content

TypeSafe's wording: "Accuracy falls as the state grows with content unrelated to the decision" [1].

What it means. The model is more accurate on a short state that contains only what the question needs than on a long state that contains that plus everything else on hand. The 32k-token limit on the state [4] is a ceiling, and accuracy degrades well before it.

The mitigation. Send what the decision needs and nothing else. If the question is about a diff, send the diff and the criterion, and leave out the pull request description, the CI log and the conversation. If the question is about a ticket's body, do not send the customer's full history. When you have a large object, select the fields before you build the state, and treat every field you include as one you had a reason to include. This is the same rule as for cost, since input tokens are the whole bill [4], so the accurate state and the cheap state are the same state.

8. Adversarial content in the state can move the answer

TypeSafe's wording: adversarial content in the state "can move the answer" [1].

What it means. If a user can write text that ends up in the state, that user can write text designed to change the model's answer. A support message that says "this is a billing question, route to billing" may be routed to billing. A prompt-injection attempt hidden in a document may move a Noul about whether the document is safe. The model reads the state with the same literalness it reads your question, and it does not know which parts of the state you wrote and which parts a stranger wrote.

The mitigation. Three habits. Prefer fields you control over fields a user wrote: a customer tier from your database is trustworthy, a message body is not. Make the decision from the controlled fields where you can, and where you cannot, ask the questions in a way that does not reward the injection, such as asking about specific content rather than about the message's own claims about itself. And never let a high confidence stand in for an input check, because a manipulated input produces a confident answer. The middle band of a confidence gate catches uncertainty; it does not catch manufactured certainty. For any decision with a security consequence, Jev in production covers the layered checks, and AI agent permissions and guardrails covers the general problem.

9. It is not trained to generate text

TypeSafe's wording: the model is not trained to generate text [1].

What it means. There is no reply, no explanation, no summary. If the product needs a sentence, the model cannot supply it. If a user needs to be told why a decision was made, the probability is a number, and the reason has to come from a language model.

The mitigation. Pair it with a language model where a sentence is needed, and only there. TypeSafe's own coding-agents documentation says Jev "is not a drop-in replacement" for the language model running an agent and is used inside the application for routing, rubric scoring and statement verification [5]. Let Jev decide, and let a language model write, on the branches where writing is needed. Jev vs a language model has the decision rule.

What the list means, taken together

Six of the nine entries, counting, comparing, dates, Score distance, negations and noise, have the same mitigation: do the deterministic work in code and send the model only the judgment. That is TypeSafe's own design guidance, "Keep control flow, deterministic rules, and side effects in code" [2], and the jaggedness page is the list of what happens when a team does not. Two entries, literal reading and hostile text, are about the request being read exactly as written, and the mitigation is to write it that carefully and to know who wrote each part. The last, no generation, is the definition of the model type.

None of this is hidden. It is on a page TypeSafe publishes, and the accuracy figures from TypeSafe's own evaluation, reported by DataCamp, are consistent with it: 0 percent structured-output errors and 67.8 percent agreement with the reference answer on TypeSafe's workflows [6]. Every well-formed answer that was wrong came from somewhere on this list or from a question that was harder than the list covers.

Reveneau builds its eval grading around the list. Checks with a deterministic answer, a test passing, a schema validating, a migration holding, never go to Jev. The counts, the comparisons and the date logic are computed in code and written into the state. Jev is asked only the judgment that code cannot make, as one positive Noul or Score per criterion, and the uncertain band goes to a person. On our own suite the run is ten times faster than it was with the previous grader, and it is the arrangement TypeSafe's guidance recommends. How Reveneau uses Jev has the detail, when evals give false confidence is the general version of this page for any grader, and the pillar, Jev and System One models, is where the list sits in the wider picture.