Extraction and verification with Jev

The first thing an engineer tries with a decision model is "extract the invoice total from this email", and the first thing they learn is that the response holds a choice from options you gave it, a score on a scale you defined, or a probability that a statement is true, and nothing else [1]. A value the model found on its own has nowhere to go.

So extraction becomes selection, and that turns out to be a better design than the one it replaces. The candidates come from code that you can test, the model does the one thing a regular expression cannot (understand which candidate the sentence is about), and the answer comes with a probability you can gate on.

Pre-parse in code, select with a Choice

TypeSafe's pre-parsed value extraction cookbook is the reference [2]. It uses three regular expressions, one each for email addresses, phone numbers and money amounts, to over-find every candidate in the text, then deduplicates them in document order. Those candidates become the options of a Choice question, plus a "none" option. The question is written in the language of the task: "Which email address does the sender want their receipt sent to?" The cookbook's worked examples return confidence between 0.98 and 1.00 on the email selection, 1.00 on picking the mobile number, and 1.00 on the invoice total; a follow-up Noul, "Is the amount a credit or refund?", returned probabilities of 0.01 and 0.99 on the two examples, gated at 0.5 [2]. Those are TypeSafe's own three worked examples, and the page gives no dataset size, so they show the shape rather than a measured accuracy.

The reason to over-find is that a missed candidate can never be selected. A regular expression that is too loose produces extra options the model can reject; one that is too tight produces a "none" answer for a value that was on the page. Write the parser loose and let the Choice question be strict. The 255-option cap on a Choice question applies [3], so a document with more candidates than that needs a first pass to shortlist, the same two-stage shape as intent routing.

A second benefit is that the state can be small. The question needs the sentence around each candidate, and the jaggedness page says accuracy falls as the state grows with content unrelated to the decision [4]. A parser that returns each candidate with a window of text around it gives the model what it needs and nothing else.

Dates: parts as choices, arithmetic in code

Dates are the case where the naive design fails hardest, because the jaggedness page says the model reads dates as plain text with no sense of order and is unreliable for ordering, duration or whether a date falls in a window, especially with mixed formats and relative references [4].

TypeSafe's date extraction cookbook designs around that rather than through it [5]. For an absolute date it asks seven Choice questions that each read one component: which month, which day, which year, and whether each part is present in the text. No question asks the model to compute anything. For a relative date ("next Tuesday", "the day after tomorrow") it asks a Choice for the anchor (today, tomorrow, the day after, or a named weekday) and resolves the calendar in code from a fixed reference date. An assemble step in code then validates the parts, fills a missing year with the current one and moves it forward if the result is more than 31 days in the past, rejects impossible dates such as 30 February, and flags years outside 1900 to 2050 rather than guessing [5].

The confidence of the assembled date is the lowest confidence among the parts that went into it, and the cookbook routes anything below 0.60 to review. On its six test cases across four documents, TypeSafe reports five accepted at 0.91 to 0.97 and one flagged at 0.46 for an incomplete absolute date [5]. Six cases is a demonstration, and we cite it for the design, which is the part that generalises: the model reads, the code calculates.

The same rule covers every numeric comparison. The jaggedness page says the model cannot reliably judge whether two numeric values are near each other, and that it does better on semantic representations than numeric ones (a colour name rather than a hex code) [4]. So a question like "is the quoted price within 10 percent of the list price" is two code steps and no model call. If a threshold has to be judged by the model, convert the number to a category in code first ("above the limit", "within the limit") and ask about the category.

Counting stays in code

The jaggedness page is direct: Jev "does not count reliably", whether the count is characters in a word, occurrences of a term or items in a long list, and the error grows with size [4]. Its suggested design is to ask one question per item and sum the answers in code. A Noul per line item ("this line is a shipping charge") with the total done in code is reliable; "how many shipping charges are there" is a question the model is not built to answer.

That is also the general shape for a list: iterate in code, ask a narrow question per element, and combine. Because questions run in parallel, the per-element questions can share one request up to the context limit of 64k tokens per request, with 32k for the state plus the longest question [6].

Verifying a citation against a source

Verification is the case where a decision model is at its most natural, because "is this statement true of this document" is a Noul or a Choice question with no extraction involved.

TypeSafe's citation check cookbook does it in two steps [7]. First, code: the quoted text is normalised (whitespace collapsed, curly quotes straightened) and searched as a substring of the source. A quote that is not in the source is marked fabricated without a model call. Second, for the quotes that survive, one Choice question asks how the cited section relates to the claim, with three options: supports, contradicts, or says nothing. The verdicts map to verified, contradicted and unsupported, and anything below an auto-accept confidence of 0.8 goes to a person.

The cookbook's test is eight citations against RFC 7519, a 58,365-character document in 45 numbered sections. TypeSafe reports four verified at confidence 0.93 to 0.99, one fabricated caught by the string match, one contradicted at 0.99, and two unsupported at 0.27 and 0.56, both of which fell below the 0.8 gate and were flagged for review [7]. Eight citations is a small test; the design is what matters, and the design is that code catches the mechanical failure and the model judges the semantic one.

We use this exact shape to check a language model's output against its source before it is shown to a user, and TypeSafe's coding-agents page lists it as one of the three core uses inside an application: "Check whether a statement is true of a document, message, or record before taking an action" [8].

Matching entities with a Score and three Nouls

Deciding whether two records describe the same thing is the last common extraction-adjacent task, and TypeSafe's entity alignment cookbook shows the combination of question types that fits it [9].

The data is 450 candidate pairs from two beer catalogues in the Magellan benchmark collection, each with a name, a brewery, a style and an alcohol percentage. One Score question with three levels gives the overall verdict: 0 for two different products, 1 for closely related products that may or may not be the same, 2 for one and the same product. Three Noul questions ask about the fields the model can read: same name, same brewery, same style. The alcohol percentage is compared in code, because that is a numeric comparison [9].

The decision rule is the nearest level: below 0.5 leave the pair unlinked, above 1.5 merge, and between them send to a curator. On the 450 pairs with jev-1.12, TypeSafe reports 40 merged (8.9 percent), 50 sent to the curator queue (11.1 percent) and 360 left unlinked (80.0 percent), with most scores clustering near 0.25 and 47 pairs within 0.1 of the 0.5 threshold [9]. The example the cookbook highlights is a pair scoring 1.10 at confidence 0.77: same brewery and same alcohol percentage, but one is a variant with a different name, and it went to the curator. That is the middle band doing what confidence-gated routing describes.

The per-field Nouls are what make the verdict explainable. When a merge is wrong, the log shows which field the model thought matched, and the fix is a field question rather than a mystery.

The rules, collected

Over-find candidates in code and let the Choice be strict. Always include a none option. Ask for date parts, never date arithmetic. Compare numbers in code, or convert them to categories first. Count by asking per item and summing. Verify with a string match first and one Choice second. Match with one Score for the verdict and one Noul per readable field. Gate every result on confidence and send the uncertain band to a person, whose label you keep.

Where Reveneau uses this

Reveneau builds these steps into the agents and pipelines it delivers, and grades its own eval suite the same way: whether a diff stays in scope, whether generated text follows an instruction and whether an agent's trace followed the plan are each a Noul or a Score question with the rubric written into the criteria, while every check with a deterministic answer stays deterministic and never reaches the model. Evals with Jev covers that grader. For the agent-level view of verification, how to evaluate an AI agent before you trust it is the page in our agents guide, and Jev and System One models has the full definition of each question type.