Questions to ask a vendor or development partner that uses Jev

A supplier who says "we use Jev" has told you which API they call, and nothing more. Whether the feature works, whether it will keep working after the next version, and what you own if you leave are separate questions. The eight questions below get those answers, and each comes with what a specific reply sounds like and what a vague one sounds like. We answer each for Reveneau's own use of the model at the end of its section, so you can see one concrete example and hold us to it.

Ask them in this order. The first two establish whether there is any measurement at all; if there is none, the rest do not matter.

1. Where is the labelled set?

What you are asking: the set of real cases, labelled by a person, that the feature is measured against. Where it lives, how many cases, how they were labelled, when it was last refreshed.

A specific answer: "In the repository beside the questions, 180 cases from the last quarter, labelled blind by one person with a second on 40 of them, refreshed monthly and after every version change." A vague answer: "We tested it extensively." A supplier with no labelled set has no way to tell you the agreement rate, no way to detect a version regression, and no way to hand you anything measurable at the end.

Reveneau's answer: our eval suite's rubric-graded checks each carry a set of labelled cases in our criteria, and the labels from the uncertain band, where a person decided, are kept and re-run against the grader. We publish no count of them.

2. What is the agreement rate on it?

What you are asking: at the threshold the feature uses, on how many cases, against what human agreement rate.

A specific answer gives all three: "91 percent at a 0.8 threshold on 180 cases, where two labellers agreed with each other on 94 percent." A vague answer gives one number with nothing beside it. A rate above the human agreement rate should make you ask how the labels were made. And a rate with no threshold beside it hides the escalation rate, because sending more cases to people raises agreement without improving the model.

Reveneau's answer: we do not publish an agreement rate, because we have not measured one across a set we could show the working for. The one figure we publish about our grader is that the suite runs ten times faster with Jev grading it than with a language model, on our own suite, and that figure says nothing about agreement.

3. What runs when confidence is low?

What you are asking: the path for a case below the action threshold. TypeSafe's own guidance is 0.9 and above to act automatically and below 0.5 to route to a person or another system, with different thresholds for actions of different consequence [1].

A specific answer names the threshold per action and the destination: "Below 0.8 on a route, the ticket goes to the general queue and a person picks; below 0.95 on a spam close, it is never closed and goes to review; the person's label is stored." A vague answer: "It falls back gracefully." Ask what happens to the label the person gives, because if it is thrown away the labelled set never grows on the hard cases.

Reveneau's answer: a grade in the uncertain band goes to a person, and that person's label is kept to re-check the grader later. A grade below the threshold fails the change. Deterministic checks, a test passing or a schema validating, never go to the grader at all.

4. What happens when the API is down?

What you are asking: the sequence for each error the API documents: 401 invalid key, 422 validation, 429 rate limit, 529 temporarily overloaded, which TypeSafe's docs say to handle with exponential backoff [2]. And whether that sequence has run in a test.

A specific answer: "Three retries with backoff, then the same questions to a language model, then a default route; we cut the API in a test on every change and the test fails if the fallback takes more than four seconds." A vague answer: "The API has been reliable." A fallback that has never executed cannot be relied on, and a supplier who has not cut the API on purpose does not know what their feature does when it happens on its own.

Reveneau's answer: a check that cannot reach the grader fails closed, meaning the change does not pass, rather than passing by default. The suite reports the failure as an infrastructure failure rather than a grade, so nobody reads an outage as a clean run.

5. Which model version is pinned?

What you are asking: the version string in the request. TypeSafe lists jev-1.13.0 as the current version, with the aliases jev-latest and jev-preview both pointing to it [3].

A specific answer is a string: "jev-1.13.0, and a change requires a re-run of the labelled set and a recorded comparison." A vague answer: "We use the latest." An alias moves on the vendor's schedule, which means the feature's behaviour can change with no change in the supplier's code and no notice to you. Early-access risk and vendor lock-in has the contract clause that backs the pin.

Reveneau's answer: the grader is pinned by version string in the suite configuration, and a move to a new version is a change like any other, which means it runs the labelled cases and is compared before it is adopted.

6. What is logged per decision?

What you are asking: the fields in the record for every decision, escalated or not. The complete record is the state, the questions, every probability, the confidence number where one exists, the threshold, the action, the model version, the timestamp, and the human label if escalated. There is no written reason, because the model does not produce one [4]; auditability when the model gives no written reason covers what that means for you.

A specific answer lists the fields and where you can read them. A vague answer: "We log everything." Ask to see one record for one real decision. If it takes more than a minute to produce, the logging is not what was described.

Reveneau's answer: each graded check records the criterion, the state sent, the probabilities returned, the threshold, the verdict and the grader version in the suite's output, so a failed change can be traced to the grade that failed it.

7. Who owns the questions and thresholds?

What you are asking: whether the questions in their exact words, the threshold per action, and the labelled set live in your repository and are yours at the end of the engagement, or live in the supplier's system and leave with them.

A specific answer: "They are three files in your repository; here are the paths." A vague answer: "It is all in our platform." The questions and thresholds are the feature. The code around them can be rewritten in a week if you have them, and cannot be rewritten at all if you do not. For a partner, this is a contract clause; what changes in a contract when AI writes the code has the wider set.

Reveneau's answer: when we build a decision-model feature for a client, the questions, the thresholds and the labelled set sit in the client's repository beside the eval suite, and the suite is the client's too. We take responsibility for the project through production and after release, and that responsibility includes leaving those artefacts in a state someone else could run.

8. What did the last version change break?

What you are asking: evidence that the supplier has been through a version change and measured it. At writing there has been one version, jev-1.13.0, released on 15 September 2026 [3], so the honest answer today is "there has not been one yet, and here is what we will do when there is."

A specific answer describes the procedure and the comparison record it will produce. A vague answer: "Nothing has ever broken." TypeSafe's jaggedness page for the current version lists literal reading of negations, unreliable counting, unreliable numeric comparison, dates read as text, and sensitivity to adversarial content in the state [4]. A supplier who knows that list can tell you which of their questions is closest to a documented weakness, and that is the question most likely to move on a version change.

Reveneau's answer: there has not been one yet. When there is, the new version runs against the kept human labels, the agreement is compared against the current version's, and the comparison is recorded before the move. We will not publish a result from that until we have one.

Reading the answers as a buyer

Count the specific answers. Eight is a supplier who has built the thing and measured it. Five to seven is a supplier who has built it and has gaps you can name and put in the contract. Fewer than five, and the feature is a demonstration, and you should pay for it as one: a pilot at pilot prices, using the two-week pilot, with the labelled set as the first deliverable.

For an investor, these eight go into the management meeting beside the questions on the model dependency risk page and the questions for the management meeting list in the AI startup diligence guide. The technical due diligence questionnaire covers the codebase around the feature.

For a buyer of a product that embeds the model, the same eight apply, with the difference that the answers to 5 and 7 are the vendor's policy rather than your files. Ask whether the vendor pins the version on your behalf and what notice you get.

The pillar guide has the sequence this page sits in, and the questions to ask before approving an AI build is the general list for any AI feature. Print this page, write the eight answers next to the eight questions, and count.