Turning a rubric into Score questions

Most rubrics start life as a table with numbers down the side: 1 is poor, 5 is excellent, and a paragraph of adjectives in between. That table was written for a person, and a person fills in the meaning of "poor" from experience. A decision model has no experience to fill it in from. It has the words you wrote, read exactly as written, which TypeSafe's own jaggedness page states as the first weakness of the model: it answers the question you wrote, not the one you meant [1]. Rewriting a rubric as a Score question is the work of making the words carry the whole meaning. Reveneau grades its eval suite with Jev, and every rubric in it went through the process on this page.

What a Score returns

TypeSafe's Score primitive takes a spectrum of 2 to 10 levels, each described in words, and returns a probability for each level, a combined score, and a confidence number. The score is each level number multiplied by its probability, added up. The docs' own example returns a score of 1.43 with confidence 0.35 and probabilities of 0.0 on level 0, 0.57 on level 1, and 0.43 on level 2 [2]. Read that as: the model thinks the state is most likely at level 1, with a real chance it is at level 2, and it is not confident which.

Two things follow. The score is continuous even though the levels are discrete, so 1.43 is a real answer and a pass line at 1.5 is a meaningful threshold. And the confidence number, which for Choice and Score is computed from how spread out the probabilities are [3], tells you whether the score is a decision or an admission that the model cannot tell. A score of 1.5 from probabilities of 0.5 on level 1 and 0.5 on level 2 is that admission. The full definitions are in Jev and System One models; this page uses them.

Rule one: one observable behaviour per level

A level is a sentence that describes something a reader could point at in the state. For a code change, the state is the diff and the spec line, so the level has to describe something visible in a diff.

Take a rubric for error handling that a person would have written as "1: no error handling, 2: some error handling, 3: good error handling". "Some" and "good" are judgments, and a judgment is what the model is supposed to produce, so putting one in the level asks the model to grade against its own opinion. Rewrite each level as a behaviour:

Level 0: the new code path has no handling for a failed call; an exception from the dependency propagates to the caller unchanged.

Level 1: the failed call is caught, and the handler returns a generic error to the caller without the error code the spec names.

Level 2: the failed call is caught, the handler returns the error code the spec names, and the failure is logged with the request identifier.

Each level names things a reader could find: a catch block, an error code, a log line. The ordering is clear, and a model that reads exactly as written has something to read. This is the same discipline as writing a testable spec line, covered in writing specs an agent can verify, applied to the grader rather than the requirement.

Rule two: keep numbers out

TypeSafe lists Score levels as weak in numerical calibration, and separately says the model does not count reliably and cannot reliably judge whether two numeric values are near each other [1]. A level that says "fewer than three unhandled cases" asks the model to count unhandled cases and compare the count to three, which is two of the three listed weaknesses in one sentence.

The fix is to describe what the count would look like instead of the count. "No unhandled case remains in the new code path" is a level. "One or more unhandled cases remain" is the level below it. If you need to know how many, that is a deterministic check: a static analysis rule, a test per case, or a lint rule (an automated style and error check), all of which count correctly and cost nothing. What to check in an eval suite covers where those belong.

The same rule applies to dates, sizes, and percentages. "The migration completes within the maintenance window" is a level that asks the model to compare a duration to a window; the model reads dates as text [1]. Make the duration a deterministic measurement and grade only what has no exact answer.

Rule three: fewest levels that separate the outcomes you act on

TypeSafe allows up to 10 levels [2]. Most eval rubrics need three or four, and the test for the right number is whether your pipeline does something different at each level. If levels 2 and 3 both pass and both do nothing else, they are one level. If level 1 fails and level 0 fails and there is no difference in what happens next, they are one level.

There are two reasons to keep the count low. Adjacent levels that differ by a shade of wording split the probability between them and pull the confidence number down, so a ten-level rubric produces low-confidence scores on changes that a three-level rubric would grade cleanly. And every level is a sentence that has to be maintained when the spec changes. Fewer sentences, fewer places for the rubric to drift from the requirement.

The one case for more levels is when the score feeds a composite, where you combine several Scores in code with weights, and you want finer resolution on one dimension. That is the pattern in composite scoring for code quality, and even there five levels is usually the ceiling.

A worked rubric for a code change

Here is a Score we would write for the question of whether a change is consistent with the conventions file in a repository. The state holds the diff, the relevant section of the conventions file, and nothing else, because TypeSafe says accuracy falls as the state grows with unrelated content [1].

Level 0: the diff introduces a pattern the conventions file names as prohibited, for example a direct database call from a route handler where the file requires a repository layer.

Level 1: the diff follows the required structure but names things in a way the conventions file rules out, for example an abbreviation in a public function name where the file requires full words.

Level 2: the diff follows the required structure and the naming rules, and any deviation is in a file the conventions document exempts.

Level 3: the diff follows every rule in the section provided and also extends an existing pattern rather than adding a parallel one.

Four levels. Each names something a reader could find by comparing the diff to the conventions text. Levels 0 and 1 fail, levels 2 and 3 pass, and the difference between 2 and 3 exists because we record it as a quality signal over time even though both pass. The pass line is 1.5.

Note the phrase "in the section provided". The model has only the state, and TypeSafe names hidden context, relying on model knowledge instead of the state, as an anti-pattern [4]. If the conventions section is not in the state, the model is grading against nothing, and the words in the level should say what it is grading against.

Reading a score between levels

With the pass line at 1.5, a score of 1.43 fails and a score of 1.62 passes. Both are near the line, and both probably came from probability split across levels 1 and 2. The confidence number tells you how split. TypeSafe's guidance is to act automatically at 0.9 and above and to route to a person below 0.5 [3], and for a Score we apply that as a second gate on top of the pass line: a score above 1.5 with confidence below 0.5 does not pass automatically; it goes to a person with the score, the probabilities, and the diff. Thresholds, confidence and escalation covers the bands and where to set them by consequence.

The person's label matters more than the grade in these cases, because it is the label that tells you whether the rubric or the model is at fault. If people keep passing changes the model scored at 1.4, the levels are too strict or too vague near the line. If people keep failing changes at 1.6, the level 2 description is letting something through. Either way the fix is in the words, and calibrating Jev against your own human labels is how to find out which.

Where the rubric comes from

At Reveneau the rubric is written from the specification before the code exists, in the same pass as the deterministic checks, and that order is not negotiable. A rubric written after the code is finished describes the code, and the point of never let the model grade its own work is that a check derived from the implementation can only agree with it. The Score levels are the standard; the diff has to meet them; the grader reports how well it does.

Rewriting our rubrics as Score questions took longer than switching the API call did. Every level that said "adequate" or "mostly" had to become a sentence about something in the diff, every number had to move to a deterministic check, and several rubrics lost half their levels because the pipeline did the same thing at each. The rubrics are better for it. A level a decision model can grade is also a level two engineers will agree on, which the older guide on using a model as a judge names as the precondition for any model grading at all.

Write the level you could point at in the diff. Then the score means what it says.