Composite scoring for code quality

"Is this change good?" is a question a senior engineer can answer in a second and cannot explain in under a page, because the answer is a weighted sum of a dozen smaller judgments the engineer makes without noticing. A language-model judge given the broad question writes that page. A decision model given the broad question returns a probability that means whatever the model's unstated weighting happened to be. TypeSafe's own design guidance lists broad questions, with "Is this spam?" as the example, among the anti-patterns for building with a System One model, and gives the alternative in one sentence: break a broad judgment into atomic questions asked in parallel, then combine them in code with weighted arithmetic [1]. Reveneau grades its eval suite this way, and this page is the pattern applied to a code change.

Why the broad question fails

A broad question has a hidden standard. When you ask whether a change is good, the model has to decide what "good" weighs: does a missing log line matter as much as a missing error code, does naming matter as much as correctness. Those weights are not in the question, so they come from the model's training, and you cannot read them, review them, or change them. TypeSafe names the closely related failure, hidden context, as a second anti-pattern: relying on model knowledge instead of the state [1].

The atomic version makes the standard visible. "The diff returns the error code the spec names" has no hidden weighting; it is true or false of the diff. The weighting moves to code, where it is a number next to a comment, and the model's job shrinks to answering questions whose answers a reviewer could check by hand.

There is a second benefit that matters more once the suite is running. A low grade on the broad question points nowhere. A low grade on one of twelve atomic questions points at one thing. The explanation a decision model cannot write is replaced by the shape of the answers, which is most of the reason what a Jev grade cannot tell you is a manageable limit rather than a fatal one.

The split for a code change

Here is the set of atomic questions we would ask about a change that implements one feature from a spec. Each is a Noul statement or a Score rubric, written following writing Noul checks for acceptance criteria and turning a rubric into Score questions, and all of them go in one request, because every question is evaluated in parallel and adding questions barely changes the response time [2].

Correctness against the spec. One Noul per acceptance criterion, as many as the spec line has claims. State: the diff and the quoted spec line.

Scope. A Noul: "Every file changed in the diff is one the task description names or one that the named files import directly." State: the diff's file list and the task description.

Error handling. A Score with three worded levels: absent, generic, specific to the spec's error codes. State: the diff and the spec's error section.

Conventions. A Score with four levels against the section of the conventions file that applies. State: the diff and that section.

Security-relevant behaviour. Nouls for the statements that matter in this repository: "No secret value appears in a log statement added by the diff", "Every new query with user input uses a parameter rather than string concatenation". State: the diff.

Tests added. A Noul: "The diff adds or changes at least one test that exercises the behaviour named in the spec line." This is on the edge of what a decision model should grade, because "at least one" is a count; in practice we run a deterministic check for whether test files changed and use the Noul only for whether the changed test exercises the named behaviour.

Twelve to twenty questions for a typical change. Each answer comes back as a probability, and none of them is the verdict.

The combination, in code

The verdict is a function in a file. Its inputs are the probabilities; its outputs are pass, fail, or route to a person. Three things belong in it.

Gates first. Some questions are conditions rather than contributions. A security Noul below its threshold fails the change regardless of everything else, and so does any correctness Noul below its fail line. A gate is an if statement before the arithmetic, and it is where the thresholds by consequence live: the fail line on a security statement is higher than the fail line on a naming convention.

Weights next. For the questions that contribute rather than gate, a weight each, and a weighted sum normalised to the 0 to 1 range. Example weights we would start from for the set above: correctness Nouls 0.40 in total, split evenly across the criteria; error handling Score 0.20; conventions Score 0.15; scope Noul 0.15; tests Noul 0.10. Those numbers are a starting point and nothing more; they get adjusted from the calibration set, which is the subject of calibrating Jev against your own human labels.

A threshold last. The weighted sum passes above one line, fails below another, and goes to a person in between. TypeSafe's guidance for its confidence number gives the shape, act automatically at 0.9 and above and route to a human below 0.5 [3], and the same shape applies to the composite. A Score's own confidence number adds a second condition: a contributing Score with confidence below 0.5 marks the composite as uncertain even if the sum is high, because a Score whose probability was spread across levels is a guess in the sum.

TypeSafe's patterns page names this arrangement composite scoring, one of four patterns it documents, alongside confidence-gated routing, which is what the gates and the uncertain band are [4]. The guardrail cookbook shows the same structure for a different purpose, with two example policies using a review threshold of 0.35 and an action threshold of 0.70 or 0.85 [5]; the numbers differ, and the shape is the one above.

Why the arithmetic has to be diffable

This is the part teams skip, and it is the part that keeps the grader honest.

A weight is a decision about what the suite cares about. If the weights live in a dashboard or an environment variable, changing one is invisible: the suite starts passing changes it used to fail, and nobody can see when or why. If the weights live in a file in the repository, changing one is a pull request with a diff, a reviewer, and a commit message, and the change to the grader is held to the same standard as a change to the code. That is the rule we follow for every part of the pipeline, described in evals in CI for coding agents, which says nothing in the pipeline configuration may be weakened without review. Weights and thresholds are pipeline configuration.

Two practices make this work. Every weight has a comment naming the calibration run that set it, so a reviewer can ask whether that run is still current. And a change to any weight or threshold re-runs the labelled calibration set before it takes effect, so a weight that makes the grader more lenient shows up as a drop in agreement before it shows up as a bad merge. The migration checklist has this as a numbered step.

What the pattern costs

Writing twenty atomic questions takes longer than writing one broad prompt, and that is the honest cost. At Reveneau the questions are written from the spec before the code exists, in the same pass as the deterministic checks, and the question set for a feature is part of what "the spec is done" means. Most of the questions are reusable across changes to the same area, so the cost is paid at the start, once per area rather than once per change.

The run-time cost is small by construction. One request per change carries every question, and TypeSafe's batching cookbook measured thirteen questions in one call at 0.27 seconds against 2.71 seconds for thirteen separate calls, with a standard deviation of 0.0 across repeats for eleven of the thirteen questions [6]. The cost of the request is the price of its input tokens, $0.042 per million with output tokens free [7], and the arithmetic for a typical change is on the pull request page.

The cost worth watching is a different one. A composite can hide a gate that should exist. If a change scores 0.88 because eleven questions passed and one correctness statement came back at 0.2, and that statement is a contributor rather than a gate, the composite passes a change that fails a requirement. The defence is the rule above: every correctness statement and every security statement is a gate, and the weighted sum is for the questions where degree matters. Check the gate list in review, every time the question set changes.

The shape, once more

Ask small questions. Combine them in code. Gate the ones that must hold, weight the ones where degree matters, and put the numbers in a file a reviewer can diff. The model tells you what is true of the diff; the file tells you what the suite cares about; the pull request that changes the file tells you when that changed and why. That division is what TypeSafe's guidance means by keeping control flow, deterministic rules, and side effects in code [1], and it is the reason a decision model can grade a suite without anyone having to trust its opinion of what "good" means. Nobody asked it.