AI NewsProductivityAnnouncement
Stripe trained a fraud model on human review notes and lifted recall 39 percent
Stripe trained a risk model in three stages using its reviewers' own written notes. The post reports 39 percent higher recall at human-review precision on fraud.

Why it mattersRisk models often ignore the reasons written by human reviewers. Stripe shows those notes can lift decisions and also give an auditable reason for each one.
A risk model is told to say yes or no. Most of what a human reviewer writes about the case, the reason behind the decision, is thrown away. Stripe's engineering blog on 19 November describes a three-stage training method that keeps those reasons and uses them.
The post is by Hangzhi Guo at Stripe. It covers two risk tasks the team studied: fraud review and credit decisions. The method trains a small model in three steps. First, it picks a label and writes a short reason. Then it is sharpened on labels alone. Then the reason is brought back without losing decision quality.
The three stages
Stage one is knowledge distillation. The small model learns from a teacher model that writes a reason for each case. The teacher uses private information the student will not see at test time. Rejection sampling keeps only the reasons that match the label.
Stage two is label-only training. The same model is fine-tuned on labels alone. The reason is dropped. The post calls this decision calibration: it makes the model better at the choice.
Stage three is on-policy distillation. The stage-one checkpoint is frozen and used as a teacher. The stage-two model learns to copy its reasons again, while keeping its stage-two decisions.
What Stripe reports
On fraud review, recall at human-review precision was 0.471 after stage one. It was 0.539 after stage two, and 0.547 after stage three. The post states the stage-two number is 39 percent higher than label-only supervised fine-tuning.
On the reasons, a rubric score went from 0.191 after stage two to 0.835 after stage three. A separate review-alignment score went from 10.92 to 41.13 over the same change. The reason, in other words, came back strongly.
The post reports 15,000 stage-one examples for credit decisions and 25,000 for fraud decisions. Credit decision PR-AUC improved 9.8 percent at that size. Fraud decision PR-AUC improved 3.2 percent.
What to take from it
The work is framed as research. The post does not say the method is live in Stripe's production risk stack today. The reader can still copy the shape of it on their own data.
Three parts are worth keeping. The teacher writes reasons from private information a production model cannot use. The student is trained in order: reason, then label, then reason again. The reason is scored separately from the decision, by a rubric and by a check against human reviewer notes.
A risk team that hand-writes review notes already has the training data for stage one. The reason is not only a model lift. It is a line a reviewer can read when the model says yes or no to a case.
Source
Learning from explanations, by Hangzhi Guo at Stripe Engineering.
This item was written by an AI system from the linked source. Reveneau is responsible for what it publishes.
Get AI News in your inbox
New developer tools, model and agent releases, and how teams are actually using them to release software. Short, and only when there is something worth reading.


