How do you decide whether an AI feature needs human approval?

Work through four questions in order: can the action be undone and by whom, what is the worst single outcome rather than the average one, who will be asked to explain the decision afterwards, and does a regulation require a person. Any single answer can require a person, so treat them as separate tests rather than a score you average.

Why is reversibility the first question to ask?

Because it is objective and it separates the cases where a mistake costs a click from the cases where it cannot be recovered at all. A sent email, a released payment, a deleted record, and a message to a customer are all actions that cannot be undone, and such an action deserves a person regardless of how accurate the model is.

What is rubber-stamping and why does it stop the check from working?

It is the state where a reviewer approves nearly everything because they have never seen the system be wrong, so the approval step becomes a delay rather than a check. The measurable symptom is an approval rate close to one hundred percent, and the fix is to move the review to the cases where disagreement is actually likely.

Is high model accuracy a reason to remove the human?

Not on its own, because accuracy describes the average case and the human is there for the worst case. Five percent error on suggested tags is a minor problem, and five percent error on account suspension is a week that could end the company, so the same accuracy number supports opposite decisions depending on what the action does.

Where should the human review go in the workflow?

At the last point where the action can still be stopped, and as close as possible to the cases where the system is likely to be wrong. A reviewer placed early sees output that later steps will change anyway, and a reviewer placed after the action has been taken is writing an incident report rather than preventing one.

What is a confidence gate and when does it work?

It routes only the uncertain cases to a person and lets the confident ones through automatically. It works when the model's confidence score is genuinely calibrated against real outcomes, and it fails quietly when it is not, because a badly calibrated model is most confident exactly where it is most wrong.

How do you give a reviewer enough context to be useful?

Show what the system decided, the inputs it used, what it was uncertain about, and what happens next if they approve. A reviewer looking at an output with no visible reasoning can only judge whether it reads plausibly, and reading plausibly is exactly what generated content is best at, so the check fails where it matters.

How do you measure whether the human step is doing anything?

Track the disagreement rate, which is how often the reviewer changes or rejects what the system produced. A rate near zero means the check is not catching anything and should be moved or removed, and a rate that suddenly rises is one of the earliest signals that model behaviour has changed.

Can you remove human review later once the system is proven?

Yes, and it should be a deliberate decision with a stated threshold rather than something that happens slowly with no decision. Decide in advance what disagreement rate over what period would justify removing the step, write it down before launch, and check whether the answers on reversibility and on how much breaks when it fails have changed as well.

What happens when the human is the slowest step?

Volume forces the choice: narrow what goes to review, add reviewers, or accept the queue. Narrowing is usually right, because sending everything to a person guarantees shallow attention on all of it, while sending the genuinely risky ten percent gives real attention to the cases that need it.

Does adding a human make the feature compliant?

No, and treating it that way is a common mistake. An oversight step only satisfies a requirement when the reviewer has the information, the authority, and the time to reach a different conclusion, so an approval button on a screen with no explanation is a record of a click rather than evidence of oversight.

How to decide if a feature needs a human in the loop: Reveneau

Q: What does the EU AI Act require about human oversight?

High-risk systems carry human oversight obligations, and the implementation timeline has shifted: general application is 2 August 2026, while the Annex III high-risk obligations now apply from 2 December 2027 and Annex I from 2 August 2028. Check the current timeline directly rather than relying on a summary, since the dates have moved once already.

The question comes up in almost every product conversation about an AI feature, and it is nearly always answered the same way: somebody says how they feel about it, and the others in the meeting agree or do not. If enough people are nervous, an approval screen gets added. If nobody is nervous, the feature is released with no approval step.

That is a mood, not a decision procedure. And the actual rule usually gets discovered later, in an incident review, at a cost.

There is a better way to do this, and it takes about twenty minutes. Four questions, asked in order, each of which can on its own require a person to approve the action, which is what "human in the loop" means. Only one of them is about how accurate the model is.

1. Can this be undone, and by whom?

Start here because it is objective. Nobody has to guess.

An action is either recoverable or it is not. A draft can be edited by anyone before it goes out. A sent email cannot be recalled. A payment that has cleared is somebody else's money now. A deleted record may or may not exist in a backup, and the honest answer to that is usually "we think so".

One-way actions get a person. That is the simplest rule on this list and the one most worth keeping absolute, because reversibility does not depend on how good the model is today or how good it might be next quarter.

The useful follow-up is the second half of the question: undone by whom. Something the user can fix themselves in two clicks is genuinely reversible. Something that requires a support ticket, an engineer, and a database change is technically reversible and practically not. Ask what the recovery actually looks like for the person affected, not for you.

2. What is the worst single case, not the average one?

This is where accuracy numbers mislead people, and it happens constantly.

A model that is right ninety-five percent of the time sounds reassuring. It tells you nothing until you ask what the other five percent does. Five percent error on suggested tags in a content tool is a minor error that nobody notices. Five percent error on "this account is suspended for fraud" is a week that ends a company. Identical accuracy, opposite decisions.

So describe the worst outcome in plain language, as a sentence about a real person. "A customer is locked out of their account and cannot reach anyone until Monday." "A wrong figure appears in a report a client sends to their board." Once it is written that way, the answer is normally obvious, which is exactly why so few teams write it down.

How much breaks when it fails is the other half of this. One wrong output affecting one user is a support ticket. One wrong rule applied to everyone at once is an incident. Anything that acts on every user at the same time deserves a person, even when each individual outcome is small.

3. Who will be asked to explain it?

Every automated decision eventually gets questioned by somebody: a customer, an auditor, a regulator, a journalist, or a board member. Somebody has to answer.

If that person cannot see what the system did and why, you have automated the action and left nobody able to account for it. That gap turns a normal mistake into a crisis, because "we do not know why it did that" is a much worse sentence than "it was wrong, here is the reasoning, here is the fix".

So the practical test is whether the system produces a record a human can read afterwards: what the inputs were, what it decided, what it was unsure about. If that record does not exist, either build it or add a person who approves each action, because that person is currently the only audit trail you have.

The related problem of where those records live and how long you keep them is covered in retention and deletion across copies, and it is worth reading before you decide what to log.

4. Does the law require a person?

Sometimes the decision is already made and the only mistake available is not knowing it.

The EU AI Act sets human oversight obligations for systems it classifies as high risk, and the timeline has moved. General application is 2 August 2026, while the Annex III high-risk obligations now apply from 2 December 2027 and Annex I from 2 August 2028. Sector rules add to that: financial services, health, employment, and credit each carry their own requirements about automated decisions affecting people.

Two things follow. Check the current timeline yourself rather than trusting a summary, because these dates have already shifted once. And do not read the delay as permission to design as though the requirement is optional, since retrofitting oversight into a system built without it is far more expensive than including it from the start. The fuller treatment of that is in when a human must stay in the loop.

The failure mode nobody plans for: the human stops looking

Here is what actually happens to approval steps six months after launch, and it is the reason this article exists.

A reviewer processes two hundred items an hour. The system has been right every time they can remember. They approve everything, quickly, because the alternative is to slow down a queue that management is watching. The approval step still exists. It stopped being a control some time ago and nobody noticed, because nothing went wrong.

This is rubber-stamping, meaning approval without real checking, and it is the default end state of any review design that sends everything to a person. The measurable symptom is an approval rate near one hundred percent.

So measure the disagreement rate. How often does the reviewer change or reject what the system produced? If the answer is almost never, you have three honest options: move the review to the subset of cases where disagreement is actually likely, remove it and say out loud that you have, or keep it and accept that it is only for show. Pretending is the only bad choice.

A sudden rise in the disagreement rate is also one of the earliest signals that model behaviour has changed, which makes the measurement worth having for reasons beyond this decision.

Where the human review should go

At the last point where the action can still be stopped, and as close as possible to the cases where the system is likely to be wrong.

Reviewers placed too early spend their attention on output that later steps will change anyway. Reviewers placed after the action has happened are writing incident reports, not preventing incidents.

Confidence gating helps when it is real. Route the uncertain cases to a person, let the confident ones through, and you get attention where it is worth something. This only works if the confidence score is calibrated against real outcomes. An uncalibrated model is often most confident exactly where it is most wrong, and a gate built on that will route the safe cases to a human and let the dangerous ones through. Check the calibration before you rely on it. The methods for that are in how to evaluate an AI agent.

Whatever you decide, give the reviewer something to work with: the inputs, the decision, the uncertainty, and what happens next if they click yes. A reviewer looking at the output alone can only judge whether it reads plausibly, and reading plausibly is the one thing generated content is reliably good at.

The part that is genuinely hard

Volume. If every case goes to a person and volume triples, you get shallow attention across all of it, which is worse than deep attention across ten percent.

The answer is nearly always to narrow rather than to scale. Decide which cases actually need judgment, send those, and automate the rest with a good record and a way to reverse. That is a harder conversation than adding a review screen to everything, and it produces a system that still works at ten times the load.

Thanks to the product teams who told us about the approval steps that had quietly stopped working. Not one of them had been removed. They had just become a button people clicked.

A human review step only works while the reviewer can still say no and mean it.

Related guide: When a human must stay in the loop.

Sources

EU AI Act implementation timeline: general application 2 August 2026, with Annex III high-risk obligations deferred to 2 December 2027 and Annex I to 2 August 2028.

How to decide if a feature needs a human in the loop