What is the core design question for agent escalation?

The core question is: what is the cost of proceeding incorrectly versus the cost of interrupting the human? For a low-cost, reversible action, proceeding without asking is usually right. For an action that cannot be undone, or that affects systems outside the agent's scope, asking before proceeding is right. The threshold between them should be set by the person deploying the agent, not left for the agent to infer.

What kinds of actions should always require a human check before an agent proceeds?

Actions that cannot be undone. Sending an email or a message to a real person. Writing to a production database. Deleting a record. Calling an external API that charges money or triggers a downstream process. The common thread is that the cost of an incorrect action is high and the action cannot be reversed after the fact. These categories should be listed explicitly in the agent's constraints, not left to the agent's judgment.

What makes an agent ask too often?

Ambiguity in the task description. When the task does not say what to do in a specific situation, an agent that is designed to be cautious will ask. The fix is usually in the task description: specify the cases the agent will encounter and what to do in each, so the agent can proceed rather than stopping to ask. An agent that asks ten times in one run is usually an agent that received a ten-gap task description.

How do you distinguish a task the agent should decide from one it should escalate?

Ask whether the decision is inside the scope the agent was given or outside it. If the agent was asked to draft a reply to a customer email and encounters a request for a refund, is handling a refund inside the scope it was given? If the task description does not say, the agent cannot know either. Scope boundaries have to be stated explicitly. What the agent can decide, what it should escalate, and what it should decline to handle are all things that need to be written into the task before the agent runs.

What is the difference between a soft checkpoint and a hard stop?

A soft checkpoint is a point where the agent surfaces a decision to the human but can proceed if no response comes within a time limit. A hard stop is a point where the agent will not continue until a human gives explicit confirmation. Hard stops belong on irreversible actions and actions outside the agent's stated scope. Soft checkpoints work for decisions where the agent has a reasonable default but the human might want to override it. Using hard stops on every uncertain decision makes the agent too slow to be useful.

How should an agent handle a situation it was not designed for?

Stop and report rather than proceed with its best guess. When an agent encounters a situation that falls outside the cases the task description covers, the right behavior is to describe the situation and ask what to do next, rather than to infer from the surrounding context and act. An agent that infers and acts in a novel situation can produce a confident wrong answer, and the human may not realize the agent was operating outside its design.

How do you test whether an agent's escalation threshold is set correctly?

Run the agent on a set of tasks that includes both clear cases and boundary cases. For clear cases, the agent should proceed without asking. For boundary cases, it should ask. If the agent asks on clear cases, the threshold is too low and the task description is probably too ambiguous. If the agent proceeds on boundary cases without asking, the threshold is too high and the agent is making decisions it should not be making. Adjust the task description and the checkpoint conditions based on what you observe.

Can an agent learn when to escalate from examples?

An agent can follow escalation examples if they are included in the prompt, but following examples is not the same as learning a general principle. If you include three examples of when to ask and the agent encounters a fourth case that is similar but not identical, it may or may not apply the pattern. For safety-critical escalation decisions, explicit rules are more reliable than examples. State the rule directly: "If the action modifies data outside the scope of this task, stop and report."

What information should an agent provide when it stops to ask?

What it was doing, what specific thing it is uncertain about, and what the options are. An escalation that says only "I am not sure what to do" is not useful. An escalation that says "I need to send a reply to this customer but the email mentions a billing dispute. The task description covers support questions but not billing. Should I reply as if this is a support question, escalate to billing, or decline to reply?" gives the human what they need to respond quickly. The agent should do the work of framing the question before escalating.

How does escalation design change when an agent runs unattended overnight?

The cost of a hard stop increases when the agent runs without a human available to respond. For unattended runs, the agent either needs a large enough set of explicit rules to handle every case it will encounter, or it needs a fallback behavior for uncertain situations that is safe to leave in place until a human can review it. Stopping mid-task and waiting for a human who is not available is usually not the right design. The fallback should be something the agent can do safely: finish the cases it is certain about, skip the uncertain ones, and report which ones it skipped and why.

When an agent should ask for help instead of proceeding: Reveneau

An agent that proceeds when it should stop is dangerous. An agent that stops when it should proceed is useless. The design question is finding the right threshold between them, and the answer is specific to what the agent is doing and what the cost of a wrong action is.

Most agents are deployed without a clear answer to this question. The assumption is that the agent will figure it out: proceed when it is confident, ask when it is uncertain. The problem is that confidence and safety are different things. A model can be highly confident about a wrong answer. The right threshold for escalation is not based on how confident the agent is. It is based on what happens if the agent is wrong.

Reversibility is the first filter

The clearest line between "proceed" and "ask" is reversibility. If an action can be undone, the cost of a mistake is bounded. The agent can proceed, the human can review the output, and anything wrong can be corrected. If an action cannot be undone, an error is permanent.

This is a list, not a heuristic: sending an email to a real person, writing to a production database, deleting a record, calling an external API that charges money or triggers a downstream process, posting to a service that notifies other people. These are the actions where the cost of proceeding incorrectly is high enough that asking before acting is almost always right, regardless of how confident the agent is.

The list of irreversible actions should be in the agent's task description explicitly, not inferred. If the agent is supposed to stop before sending any email, that constraint needs to be written as a rule, not expressed as an example the agent might or might not generalize from.

Scope is the second filter

An agent operates within a defined scope. The question for any potential action is whether that action is inside the scope or outside it.

A customer support agent is scoped to answering questions about a product. If a customer's message contains a request for a refund, is handling a refund inside the scope? The agent cannot know unless the task description says. If the task description is silent, the agent will either proceed with its best guess or stop and ask, and both outcomes are unpredictable.

Write the scope boundary explicitly. What the agent can handle on its own, what it should escalate to a human, and what it should decline to handle are all things that need to be in the task before the agent runs. An agent that encounters a situation outside its stated scope should stop and report what it encountered, not infer from surrounding context and proceed.

Soft checkpoints versus hard stops

Not every uncertain situation needs a hard stop. A hard stop is a point where the agent will not continue until a human gives explicit confirmation. That is the right design for irreversible actions and actions outside scope.

For decisions where the agent has a reasonable default but a human might want to override it, a soft checkpoint works better. The agent proposes the action, notes that it is proceeding unless it hears otherwise, and waits a short time before continuing. For synchronous tasks where a human is watching, this is often enough. The human can intervene when they want to without the agent stopping on every decision.

Using hard stops on every uncertain decision makes the agent too slow to be useful. The goal is not maximum caution. It is an appropriate level of caution for each category of action. High-cost irreversible actions get hard stops. Low-cost reversible decisions get soft checkpoints or no checkpoint at all.

What the agent should say when it stops

An escalation that says only "I am not sure what to do" is not useful. By the time the agent escalates, it has already processed the situation. The human gets the most value when the agent also does the work of framing the question.

A useful escalation tells the human three things. What the agent was doing when it encountered the question. What specifically it is uncertain about. What the options are, with a default recommendation if the agent has one.

Compare two escalations for the same situation:

Weak: "I encountered a billing question and I am not sure how to proceed."

Useful: "This customer email asks about a billing dispute on their invoice from last month. My task covers product support questions. I can either reply as if this is a product question and try to help, escalate it to your billing team by flagging the email, or reply to let the customer know this needs to go to billing. I would recommend flagging it for billing, but I need your confirmation before I take any action on it."

The second version gives the human what they need to respond in one message. That is the design target for an escalation.

Unattended agents need a different default

When an agent runs overnight without a human available to respond to escalations, a hard stop mid-task is often the wrong design. The agent stops, the task does not finish, and nobody sees the escalation until the next morning.

For unattended runs, the agent needs either a complete enough set of rules to handle every situation it will encounter, or a safe fallback for uncertain situations. The safe fallback is usually: finish the cases the agent is certain about, skip the uncertain ones, and produce a report of which cases were skipped and why. The human reviews the report when available and handles the skipped cases.

Designing for unattended runs forces the design question that matters: what are all the situations this agent will encounter, and what should it do in each? The answer is the task description. An agent that runs well unattended is an agent whose task description accounts for its full operating range.

Testing the threshold

The right way to know whether the escalation threshold is set correctly is to run the agent on a set of tasks that includes clear cases and boundary cases. For clear cases, the agent should proceed without asking. For boundary cases, it should stop and ask.

If the agent asks on clear cases, the task description is probably too ambiguous. If the agent proceeds on boundary cases, the threshold is too high. Adjust the task description and the checkpoint conditions based on what you observe.

This is a design process, not a one-time calibration. As the agent encounters new situations, some of them will be cases the original design did not account for. Each one is an input to the next version of the task description.

When an agent should ask for help instead of proceeding