Anthropic calibrated graders to test if Claude Duet prioritises the right task

An agent that works in the background has to pick the right task first. Anthropic calibrated internal graders to measure whether Claude Duet does that. The New Stack's Matthew Burns reported this on 29 October, with direct statements from an Anthropic spokesperson. The company called urgency "an inherently subjective call" and the eval "imperfect but quite helpful".

Claude Duet runs on Claude Opus 5.5. It can prepare work and flag items without being asked. Anthropic has been testing Duet on its own staff, using their day-to-day work to see how it handles competing priorities. The eval work came from that testing.

What the eval actually grades

The Anthropic spokesperson told The New Stack that the team calibrated graders after using Duet themselves. Those graders assess correctness in task prioritisation. They run offline against cases that represent internal use and early external testing. Each grader checks whether Duet can tell what deserves attention from what can wait.

Duet can spot an approaching deadline or an email that needs a reply. The company admits that it does not always know which task should go first. It can misread a conversation and misjudge urgency. The company's public materials do not say how often that happens, Burns writes.

A separate check runs on each action

Anthropic keeps the part of Duet that does the work separate from the part that checks the result. The spokesperson said a separate automated check runs alongside each action. The check is built on the auto mode classifier from Claude Code. Anthropic extended it for Duet. The check assesses whether an action matches the user's intent.

The check runs independently of the model doing the work. The company says that means instructions hidden on a webpage cannot steer the classifier. An outside firm penetration-tested Duet. The public materials do not show how reliably the check catches actions outside the user's intent, Burns adds.

The limits Anthropic will not shift

Some actions stay off-limits even with user approval. Trading securities. Moving money between bank accounts. Accessing someone else's account. For a purchase, Duet pays with a one-time card scoped to that merchant and amount. The user sees the item and total in a wallet app before approving.

Duet uses the apps already connected to a user's Claude account. Disconnecting an app stops new access. Information already in memory stays until the user resets it. Duet never sees passwords, browser cookies, or sign-in tokens, Anthropic says.

What the testing cost picture looks like

Duet uses the same allowance as regular Claude conversations. Background work counts against a subscriber's plan limits. The Anthropic representative told The New Stack: "Larger tasks, like building a financial model or a deck, will require more model turns and background work."

Routine checks across connected apps count against the five-hour session and weekly limits. Those checks happen every few hours, not continuously, partly to avoid using up a subscriber's allowance. The downside is that an urgent item could land between checks and wait for the next one. Anthropic does not quantify how much of a typical subscriber's usage goes to background work.

What a team building an agent can borrow

The method here is reusable. A team can describe a judgement call its agent makes. Then run it against a set of representative cases and record what the right answer looks like. The grader is then a number the team can watch across model changes. Anthropic admits that the grader is imperfect. The company's position is that an imperfect check beats no check.

Duet is in research preview by waitlist for US Pro and Max subscribers. Anthropic plans to keep adjusting the product as testing goes on.

Source

Urgency is an inherently subjective call, The New Stack, by Matthew Burns. Anthropic's safety and privacy write-up for Duet is at claude.com.