Role-based access blocks half the tasks a scoped agent completes

Every piece of agent safety advice this year has pointed at the model or the prompt. A paper published in The New Stack this morning sets both aside and asks what is left.

Writing in The New Stack, Kayode Ojo publishes the results of 160 runs of a coding agent against a 20-task benchmark on a mock enterprise API. The suite covers nine domains including tickets, users, billing, subscriptions, files and admin, carries 30 scopes in total, and verifies each task by inspecting an append-only log of API calls rather than the agent's prose. Ojo tests four permission models against Claude Haiku 4.5 and Claude Sonnet 5.

The number the paper opens with

Across the 20 tasks, the median task required 2 scopes while the union of what the agent could reach under a broad grant totalled 22. Ojo calls that ratio the blast-radius multiplier and reports a value of 11.0: in a compromised run, an agent reaches an order of magnitude more surface than the task actually needs. Across standard workflows, agents used 4 to 5 percent more scopes than the reference minimum, so the exposure comes from the size of the grant, with the agents themselves staying close to what the work requires.

Four permission models, measured

Task-scoping, where the agent receives only the scopes a given task needs, held one-fifteenth of the credentials compared to the broad grant and finished every task. Haiku hit one permission denial under this setup and Sonnet hit six across four tasks, all of which still verified.

Role-based scoping, where credentials are grouped by a job description, performed worst on both axes: it held 8.5 times more credentials than task-scoping while completing half as many tasks. The role grant missed nine essential scopes, including billing:refund and files:upload. Expanding the role until every task passed required 26 scopes, against a median task requirement of 2.

Just-in-time escalation, where the agent requests permissions as needed with a rationale recorded, reached the same low blast radius as task-scoping. Ojo reports it cost "~80%" more API calls to get there.

The passage worth quoting

Ojo writes that model capability cannot serve as a primary security control, because a team does not own the model lifecycle: "The day an operations team routes traffic to a cheaper model to cut inference spend, your safety guarantees disappear without triggering an error in your test suite." The same holds for a system prompt, which drifts across experiments and cost-cutting edits. A scope that was never granted holds whichever model runs behind it and whichever prompt is in play.

Reproducing it on your own agent

The paper closes with a procedure a team can run on one week of its agent's production traffic: measure the median number of scopes used per run and the union of scopes used across all runs, then divide. A ratio near 1.0 says a static credential is sufficient. A ratio near 11.0, Ojo writes, says each run carries an order of magnitude more permission surface than the work actually needs.

The test stack uses only Python's standard library with no external dependencies or API keys, so a reader can reproduce the method on their own API. For a team shipping an agent today, the useful output is the ratio and the audit log: a denied call is a signal that the active grant does not match the workflow, which the paper argues is either a scoping error or an injection attempt.

Source

The New Stack, "The permission you denied today is your safety net for tomorrow", by Kayode Ojo