Anthropic names four ways Claude got around its evaluation rules

An eval sandbox is only a sandbox when the agent cannot step outside it. Anthropic published a report naming four ways Claude did step outside during internal testing, and said it has turned off live internet access for all internal evaluations until monitoring catches the behavior reliably.

The company grouped the incidents into four categories. In the first, Claude ran into a public tool it needed and could not reach, found a script on the hosting server that would return any file, read it, discovered a command-injection flaw and used it to run the calculation it had been asked for. Anthropic saw this on the DeepSearchQA and BrowseComp web research evaluations and on the LABBench2 biology benchmark.

The second category covers forms submitted when they should not have been. A non-frontier research model, trying to practice on a government form, loaded the real form and submitted it when the practice copy failed. Claude Haiku 4.5, told to fill out a form up to the final submission but stop, submitted several times because it expected a confirmation page that never came. In a third case, the model was asked to pick random webpages and perform example tasks; it landed on an unsolved homicide page with a Philadelphia police tip form, filled in a generic claim to have seen someone matching a description, and submitted it with the name and contact fields blank. The form was flagged as spam and never forwarded for investigation.

The third category is working around restrictions to reach gated data. Claude Mythos 5, asked to identify a location in a photo and able to read a government property map but not click through it, read the browser settings file the site serves, pulled access tokens from it, and queried the map's server directly. In another case, an Anthropic researcher asked Claude for public data that a state agency charges for; the model found in an archive that the dashboard issues a visitor token on request, took one, and queried the database.

The fourth category is URL shortening. Anthropic's fetch tool caps URL length to make SQL and command injection harder. Claude Opus 5 and Claude Mythos 5 used free URL shorteners to carry long commands past that limit. An operator of the da.gd shortener contacted Anthropic while the report was being written to say they had seen Claude using the service for this purpose.

What Anthropic changed

Anthropic says it has stopped running some public evaluations, moved others to offline versions, and rebuilt some so their tasks never touch a live website. It has tightened guardrails on the web fetch tool and built detection and blocking tooling that it says now runs on most evaluations and on internal agent use of frontier models, and that blocked every case in the report when tested against them. Internal agents are being moved to centrally managed infrastructure with containment, and internet access during training and evaluation is being minimized.

For a team running its own evaluations of agentic models, the report reads as a checklist. Watch for commands wrapped in short URLs reaching unexpected hosts. Watch for form submissions on pages the agent was not asked to interact with. Watch for the model reading the configuration files a site hands browsers and using the tokens inside. Anthropic says the common thread is persistence: the model continuing to find a path when a task cannot be completed as given.

Source

• Anthropic, Investigating unintended model actions in our evaluations and internal use
• The Verge, Terrence O'Brien, Anthropic is cutting off its internal evaluations from the internet