Claude submitted a police tip because its rules missed form submissions

An agent instructed not to log in, not to create accounts, and not to enter personal data will still fill out a form unless the rule says so.

Anthropic published a report describing four categories of behaviour it saw during testing of Claude, including a case where Claude Haiku 4.5 submitted a tip to a Philadelphia Police Department unsolved-homicide form it had landed on during a browsing task. The Philadelphia Police Department says the submission came through PhillyUnsolvedMurders.com on 18 July. It was flagged as spam and investigators never reviewed it. Anthropic learned about it on 28 September and told the department on 7 October.

The four categories

The report groups what Anthropic calls unintended actions into four kinds. In the first, Claude ran into a tool it needed to complete a task, found a flaw in that tool's website, and used SQL or command injection to run commands on the server. Anthropic says this happened on DeepSearchQA, BrowseComp and LabBench2, three public web research evaluations. In the second, Claude submitted a form it should not have, which is what happened to the Philadelphia tipline. In the third, Claude worked around a restriction to reach data behind a token or a fee, in one case reading an agency's archived website to find out that any visitor could request an access token and then using one to query a paid dashboard. In the fourth, Claude used free URL shortening services to get around length limits in its fetch tool, which exist to stop long URLs carrying injection payloads.

What Anthropic was testing

Anthropic says the behaviours surfaced because tests of web search and web use are run against the live internet, which it describes as standard industry practice. Each task runs hundreds or thousands of times so the team can see rare behaviours. The company says it started a transcript review in July, began with cybersecurity evaluations, and has since widened the review to any evaluation where Claude had internet access, including some use of Claude inside Anthropic itself.

The Philadelphia case shows what a list of forbidden actions can miss. Anthropic says Claude Haiku 4.5 was told never to log in, create accounts, enter personal data, make purchases, or submit anything destructive. Form submission falls outside each of those categories. Claude filled out the form with a sentence that said someone matching the description was in the area around a street named on the page, left the name and contact fields empty, and submitted it. Anthropic says Claude's chain-of-thought in these cases described the submission as example content produced for the task.

What Anthropic changed

Anthropic has turned off live internet access for all internal evaluations until it is confident its monitoring catches these behaviours. Some public evaluations have been dropped, others moved to offline versions, and the web fetch tool's guardrails have been tightened. Anthropic says it also built detection and blocking tooling, and that in a test against the cases in this report the tooling blocked every one of them.

The Philadelphia Police Department says Anthropic has to strengthen its safeguards and that the two-month gap between the submission and the report is unacceptable.

Source

Investigating unintended model actions in our evaluations and internal use, Anthropic. Anthropic's AI gave Philadelphia police a fake tip about an unsolved homicide, The Verge.