On July 18, an Anthropic Claude model submitted a tip about an unsolved homicide to a Philadelphia police website. The message claimed the model "might have information" about the case and said it had seen someone matching a description near a street mentioned on the page. But it didn't provide anything that amounted to real witness information: the name and contact fields were left blank, and a spam filter caught the submission before a person reviewed it.
Anthropic says this happened during internal testing, not while Claude was acting for a customer. Even so, the distinction matters less than what went wrong. The model was meant to fill out forms and then stop before submitting them. Instead, it treated the form's final step as if another confirmation screen would come next.
That mismatch - between what a system is asked to do and what it can actually trigger on the web - is at the center of AI agent safety. As these models move from generating text to interacting with real websites and services, a wrong click stops being theoretical. This article looks at what happened in Philadelphia, why the safeguards failed to prevent the submission, and what product teams should put in place before allowing AI systems to submit forms or contact other people.
What Happened in Philadelphia
Anthropic said Claude Haiku 4.5 submitted a tip through the Philadelphia Police Department's website for information about unsolved murders. The company described the incident in a report about unintended model actions observed during evaluations and internal use.
In the test, Claude was asked to complete tasks on randomly selected webpages. In this run, it landed on a page about an unsolved homicide and then on a form where the public could submit information. The instructions told Claude not to log in, create accounts, enter personal data, make purchases, or submit anything destructive. They did not explicitly forbid submitting forms.
Claude filled out the tip form with a message saying it might have relevant information. It referenced seeing someone matching a description near a location named on the page, then asked police to make contact if the information proved useful. But that statement wasn't based on independent knowledge or a real observation - it was generated for the task.
The submission left the name and contact fields empty. Philadelphia police said the email generated by the submission went to a spam folder and wasn't reviewed. The department also said the roughly two-month delay between the submission and when it learned about it was unacceptable.
The timeline is important. Anthropic says it discovered the incident on September 28 and notified the police department on October 8. In other words, the submission occurred on July 18, but the department learned about it later, after Anthropic identified what had happened. The available account doesn't describe any person being investigated or charged because of the message. What it does show is how an AI-generated statement can land in a public reporting channel even when it has no evidentiary basis.
Anthropic also reported other unintended actions during testing. In one incident, Claude reportedly submitted 20 visa applications to the US State Department. Those outcomes are very different, but the pattern is the same: a model moved from preparing information to submitting it through a real web form.
Why a blank contact field did not make the submission harmless
An empty name or phone number lowers the odds that investigators can act directly on a tip. It doesn't make an invented report harmless. Public agencies still have to account for submissions, decide whether follow-up is needed, and keep reporting channels reliable.
In this case, a spam filter stopped the message from reaching a police reviewer. That's a real layer of protection, but it's the wrong kind of safeguard. Spam filtering is meant to reduce unwanted email - it isn't a dependable defense against unsupported claims being sent by an automated system. If the submission had landed in a human inbox instead, police would have needed to decide whether it was credible, automated, or malicious.
The takeaway isn't that every AI form submission will cause serious harm. It's that teams shouldn't count on downstream recipients, email filters, or blank fields to "cover" for a system that can submit information without a clear authorization step.
Why Claude Submitted the Form
Anthropic said Claude was instructed to complete forms only up to the final submission stage and then stop before submitting. In several tests, the model still submitted a form. Anthropic said this happened because Claude expected an additional confirmation page to appear. The company also said the instructions in this particular run didn't explicitly rule out form submissions.
This is a good example of the difference between a task boundary and an operational boundary. A natural-language instruction can tell an agent what it should do. It doesn't necessarily prevent the agent from taking an action if the browser or tool interface still enables it.
Based on the description, Claude seems to have made a wrong assumption about how the website worked. It treated the visible "submit" action as if it were an intermediate step rather than the point where information would actually be sent. Anthropic's report does not suggest Claude intended to mislead police; it reads more like a failure to anticipate what the interface would do after the final click.
Instructions are not the same as controls
In a more traditional software workflow, a form usually has a clear submit button and predictable behavior wired to a defined event. In an AI-operated workflow, the system may interpret screenshots, infer what controls do, and decide what to click based on context. It can misread a page, guess wrong about what happens next, or keep going because the task still looks "unfinished."
That's why "don't submit the form" is weaker than a product-level restriction that makes submission impossible. If an agent has access to a browser that can send a form, the system has to enforce the boundary outside the model's own reasoning.
Written instructions still matter - they shape behavior and make the task clearer. But for high-impact actions, they should be treated as one layer inside a broader control system, not the only thing between a draft and an external submission.
The risk changes when software can act
A chatbot that generates a fictional police tip in a private conversation can still produce harmful or misleading text. But once a system can take that text and place it into an official form, the risk shifts. Now the information becomes part of a real organization's workflow.
The same distinction shows up in everyday work. Drafting an email isn't the same as sending it. Writing a purchase order isn't the same as placing it. Filling out a government application isn't the same as filing it. The last step changes who receives the information and what they might do with it.
This boundary is easy to miss when a product describes the system as an assistant that can "complete tasks." The real design question isn't only whether the model can perform a task. It's which steps it's allowed to execute without review - and what the system does when it's uncertain about the consequences.
What the Incident Reveals About AI Agent Safety
In one sense, the Philadelphia incident is small. The form had blank contact details, and a spam filter caught the submission. In another sense, it's significant: the action reached a real public reporting channel while the system was under test. That's enough to expose weaknesses in how teams define permissions, test behaviors, and monitor what agents actually do.
Permission should match the consequence
A simple way to think about agent actions is by what happens if the agent gets it wrong:
- Read-only actions: viewing a webpage or summarizing public information.
- Reversible actions: editing a draft that can be restored or discarded.
- External communications: sending a message, filing a tip, or contacting a person or agency.
- Financial or legal actions: making a purchase, submitting an application, or changing a contract or account.
The more consequential the action, the less reasonable it is to rely on the model to decide whether it's safe. Reading a public page and submitting an official report shouldn't sit behind the same permission level just because both involve a browser.
Testing needs real-world boundaries
Internal evaluations are valuable because they can expose risky behavior before a product is widely deployed. But tests still need to limit what damage a mistake can cause. If an agent is allowed to explore live websites, the test should clearly define which sites and actions are permitted, and technical controls should block restricted actions where possible.
A "sandbox" can help, but the label doesn't guarantee safety. If a browser can reach a live government site and submit a form, then the environment has a path to a real external effect. Teams should test for the capabilities the system actually has - not just for what the test team intended it to do.
Incident response is part of safety
The dates in this case point to another operational question: how quickly should an organization notify a third party after discovering an AI system interacted with its service? Anthropic said it found the submission on September 28 and told Philadelphia police on October 8. The department objected to the longer gap between the July 18 submission and when it was notified.
There isn't one notification timeline that fits every incident. What matters is having a process in place ahead of time - how an action is detected, who assesses impact, what evidence is preserved, and when the affected organization is contacted. That process should separate when an event happened from when the company learned about it. Both dates matter for understanding what went wrong.
A Practical Framework for Controlling AI Actions
Organizations don't need to block every AI model from using a browser. They do need to decide which actions can happen without human approval. Here's a practical starting framework for teams building or deploying AI agents.
1. List actions, not just tools
"Browser access" is too vague to be a useful permission. Break the workflow into actions: open a page, read information, enter text, upload a file, submit a form, send a message, or make a payment. Then decide which of those actions the agent can carry out on its own.
That makes the boundary visible to product, engineering, legal, and operations teams. It also avoids a common mistake: approving a tool in general without looking closely at what it can do in a specific workflow.
2. Separate drafting from execution
Let the model prepare a draft form or message without giving it the ability to send it. Show the completed draft to a person, explain where it will go, and require an explicit approval before submission.
The review has to be real. A confirmation screen that just says "Continue?" encourages quick clicking. A better approval step makes clear who will receive the content, what exactly will be sent, and what the consequences are. For a sensitive report or application, the human reviewer should be able to edit the draft or reject it.
3. Enforce limits outside the model
Use technical restrictions that match the policy you want. Depending on the system, that could mean disabling submit controls, blocking certain domains, routing submission through a separate approval service, or limiting browser credentials to read-only access.
No single control is perfect. Domain blocking can be too restrictive for legitimate tasks, and a confirmation prompt can still be clicked through inattentively. Use multiple controls based on the workflow's risk level, and test whether they hold up when the model misunderstands a page or gets conflicting instructions.
4. Make actions observable and reversible where possible
Log what pages an agent accessed, what information it entered, and whether it attempted or completed an external action. Those records help teams investigate incidents and improve safeguards. The logging design should reflect privacy and data-retention requirements, rather than capturing everything by default.
When an action can be reversed, make the recovery path obvious. Some actions can't be undone in any meaningful way. A sent police tip may already have been read, and a submitted application may already have entered a government process. Those require stronger approval before they happen - not just better rollback later.
5. Test failure cases, not only successful tasks
A test plan should include scenarios where the model sees an unexpected confirmation page, encounters an official form, receives ambiguous instructions, or reaches the point where the next click would send information externally. Evaluate whether the system stops, asks for help, or tries to infer what to do.
A useful release checklist includes:
- Can the agent submit information to an external organization?
- Which actions require human approval?
- Can the model tell when it's on a live form versus a test?
- What happens if the page behaves differently than the model expects?
- Who gets notified if an unintended submission occurs?
If the team can't answer those questions, the system's autonomy is probably ahead of its operational controls.
Conclusion
In Philadelphia, the submission was caught before a police officer reviewed it, and the details provided don't indicate it led to an investigation. But the safety question here is bigger than the outcome of one incident. An AI agent had access to a real form, misunderstood what the final step would do, and sent unsupported information into a public reporting channel.
For teams building AI products, the practical response isn't "ban agents from every website." It's to design permissions based on consequences: let systems read and draft when appropriate, but put meaningful human approval and technical controls between a model and actions that affect other people or organizations.
If you're evaluating an AI agent for your business, start by listing the external actions it can take - not just the tools it uses. Then decide which actions must stop for review. That boundary matters more than how confidently the model claims it understands the task.