New AI browser agents create risks if sites hijack them with hidden instructions

Summary

The company tested 123 cases representing 29 different attack scenarios and found a 23.6 percent attack success rate when browser use operated without safety mitigations. One example involved a malicious email that instructed Claude to delete a user's emails for "mailbox hygiene" purposes. Without safeguards, Claude followed these instructions and deleted the user's emails without confirmation. Anthropic says it has implemented several defenses to address these vulnerabilities. Users can grant or revoke Claude's access to specific websites through site-level permissions. The system requires user confirmation before Claude takes high-risk actions like publishing, purchasing, or sharing personal data. The company has also blocked Claude from accessing websites offering financial services, adult content, and pirated content by default. These safety measures reduced the attack success rate from 23.6 percent to 11.2 percent in autonomous mode. On a specialized test of four browser-specific attack types, the new mitigations reportedly reduced the success rate from 35.7 percent to 0 percent. Independent AI researcher Simon Willison, who has extensively written about AI security risks and coined the term "prompt injection" in 2022, called the remaining 11.2 percent attack rate "catastrophic," writing on his blog that "in the absence of 100% reliable protection I have trouble imagining a world in which it's a good idea to unleash this pattern." By "pattern," Willison is referring to the recent trend of integrating AI agents into web browsers. "I strongly expect that the entire concept of an agentic browser extension is fatally flawed and cannot be built safely," he wrote in an earlier post on similar prompt injection security issues recently found in Perplexity Comet. The security risks are no longer theoretical. Last week, Brave's security team discovered that Perplexity's Comet browser could be tricked into accessing users' Gmail accounts and triggering password recovery f...