Company apologizes after AI support agent invents policy that causes user uproar

Summary

On Monday, a developer using the popular AI-powered code editor Cursor noticed something strange: Switching between machines instantly logged them out, breaking a common workflow for programmers who use multiple devices. When the user contacted Cursor support, an agent named "Sam" told them it was expected behavior under a new policy. But no such policy existed, and Sam was a bot. The AI model made the policy up, sparking a wave of complaints and cancellation threats documented on Hacker News and Reddit. This marks the latest instance of AI confabulations (also called "hallucinations") causing potential business damage. Confabulations are a type of "creative gap-filling" response where AI models invent plausible-sounding but false information. Instead of admitting uncertainty, AI models often prioritize creating plausible, confident responses, even when that means manufacturing information from scratch. For companies deploying these systems in customer-facing roles without human oversight, the consequences can be immediate and costly: frustrated customers, damaged trust, and, in Cursor's case, potentially canceled subscriptions. How it unfolded The incident began when a Reddit user named BrokenToasterOven noticed that while swapping between a desktop, laptop, and a remote dev box, Cursor sessions were unexpectedly terminated. "Logging into Cursor on one machine immediately invalidates the session on any other machine," BrokenToasterOven wrote in a message that was later deleted by r/cursor moderators. "This is a significant UX regression." Confused and frustrated, the user wrote an email to Cursor support and quickly received a reply from Sam: "Cursor is designed to work with one device per subscription as a core security feature," read the email reply. The response sounded definitive and official, and the user did not suspect that Sam was not human. Screenshot of an email from the Cursor support bot named Sam. Credit: BrokenToasterOven / Reddit After the initial Re...