AI can't stop making up software dependencies and sabotaging everything

Summary

The rise of AI-powered code generation tools is reshaping how developers write software - and introducing new risks to the software supply chain in the process. AI coding assistants, like large language models in general, have a habit of hallucinating. They suggest code that incorporates software packages that don't exist. As we noted in March and September last year, security and academic researchers have found that AI code assistants invent package names. In a recent study, researchers found that about 5.2 percent of package suggestions from commercial models didn't exist, compared to 21.7 percent from open source models. Running that code should result in an error when importing a non-existent package. But miscreants have realized that they can hijack the hallucination for their own benefit. All that's required is to create a malicious software package under a hallucinated package name and then upload the bad package to a package registry or index like PyPI or npm for distribution. Thereafter, when an AI code assistant re-hallucinates the co-opted name, the process of installing dependencies and executing the code will run the malware. The recurrence appears to follow a bimodal pattern - some hallucinated names show up repeatedly when prompts are re-run, while others vanish entirely - suggesting certain prompts reliably produce the same phantom packages. As noted by security firm Socket recently, the academic researchers who explored the subject last year found that re-running the same hallucination-triggering prompt ten times resulted in 43 percent of hallucinated packages being repeated every time and 39 percent never reappearing. Exploiting hallucinated package names represents a form of typosquatting, where variations or misspellings of common terms are used to dupe people. Seth Michael Larson, security developer-in-residence at the Python Software Foundation, has dubbed it "slopsquatting" – "slop" being a common pejorative for AI model output. "We're in the ...