Study: 19.7% of the package names recommended by 16 code LLMs do not exist (Spracklen et al., USENIX Security 2025)
16 LLMs, 576,000 code samples: 440,445 of 2.23 million recommended packages (19.7%) do not exist. Re-running a prompt ten times, 43% of hallucinated names came back every time.
This record is a measurement reported by a paper or a vendor, not a single dated incident.
Description
"We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs" (Spracklen, Wijewickrama, Sakib, Maiti, Viswanath and Jadliwala; arXiv 2406.10279, USENIX Security 2025) measures how often code LLMs recommend packages that do not exist. With 16 models and two prompt datasets the authors generated 576,000 code samples in Python and JavaScript. The samples named 2.23 million packages, of which 440,445 (19.7%) were not in PyPI or npm; 205,474 of those names were unique. The rate is a share of recommended package names, not of code samples. It was 5.2% on average for the commercial models and 21.7% for the open-source models. To test persistence, 500 prompts that had produced a hallucination were each run ten more times: 43% of the hallucinated packages came back in all ten runs and 39% in none. The paper calls the failure "package hallucination" and the attack it enables a package confusion attack; it does not use the later term slopsquatting.
Exhibited failure modes
- Fabrication — Generating reference to non-existent APIs, packages, or symbols.
- Supply-chain attack — Slopsquatting, typosquatting, compromised MCP servers.
Affected systems
- GPT-4
- GPT-3.5
- CodeLlama
- DeepSeekCoder
- Mistral
- 11 others evaluated in the paper
Mitigated by
- Socket — Dependency supply-chain scanner that inspects packages before install.
- Snyk — Vulnerability scanning for code, dependencies, containers, and IaC.
Documented by
Sources
Notes
A study, not an incident; incident_date is the date of the first arXiv version. Corrected on 2026-11-21 from the paper (v3). Until then the record said that 19.7% of code samples reference a package that does not exist; the paper's 19.7% is the share of recommended package names. It named the venue as USENIX Security 2024 with the date 2024-08-14 and linked to a usenix.org page that returns 404; the paper is in USENIX Security 2025. harm_class was "security_breach", which the schema defines as a confirmed incident with compromise; a measurement is not one. reproducibility is "plausible" because no independent replication is recorded here.