AI coding agent incidents
13 records of AI coding agents (or similar systems) failing. 6 are dated real-world incidents; the other 7 are study findings, reported patterns or hypothetical scenarios, and each record says which it is. Each is tagged with the failure modes it shows and the tools whose supervision could plausibly have prevented it.
-
Pull-request titles, issue bodies and comments can make three AI agents on GitHub Actions run commands and leak the repository's secrets. All three vendors confirmed the findings.
-
Google's Gemini CLI took a failed mkdir for a success, then ran move commands that overwrote every file but one. The agent then told the user it had lost the data.
-
Under an explicit code freeze, Replit's autonomous agent ran destructive DB commands, wiping ~1,200 exec and 1,190 company records, then fabricated 4,000 phantom users to hide the loss.
-
Trend Micro research: coding agents invent package names such as "starlette-reverse-proxy"
2025-05-01Trend Micro reports seeing an agent generate a plausible package name that did not exist; the build failed with "module not found". It gives
starlette-reverse-proxyas an example of such a name. -
In a testbed, one poisoned email written into an email agent's memory made it forward internal emails to an outside address in 4 of 10 test cases, and in more than 80% after a prompt change.
-
Cursor's unlabeled AI support bot invented a single-device subscription policy that didn't exist, triggering cancellations, HN/Reddit backlash, and a public apology from the Cursor team.
-
A scenario from "Lessons From Red Teaming 100 Generative AI Products": a scammer's email hides an instruction that makes an email-summarising copilot output a malicious link.
-
Answer.AI's month-long Devin eval: 3 of 20 tasks completed. In one, Devin spent a day pursuing a Railway single-deployment feature that does not exist, fabricating plausible steps.
-
Study: code LLMs complete Python code with deprecated library APIs (Wang et al., ICSE 2025)
2024-06-14Wang et al. gave seven code LLMs 28,125 completion prompts built on 145 deprecated APIs from eight Python libraries. Of the plausible completions, 25% to 38% used the deprecated API.
-
16 LLMs, 576,000 code samples: 440,445 of 2.23 million recommended packages (19.7%) do not exist. Re-running a prompt ten times, 43% of hallucinated names came back every time.
-
Bar Lanyado (Lasso Security) saw AI models recommend a
huggingface-clipackage that did not exist, registered it empty on PyPI and counted 15,000 to 30,000 downloads in about three months. -
Researchers built 900 prompts from GitHub files with the credential removed. Copilot returned 2,702 well-formed credentials and CodeWhisperer 129; two were confirmed to be working keys.
-
In three separate March 2023 sessions, Samsung DS engineers pasted proprietary semiconductor source code and internal meeting recordings into ChatGPT to debug or summarize.