Devin spends >24h hallucinating Railway features rather than recognizing platform limit
Answer.AI's month-long Devin eval: 3 of 20 tasks completed. In one, Devin spent a day pursuing a Railway single-deployment feature that does not exist, fabricating plausible steps.
This record is a measurement reported by a paper or a vendor, not a single dated incident.
Description
Answer.AI's public evaluation of Devin (Cognition AI's "AI software engineer") ran for a month across 20 representative tasks, of which Devin completed 3 satisfactorily, 3 inconclusively, and failed 14 outright. The team's most-cited individual failure is the Railway deployment: asked to deploy multiple apps to a single Railway deployment (which Railway does not support), Devin spent more than a day attempting approaches based on hallucinated Railway features rather than reporting the platform constraint. Subsidiary failures included generating spaghetti code for greenfield projects and producing a DaisyUI theme that "looked right" but used default rather than requested colors. The review is the most cited cautionary public study of autonomous coding-agent incompleteness.
Exhibited failure modes
- Fabrication — Generating reference to non-existent APIs, packages, or symbols.
- Incomplete execution — Agent claims completion while subtasks are skipped, stubbed, or silently dropped.
- Scope creep — Actions outside declared task boundary.
Affected systems
- Devin (Cognition AI)
Mitigated by
No mitigating tools indexed for this incident yet.
Sources
Notes
An evaluation of 20 tasks by one team, so kind is "study"; the Railway episode is one task inside it. incident_date is the date of Answer.AI's post (the id says 2024 because ids are not changed once published). mitigated_by is empty: no KG tool is linked. A step-by-step review of the plan would have shown the Railway limit earlier, but that is a way of working, not a tool in the KG.