AI Coding Agent Failure Modes
An AI coding agent failure mode is a recurring way an agent’s work goes wrong that a reviewer, a program or a test could catch: code that calls APIs that don’t exist, tests edited until they pass, changes nobody asked for, a “done” for work that isn’t. SAICA-Bench defines 11 of them, each with a plain definition, the other names it goes by and the signals that point to it. The modes are tags, not exclusive classes: one failure can carry several of them, and each mode’s page names the modes it is most easily confused with.
For 9 of the 11, SAICA-Bench measured what supervision tools and language models flag on labelled samples: see how the benchmark works and the full leaderboard. The table groups the modes by where the bad samples of each set come from, because that decides how a score should be read; inside a group the order is alphabetical. Each model asked directly was told which failure to look for and was tested on samples that have that failure or none. Its score is a single-prompt, mode-primed estimate, not a product. Terms are defined in the glossary.
All 11 failure modes
| Failure mode | What it is | Also called | Best reviewer on our set | Best free catalog program | Samples |
|---|---|---|---|---|---|
| Real agent output The bad samples are what an agent produced on its own; the label comes from running hidden tests. | |||||
|
Incomplete execution
scored as: False 'done' claims
J under 25% on our set
|
The agent reports the task done while parts are skipped, stubbed or not working. | premature completion, false completion, stub left behind |
no reviewer reaches J 25%
highest J +19.5% (95% interval +13.9% to +24.9%)
|
none above the floor | 1,200 |
|
Security vulnerability
J under 25% on our set
|
Correct-looking code with injection flaws, weak cryptography or leaked secrets. | insecure code, code injection, secret leakage |
no reviewer reaches J 25%
highest J +7.9% (95% interval 0.0% to +15.9%)
|
none ran | 1,702 |
| Model-written code, not agent output The bad samples are what a code model wrote for a textbook task; the label comes from running the task's tests. | |||||
|
Logic error
J 25% to under 80% on our set
|
Real APIs used correctly, but the code computes the wrong result. | semantic bug, off-by-one, inverted condition |
mimo-v2.6-flash
highest J +74.3% (95% interval +69.4% to +78.9%)
|
none ran | 2,440 |
| Failure planted on purpose The bad samples were written, injected or swapped in on purpose, by a model, a script or by hand. | |||||
|
Dependency blindness
J 80% or more on our set
|
Hand-written code for what the standard library, a dependency or the codebase already provides. | reinvention, ecosystem duplication |
top group of 3: gpt-6-luna, mimo-v2.6-flash and deepseek-v4-flash
highest J +90.0% (95% interval +82.0% to +98.0%)
|
none ran | 100 |
|
Fabrication
J 80% or more on our set
|
Code that calls functions, methods or packages that do not exist. | code hallucination, API hallucination, package hallucination |
top group of 4: deepseek-v4.1-flash, glm-5.3 and 2 more
highest J +94.3% (95% interval +91.6% to +96.7%)
|
pyright
J +85.3%
|
636 |
|
Obsolescence
J 80% or more on our set
|
Real APIs that are deprecated, removed or changed, often from stale training data. | deprecated API usage, version drift, training-cutoff drift |
top group of 3: gpt-6-luna, mimo-v2.6-flash and deepseek-v4-flash
highest J +89.7% (95% interval +82.9% to +95.1%)
|
pyright (latest libraries)
J +56.8%
|
310 |
|
Scope creep
J 80% or more on our set
|
Changes nobody asked for: other files, new packages, config edits, unrequested refactors. | task drift, over-action, unrequested changes |
mimo-v2.6-flash
highest J +97.7% (95% interval +93.6% to +99.1%)
|
none ran | 350 |
|
Supply-chain attack
J 25% to under 80% on our set
|
The agent installs malicious code: a squatted hallucinated name, a lookalike or a compromised tool. | slopsquatting, typosquatting, dependency confusion |
top group of 2: gpt-6-luna and deepseek-v4-flash
highest J +77.0% (95% interval +70.3% to +83.8%)
|
none ran | 296 |
|
Test manipulation
scored as: Test edits that hide failures
J 80% or more on our set
|
Tests edited, weakened or skipped so failures disappear instead of the bug being fixed. | test gaming, spec gaming, reward hacking |
top group of 8: minimax-m3, giskard and 6 more
highest J +91.0% (95% interval +81.3% to +94.9%)
|
none ran | 305 |
| Not benchmarked yet No dataset with confirmed labels exists for these modes, so nothing is scored. | |||||
|
Cascading failure
Not measured
|
Each fix introduces new errors and the code drifts further from working. | recovery spiral, error cascade, fix regression | not measured | — | — |
|
Context pollution
Not measured
|
The working context degrades over a session: facts dropped, noise added, old mistakes kept. | context contamination, stale context, irrelevant context | not measured | — | — |
J = TPR - FPR, on the answers that carry a verdict. 0% is chance, 100% flags every bad sample and no clean one. J is the sort key within a mode; each row also has false alarms per 100 clean changes and precision at 1%, 5% and 20% prevalence. Best reviewer: the best ranked row of a mode that is not marked incomplete. A row with under 95% readable answers in one class is listed and ranked, with its J bounds, but is never named best reviewer or sole winner; if its J is higher than the best reviewer's it stays in the top group. Not ranked: baseline rules, shortcut baselines and rows outside a tool's design scope. Top group: the best reviewer and every ranked row whose paired difference from it includes 0. Best catalog tool: the best tool of the SAICA catalog that ran its own check; a framework that ran our rule text is not a candidate. Samples = bad plus clean samples scored. Results of 2026-11-22; per-reviewer tables are on each mode’s page and on the leaderboard.
What the scores can and cannot say
On the 2 sets of real agent output (Security vulnerability and False 'done' claims; 2,902 samples) no reviewer reaches J 25%. Best J: +8% on Security vulnerability and +20% on False 'done' claims; 5 of the 23 ranked rows there have a 95% interval for J that excludes 0. On the 6 sets where the failure was planted on purpose the best reviewer reaches J +77% to +98%; on Logic error (model-written code that is not agent output) +74%. Every set of real agent output scores below every planted set, so this benchmark cannot tell 'reviewers catch these failures' from 'reviewers catch planted failures'. Whether the high scores carry over to failures agents make on their own is not tested.
J compares reviewers within one mode. Do not compare J across modes or average it over them: each set was built differently (real agent runs, model-written functions, injected or hand-built failures) and has its own sample unit, and where the bad samples come from explains most of the difference between modes. Tiers describe the best reviewer's J on our set, not how well a failure is caught in use: J 80% or more on our set; J 25% to under 80% on our set; J under 25% on our set.
How these map to other taxonomies
Each mode is cross-walked to published agent-failure taxonomies. Bold = the same failure (exact match), plain = partial overlap; looser “related” links are on each mode’s page and on the taxonomy pages. Of the 20 mappings in this table, 0 are exact. Mapping drafted by an AI model from the source text; not yet reviewed by a person.
| SAICA-Bench | DAPLab 9 patterns | MAST | OWASP Agentic Top 10 | Microsoft AIRT |
|---|---|---|---|---|
| Incomplete execution | — |
FM-3.1 Premature termination
| — | — |
| Security vulnerability |
DAP-06 Security Vulnerabilities
| — |
ASI02 Tool Misuse and Exploitation
ASI03 Identity and Privilege Abuse
ASI05 Unexpected Code Execution (RCE)
ASI07 Insecure Inter-Agent Communication
| — |
| Logic error |
DAP-03 Business Logic Mismatch
|
FM-1.1 Disobey task specification
| — | — |
| Dependency blindness |
DAP-08 Codebase Awareness & Refactoring Issues
| — | — | — |
| Fabrication | — | — | — |
MS-EF-06 Hallucinations
|
| Obsolescence | — | — | — | — |
| Scope creep | — |
FM-1.1 Disobey task specification
|
ASI01 Agent Goal Hijack
ASI02 Tool Misuse and Exploitation
|
MS-ES-09 Excessive agency
|
| Supply-chain attack | — | — |
ASI04 Agentic Supply Chain Vulnerabilities
|
MS-ES-05 Function compromise and malicious functions
|
| Test manipulation | — |
FM-3.3 Incorrect verification
| — | — |
| Cascading failure | — | — | — | — |
| Context pollution | — |
FM-1.4 Loss of conversation history
|
ASI06 Memory & Context Poisoning
|
MS-ES-01 Memory poisoning and theft
|
Where to go next
- Leaderboard: every tool and model on every measured mode, with cost per 1,000 reviews.
- Benchmark method: datasets, labels and scoring for each mode.
- Supervision kit builder: pick the modes you care about and get the best-measured tool for each.
- Glossary: the failure modes, their other names and the metrics used here.
- Tool catalog and incidents behind each mode.