Characterizing Faults in Agentic AI: A Taxonomy of Types, Symptoms, and Root Causes
Shah, Morovati, Rahman, Khomh (Dalhousie University; Polytechnique Montréal) · varXiv v2 (2026-05-07) · first published 2026-03-06
Scope
Taxonomy of faults in agentic AI software, built from 13,602 closed issues and merged pull requests in 40 open-source repositories; 385 faults were sampled and coded by hand, and 145 practitioners rated the result. Version 2 of the paper has three lists: 34 fault types in 14 categories and four dimensions (Figure 2), 12 symptom categories (Table 1) and 12 root-cause categories (Table 2). The unit of study is a defect in the code of an agent framework, library, tool or application. It is not a failure of a coding agent on a user's task.
Why it's ingested
The largest hand-coded study of faults in agentic AI software that the KG knows of (385 sampled faults, 40 repositories, 145 practitioners). It shows which SAICA failure modes have a neighbour among the defects developers report in agent software and which do not. It cannot show that the SAICA modes are complete or separate, because it studies software defects, not agent behaviour; see data/crosswalks/shah-to-saica.yml.
Categories & crosswalks
Each row is one category of this taxonomy with the SAICA failure modes mapped to it and how close each mapping is: “exact” means the same failure, “partial” an overlap, “related” a neighbouring category. 12 mappings cover 10 of the 76 categories; the 66 with no mapping are listed under them. The category ids and labels were last compared with the source on 2026-11-21.
Mapping drafted by an AI model from the source text; not yet reviewed by a person.
-
SHAH-CAT-01LLM Interaction48 faults. Faults at the interface between an agent framework and an LLM provider: wrong model or provider configuration, incompatible API usage, inconsistent token accounting, errors in context-window handling.
SAICA-KG crosswalks: -
SHAH-CAT-02Agent Lifecycle and State38 faults. Faults in the execution lifecycle of an agent: incorrect scheduling, improper task delegation, inconsistent state transitions, defective termination logic. Agents may skip tasks, repeat actions, lose intermediate state or loop.
SAICA-KG crosswalks: -
SHAH-CAT-03Memory Management8 faults. Faults in the code that stores or retrieves context artefacts: serialization, timestamp handling, persistent storage.
SAICA-KG crosswalks: -
SHAH-CAT-04Input Interpretation59 faults. Inputs to the agent are parsed, validated, transformed or interpreted incorrectly: wrong assumptions about structure or type, flawed parsing, inconsistent encodings, missing validation.
SAICA-KG crosswalks: -
SHAH-CAT-10Dependency & Environment Management72 faults. Defects in specifying, resolving or installing software dependencies: incompatible versions, missing packages, inconsistent environments, specifications left outdated by upstream changes. The largest category.
SAICA-KG crosswalks: -
SHAH-CAT-12Failure Handling & Implementation Robustness47 faults. Faults in detecting, propagating and handling failures: suppressed exceptions, wrongly propagated error signals, recovery that does not work. They show up as "silent execution failures, incomplete task execution, or systems continuing operation with erroneous internal states".
SAICA-KG crosswalks: -
SHAH-RC-02Dependency Drift79 occurrences (21.9%). External libraries, SDKs and runtimes change independently: they "introduce breaking changes, deprecate interfaces, or alter platform support", which invalidates assumptions built into the framework.
SAICA-KG crosswalks: -
SHAH-RC-03State Management Complexity51 occurrences (14.1%). Execution state (context windows, tool traces, memory structures) is read and written by several components that come to interpret or update it inconsistently.
SAICA-KG crosswalks: -
SHAH-RC-04Model Interface Instability42 occurrences (11.6%). Assumptions about a model provider's API parameters, response structures and supported behaviour are broken by API changes.
SAICA-KG crosswalks: -
SHAH-RC-07External API Instability12 occurrences (3.3%). External APIs change their endpoints, authentication or request and response schemas, and the integration code is not updated in time.
SAICA-KG crosswalks:
Categories with no SAICA failure mode
These 66 categories have no mapping. The reason is the one the crosswalk file states; where it states none, that is said.
-
SHAH-DIM-01I. Agent Reasoning & Control86 faults. Faults in the agent's own reasoning and control flow: the LLM-driven logic that decides, iterates and updates state across steps.
Why it is not mapped: The crosswalk file gives no reason.
-
SHAH-DIM-02II. Context & Memory67 faults. Faults in how agents take in information, keep context artefacts and interpret mixed inputs (model text next to code, files or database records).
Why it is not mapped: The crosswalk file gives no reason.
-
SHAH-DIM-03III. Tooling, Integration & Actuation62 faults. Faults that arise when agents call tools, APIs and external systems, whose interfaces are strict while the agent's reasoning is not.
Why it is not mapped: The crosswalk file gives no reason.
-
SHAH-DIM-04IV. System Infrastructure & Reliability155 faults. System-level faults in the runtime environment, dependencies and operational tooling. The paper notes they are "not intrinsic to agent reasoning or decision-making".
Why it is not mapped: The crosswalk file gives no reason.
-
SHAH-CAT-05Tool Invocation11 faults. The agent invokes a defective tool or invokes a tool incorrectly: API-contract violations, unsupported operations, mismatched parameters.
Why it is not mapped: Tool Invocation: wrong arguments or unsupported operations on a real tool. This sits between two SAICA modes: fabrication needs an API that does not exist, logic_error needs a call that is correct at the signature level.
-
SHAH-CAT-06External Access19 faults. The agent cannot establish or keep access to an external service: invalid endpoints, misconfigured connections, poor handling of credentials or permissions.
Why it is not mapped: External Access: the agent cannot connect or authenticate. An availability fault, not a weakness an attacker can use, so not security_vulnerability.
-
SHAH-CAT-07Resource Interaction21 faults. Faults around shared resources needed for tool execution: invalid resource paths, wrong filesystem assumptions, misconfigured storage or databases.
Why it is not mapped: Resource Interaction: wrong paths and misconfigured storage. No SAICA mode is defined by this; single cases would fall into logic_error as the residual class.
-
SHAH-CAT-08System Coordination6 faults. Poor coordination of concurrent or asynchronous execution: race conditions and incoherent output.
Why it is not mapped: System Coordination: race conditions between components. No SAICA mode.
-
SHAH-CAT-09Execution Monitoring5 faults. Logging, tracing or metrics are missing or wrong, so a failure cannot be reconstructed.
Why it is not mapped: Execution Monitoring: logging and tracing defects. No SAICA mode.
-
SHAH-CAT-11Platform & Integration Compatibility15 faults. Assumptions about the platform do not hold at run time: operating system, CPU architecture, filesystem limits, system utilities.
Why it is not mapped: Platform & Integration Compatibility: operating system, CPU and filesystem assumptions. Not about deprecated or removed APIs, so not obsolescence.
-
SHAH-CAT-13User Interface Defects13 faults. User-facing interfaces (dashboards, visualisations) show system state or execution context incorrectly.
Why it is not mapped: User Interface Defects: dashboards of agent tools. No SAICA mode.
-
SHAH-CAT-14Documentation Issues8 faults. Documentation describes behaviour, configuration or usage incorrectly.
Why it is not mapped: Documentation Issues: wrong documentation of the framework. No SAICA mode.
-
SHAH-FT-01LLM Configuration Error13 faults in the sample (Figure 2).
Why it is not mapped: The crosswalk file gives no reason.
-
SHAH-FT-02LLM Invocation Error28 faults in the sample (Figure 2).
Why it is not mapped: The crosswalk file gives no reason.
-
SHAH-FT-03Token Budget Error6 faults in the sample (Figure 2).
Why it is not mapped: The crosswalk file gives no reason.
-
SHAH-FT-04LLM Authentication Failure1 fault in the sample (Figure 2).
Why it is not mapped: The crosswalk file gives no reason.
-
SHAH-FT-05Agent Step Execution Error26 faults in the sample (Figure 2).
Why it is not mapped: The crosswalk file gives no reason.
-
SHAH-FT-06Agent State Transition Error9 faults in the sample (Figure 2).
Why it is not mapped: The crosswalk file gives no reason.
-
SHAH-FT-07Agent Termination Error3 faults in the sample (Figure 2).
Why it is not mapped: The crosswalk file gives no reason.
-
SHAH-FT-08Memory Persistence Failure8 faults in the sample (Figure 2).
Why it is not mapped: The crosswalk file gives no reason.
-
SHAH-FT-09Structured Data Error46 faults in the sample (Figure 2).
Why it is not mapped: The crosswalk file gives no reason.
-
SHAH-FT-10Constraint Violation7 faults in the sample (Figure 2).
Why it is not mapped: The crosswalk file gives no reason.
-
SHAH-FT-11Representation Encoding Error4 faults in the sample (Figure 2).
Why it is not mapped: The crosswalk file gives no reason.
-
SHAH-FT-12Input Validation Error2 faults in the sample (Figure 2).
Why it is not mapped: The crosswalk file gives no reason.
-
SHAH-FT-13Tool Invocation Error9 faults in the sample (Figure 2).
Why it is not mapped: The crosswalk file gives no reason.
-
SHAH-FT-14Tool Configuration Error2 faults in the sample (Figure 2).
Why it is not mapped: The crosswalk file gives no reason.
-
SHAH-FT-15External Connection Error6 faults in the sample (Figure 2).
Why it is not mapped: The crosswalk file gives no reason.
-
SHAH-FT-16External Authentication Failure8 faults in the sample (Figure 2).
Why it is not mapped: The crosswalk file gives no reason.
-
SHAH-FT-17External Authorization Error5 faults in the sample (Figure 2).
Why it is not mapped: The crosswalk file gives no reason.
-
SHAH-FT-18Resource Access Error16 faults in the sample (Figure 2).
Why it is not mapped: The crosswalk file gives no reason.
-
SHAH-FT-19Storage Setup Error5 faults in the sample (Figure 2).
Why it is not mapped: The crosswalk file gives no reason.
-
SHAH-FT-20Synchronization Error6 faults in the sample (Figure 2).
Why it is not mapped: The crosswalk file gives no reason.
-
SHAH-FT-21Execution Monitoring Failure5 faults in the sample (Figure 2).
Why it is not mapped: The crosswalk file gives no reason.
-
SHAH-FT-22Dependency Compatibility Error34 faults in the sample (Figure 2).
Why it is not mapped: The crosswalk file gives no reason.
-
SHAH-FT-23Module Import Failure15 faults in the sample (Figure 2).
Why it is not mapped: The crosswalk file gives no reason.
-
SHAH-FT-24Dependency Installation Failure14 faults in the sample (Figure 2).
Why it is not mapped: The crosswalk file gives no reason.
-
SHAH-FT-25Package Resolver Error7 faults in the sample (Figure 2).
Why it is not mapped: The crosswalk file gives no reason.
-
SHAH-FT-26Environment Configuration Error2 faults in the sample (Figure 2).
Why it is not mapped: The crosswalk file gives no reason.
-
SHAH-FT-27Platform Capability Mismatch8 faults in the sample (Figure 2).
Why it is not mapped: The crosswalk file gives no reason.
-
SHAH-FT-28Unsupported Platform3 faults in the sample (Figure 2).
Why it is not mapped: The crosswalk file gives no reason.
-
SHAH-FT-29Component Integration Error3 faults in the sample (Figure 2).
Why it is not mapped: The crosswalk file gives no reason.
-
SHAH-FT-30Missing Function/Interface1 fault in the sample (Figure 2).
Why it is not mapped: The crosswalk file gives no reason.
-
SHAH-FT-31Exception Handling Failure27 faults in the sample (Figure 2).
Why it is not mapped: The crosswalk file gives no reason.
-
SHAH-FT-32Implementation Logic Error20 faults in the sample (Figure 2).
Why it is not mapped: The crosswalk file gives no reason.
-
SHAH-FT-33UI Rendering Defect13 faults in the sample (Figure 2).
Why it is not mapped: The crosswalk file gives no reason.
-
SHAH-FT-34Documentation Defect8 faults in the sample (Figure 2).
Why it is not mapped: The crosswalk file gives no reason.
-
SHAH-SYM-01Data and Validation Errors80 occurrences (22.7%). Intermediate artefacts passed between components cannot be interpreted: parsing failures, schema validation exceptions, malformed values.
Why it is not mapped: Symptom 'Data and Validation Errors': how a fault shows up to a developer, not a failure class.
-
SHAH-SYM-02Runtime Execution Errors68 occurrences (19.3%). Agent workflows stop unexpectedly or halt: interrupted workflows, failed steps, incomplete reasoning loops.
Why it is not mapped: Symptom 'Runtime Execution Errors': how a fault shows up to a developer, not a failure class.
-
SHAH-SYM-03Unhandled Exceptions45 occurrences (12.8%). Runtime errors propagate without being caught: abrupt termination with a stack trace.
Why it is not mapped: Symptom 'Unhandled Exceptions': how a fault shows up to a developer, not a failure class.
-
SHAH-SYM-04Build Errors32 occurrences (9.1%). Components cannot be compiled or packaged: compilation failures, unresolved imports, CI failures.
Why it is not mapped: Symptom 'Build Errors': how a fault shows up to a developer, not a failure class.
-
SHAH-SYM-05Dependency Installation Errors28 occurrences (8.0%). Required libraries cannot be installed: unresolved dependency trees, incompatible versions, missing modules.
Why it is not mapped: Symptom 'Dependency Installation Errors': how a fault shows up to a developer, not a failure class.
-
SHAH-SYM-06Agent Behaviour Anomalies24 occurrences (6.8%). The agent's actions or outputs deviate from the planned workflow: repeated reasoning steps, role violations, loss of conversational context, coordination breakdowns, often without a crash.
Why it is not mapped: Symptom 'Agent Behaviour Anomalies': how a fault shows up to a developer, not a failure class.
-
SHAH-SYM-07LLM Interaction Errors22 occurrences (6.2%). The agent gets no response from its model: rejected API calls, unsupported model invocations, authentication failures, context overflows.
Why it is not mapped: Symptom 'LLM Interaction Errors': how a fault shows up to a developer, not a failure class.
-
SHAH-SYM-08Network Connectivity Errors17 occurrences (4.8%). The agent cannot reach an external service: timeouts, unreachable endpoints, rejected requests.
Why it is not mapped: Symptom 'Network Connectivity Errors': how a fault shows up to a developer, not a failure class.
-
SHAH-SYM-09User Interface Errors15 occurrences (4.3%). Information about the agent's execution is shown incorrectly or incompletely.
Why it is not mapped: Symptom 'User Interface Errors': how a fault shows up to a developer, not a failure class.
-
SHAH-SYM-10Tool Invocation Errors9 occurrences (2.6%). A call to an external function or service fails: runtime errors, interrupted steps, unavailable capabilities.
Why it is not mapped: Symptom 'Tool Invocation Errors': how a fault shows up to a developer, not a failure class.
-
SHAH-SYM-11File and Resource Access Errors9 occurrences (2.6%). The agent cannot read or write an artefact it needs: file not found, read failures, failed saves.
Why it is not mapped: Symptom 'File and Resource Access Errors': how a fault shows up to a developer, not a failure class.
-
SHAH-SYM-12Concurrency Errors3 occurrences (0.9%). Inconsistent output order, merged responses or non-deterministic behaviour when components run at the same time.
Why it is not mapped: Symptom 'Concurrency Errors': how a fault shows up to a developer, not a failure class.
-
SHAH-RC-01Data Schema Mismatches101 occurrences (28.0%). Loosely structured producers (LLM outputs, tool return values) meet strictly typed consumers (parsers, validators) under inconsistent assumptions about data representation.
Why it is not mapped: Data Schema Mismatches: the cause the paper gives for Input Interpretation faults, which are mapped through SHAH-CAT-04.
-
SHAH-RC-05Error Propagation Failures29 occurrences (8.0%). Errors from LLM calls, tools or external services cross several layers without proper handling, so information about the original failure is lost.
Why it is not mapped: Error Propagation Failures: errors crossing layers without handling. SAICA's cascading_failure means something else (an agent's fixes adding new errors), so the similar name is not a match.
-
SHAH-RC-06Configuration Ambiguity15 occurrences (4.2%). Configuration is assembled from several sources (environment variables, files, constructor arguments, defaults) without a clearly defined or enforced schema.
Why it is not mapped: Configuration Ambiguity: configuration read differently by components. No SAICA mode.
-
SHAH-RC-08Architectural Coupling10 occurrences (2.8%). Tight coupling between components and abstractions inside the framework: small internal changes break components that relied on earlier behaviour.
Why it is not mapped: Architectural Coupling: a design property of the framework. No SAICA mode.
-
SHAH-RC-09Resource Limit Violations7 occurrences (1.9%). Implicit assumptions about memory use, payload size, context length or API rate limits are exceeded.
Why it is not mapped: Resource Limit Violations: memory, payload, context-length or rate limits exceeded. No SAICA mode.
-
SHAH-RC-10Credential Misconfiguration5 occurrences (1.4%). Missing, invalid or insufficiently scoped credentials stay undetected until they are used.
Why it is not mapped: Credential Misconfiguration: missing or invalid credentials that make a call fail. Not a leaked secret or an exploitable weakness, so not security_vulnerability.
-
SHAH-RC-11Concurrency Issues5 occurrences (1.4%). Asynchronous or parallel execution over shared mutable state without proper synchronisation.
Why it is not mapped: Concurrency Issues: missing synchronisation. No SAICA mode.
-
SHAH-RC-12Documentation Drift5 occurrences (1.4%). Outdated documentation: documented usage and actual behaviour diverge.
Why it is not mapped: Documentation Drift: outdated documentation. No SAICA mode.
Notes
Rebuilt on 2026-11-21 from the arXiv v2 PDF (2026-05-07): Figure 2 for the fault tree, Table 1 for symptoms, Table 2 and section 3.1.3 for root causes. All 76 labels and all counts were compared with the text of the PDF by script. The ids (SHAH-DIM, SHAH-CAT, SHAH-FT, SHAH-SYM, SHAH-RC) are SAICA-KG's; the paper publishes none. Counts are as printed: the fault tree adds up to 370, Table 1 to 352 and Table 2 to 361, while the sample is 385, and the percentages in the tables use the table totals. Version history. Version 1 of the paper (2026-03-06) had a different structure: five dimensions, 13 major categories, 37 fault categories, 13 symptom classes and 12 root causes with other names. Until 2026-11-21 this file followed v1 for the dimensions and the 13 major categories (ids SHAH-MAJ-01 to -13, now replaced by SHAH-CAT-01 to -14). Its 12 root-cause labels (for example "Incorrect Token Management", "Datetime Conversion Errors", "Exception Suppression") matched Table 2 of neither version. The ids SHAH-DIM-01 to -04 and SHAH-RC-01 to -12 are kept but now name the v2 entries, so an old reference by id must be re-read. The same rebuild was first made on 2026-08-18 (commit 4977acf) and was lost in the reset of 2026-11-19.