Skip to main content

Incidents

Microsoft AI Red Team case study: memory poisoning of a test email assistant

In a testbed, one poisoned email written into an email agent's memory made it forward internal emails to an outside address in 4 of 10 test cases, and in more than 80% after a prompt change.

This record is a measurement reported by a paper or a vendor, not a single dated incident.

Study finding 2025-04-24 plausible

Description

The Microsoft AI Red Team whitepaper "Taxonomy of Failure Mode in Agentic AI Systems" (April 2025) contains a case study of a memory poisoning attack on an email assistant. The assistant was a testbed built with LangChain and LangGraph on GPT-4o, with a three-part memory that it reads and writes itself. One email, received before all others, carried an instruction to forward messages about internal code and APIs to an outside recipient; the agent stored it in memory. In the baseline run the attack worked for 4 of the 10 test emails; the failures were cases where the agent did not look at its memory. After the system prompt was changed to make the agent check its memory before answering, the email was forwarded more than 80% of the time. The whitepaper uses the case to show its failure mode "Memory poisoning" and argues for checks on what an agent may store and recall.

Exhibited failure modes

Affected systems

  • test email assistant built with LangChain, LangGraph and GPT-4o

Mitigated by

  • Guardrails AI — Output validation framework with a catalog of validators and reask-on-failure loops.

Sources

Notes

An experiment on a testbed, not an incident: no deployed system was attacked. Corrected on 2026-11-21 from the whitepaper PDF. Until then this record called the case "hypothetical" and gave no numbers; the whitepaper reports a test with measured success rates. harm_class was "security_breach", which the schema defines as a confirmed incident with compromise. The case is the source example for the category MS-ES-01 in the microsoft-airt-2025 taxonomy.

View raw YAML on GitHub →