Red-team lab / Failure patterns
What the system learns or remembers was planted
A system that learns from feedback, retrieves from an index or keeps memories can be taught by whoever writes to those sources. The planted content then acts later, long after it arrived.
Design it out
Record where every learned, indexed or remembered item came from, cap the influence of any one source, review before it shapes behaviour, and let memories expire.
- Memory with provenance Every memory records its source; memories suggested by untrusted content need the user's confirmation.
- Memory review and expiry Memories expire and can be reviewed and deleted by the user.
- Review gate before retraining Sample, check for anomalies and approve new data before it is used for training.
- Per-user influence caps Limit how much any single user or account can contribute to what the model learns.
- Regression and trigger evaluations Run held-out tests and backdoor-trigger checks on every new model version before release.
The agent design check flags it as: What it learns or remembers can be planted
.
Practise it
Where it happened
- A memory that kept reporting backMemory turns a one-off injection into a permanent one. Record where each memory came from and let users review it.
- Tay learns from the worst of TwitterA system that learns from public input can be taught by whoever shows up first and loudest.
Risk codes
- LLM05 Data and model poisoning
- ASI06 Memory and context poisoning
- LLM01 Prompt injection
- LLM02 Sensitive information disclosure
MITRE ATLAS AML.T0020AML.T0018.000AML.T0070AML.T0080.000AML.T0051.000AML.T0051.001AML.T0054
OWASP Top 10 for LLM and Agentic Applications (2026) and MITRE ATLAS 2026.10, as mapped on By OWASP risk. My own mapping, not endorsed by OWASP or MITRE.
Built with Llama