Red-team lab / Failure patterns
Untrusted content, private data and a way out, in one agent
A model can't reliably tell instructions from data. When one agent reads text from outsiders, can reach private data and has any way to send something out, the outsider's text can ask for the data and the agent can deliver it. Each part is ordinary on its own; the risk is the combination.
Design it out
Never give one task all three. Split the task, keep outsiders' text away from private data, or close the way out with allowlists and app-rendered confirmation.
- User-scoped access Tools act with the signed-in user's own permissions, never a shared service account.
- Egress allowlist The agent's network tools can only reach approved hosts.
- Markdown sanitiser Strip external images and rewrite links in model output before rendering.
- Image host allowlist (CSP) The browser only loads images from your own domains.
- Recipient allowlist The send tool only reaches approved domains.
- System-rendered confirmation Before a risky action, the app (not the model) shows the user the exact action and parameters to approve.
- Untrusted-content markers Wrap retrieved or received content in delimiters and tell the model it is data.
The agent design check flags it as: Outsiders could make it leak your data
.
Practise it
Where it happened
- The support ticket that read the tokens tableGive agents read-only, project-scoped access, and never a role that bypasses the database's own permission checks.
- EchoLeakAn assistant that reads incoming mail has untrusted input by design: control what it can send out, not only what it reads.
- A public issue reads the private reposOne broad token across public and private data is the problem, not the server code: scope the agent to one repository per task.
- A memory that kept reporting backMemory turns a one-off injection into a permanent one. Record where each memory came from and let users review it.
- Slack AI and the private channelRetrieval that mixes content from people with different access lets the least trusted author steer answers built from the most private data.
- The shared docIf the interface renders model output as images or links, the model has a way to send data out.
Risk codes
- LLM01 Prompt injection
- LLM02 Sensitive information disclosure
- LLM10 Improper output handling
- ASI03 Identity and privilege abuse
- ASI01 Agent goal hijack
MITRE ATLAS AML.T0051.000AML.T0051.001AML.T0054AML.T0024AML.T0136AML.T0077AML.T0067AML.T0050
OWASP Top 10 for LLM and Agentic Applications (2026) and MITRE ATLAS 2026.10, as mapped on By OWASP risk. My own mapping, not endorsed by OWASP or MITRE.
Built with Llama