Red-team lab / Failure patterns
Content the agent reads steers what it does
Anything the agent reads (a web page, an email, a ticket, a tool's output) is input to the same model that decides what to do next. Instructions hidden in that content compete with yours, and filters only catch the phrasings they were written for.
Design it out
Assume what the agent reads contains instructions. Design actions so that following them can't do harm: narrow tools, fixed recipients, and approvals shown by the app, not described by the model.
- Least-privilege tools Each tool does one narrow thing, with only the scopes the task needs.
- Per-tool scoped permissions Each tool gets only the access its job needs, granted separately from the agent's other tools.
- System-rendered confirmation Before a risky action, the app (not the model) shows the user the exact action and parameters to approve.
- Human checkpoint for high-impact actions Payments and other irreversible actions wait for a person to approve them.
- Untrusted-content markers Wrap retrieved or received content in delimiters and tell the model it is data.
The agent design check flags it as: Outsiders, or what they write, can steer what it does
.
Practise it
Where it happened
- CurXecuteAn agent must not be able to change its own tools or configuration without a person approving it.
- The support ticket that read the tokens tableGive agents read-only, project-scoped access, and never a role that bypasses the database's own permission checks.
- EchoLeakAn assistant that reads incoming mail has untrusted input by design: control what it can send out, not only what it reads.
- A public issue reads the private reposOne broad token across public and private data is the problem, not the server code: scope the agent to one repository per task.
- Slack AI and the private channelRetrieval that mixes content from people with different access lets the least trusted author steer answers built from the most private data.
- The shared docIf the interface renders model output as images or links, the model has a way to send data out.
- The hijacked tweet botAnything a public bot reads is input, and input can carry instructions.
Risk codes
- LLM01 Prompt injection
- LLM02 Sensitive information disclosure
- ASI01 Agent goal hijack
- ASI03 Identity and privilege abuse
- LLM10 Improper output handling
MITRE ATLAS AML.T0051.000AML.T0051.001AML.T0054AML.T0024AML.T0136AML.T0077AML.T0080
OWASP Top 10 for LLM and Agentic Applications (2026) and MITRE ATLAS 2026.10, as mapped on By OWASP risk. My own mapping, not endorsed by OWASP or MITRE.
Built with Llama