Red-team lab / Failure patterns
Hidden instructions came back out
Whatever is in the model's context can come back out: system instructions, retrieved policy text, tool definitions. Rules telling the model not to reveal them reduce leaks; they don't stop them.
Design it out
Treat everything in the context as visible to the user. Keep secrets and authorization logic out of prompts, and enforce permissions outside the model.
- No credentials in model context API keys and tokens stay in the application; the model calls tools that use them.
- No standing secrets for the agent The agent gets short-lived, task-scoped credentials instead of the developer's own keys and tokens.
- Least-privilege tools Each tool does one narrow thing, with only the scopes the task needs.
Practise it
Where it happened
- Sydney's rulesTreat the system prompt as public: anything in it can come back out.
Risk codes
MITRE ATLAS AML.T0069.002AML.T0084.001AML.T0083AML.T0051.000AML.T0051.001AML.T0054
OWASP Top 10 for LLM and Agentic Applications (2026) and MITRE ATLAS 2026.10, as mapped on By OWASP risk. My own mapping, not endorsed by OWASP or MITRE.
Built with Llama