Red-team lab / Failure patterns
A public bot was talked into saying what it shouldn't
A public chatbot speaks for the organisation to anyone, including people trying to make it say something embarrassing. Screenshots travel faster than fixes.
Design it out
Keep public bots narrow, test them against abuse before each release, and have someone watching who can switch them off.
- LLM judge A second model reviews every reply before it is shown.
- Regression and trigger evaluations Run held-out tests and backdoor-trigger checks on every new model version before release.
- Logging and anomaly alerts Log prompts, tool calls and outputs; alert on unusual patterns.
- Per-user rate limits and spend caps Cap requests and model spend per user and per day.
- Hardened system prompt Detailed rules in the prompt about secrets, untrusted content and tool use.
The agent design check flags it as: Anyone can try to make it misbehave in public
.
Practise it
Where it happened
- The bot that sworeAfter an update, test the bot for what it must never say, not only for what it should say.
- The $1 TahoeA customer-facing bot speaks for the business: keep it away from prices and commitments it can't make.
- Sydney's rulesTreat the system prompt as public: anything in it can come back out.
- The hijacked tweet botAnything a public bot reads is input, and input can carry instructions.
- Tay learns from the worst of TwitterA system that learns from public input can be taught by whoever shows up first and loudest.
Risk codes
- LLM01 Prompt injection
- LLM07 Misinformation
- LLM08 Hidden context exposure
- LLM05 Data and model poisoning
MITRE ATLAS AML.T0051.000AML.T0051.001AML.T0054AML.T0048AML.T0069.002AML.T0084.001AML.T0083
OWASP Top 10 for LLM and Agentic Applications (2026) and MITRE ATLAS 2026.10, as mapped on By OWASP risk. My own mapping, not endorsed by OWASP or MITRE.
Built with Llama