Red-team lab / Learn / Reading an AI incident
Reading an AI incident
Intent over keywords, and why some incidents have no attacker at all. · 3 min read
Intent, not trigger words
In a log, the lines that matter are the ones that give the system a new task or rule. Alarming words are a poor guide: angry customers swear, people say "ignore" about each other, and some of the most effective instructions sound polite.
Failures without attackers
Some incidents are misinformation: a chatbot confidently states a policy or a law that isn't so, and someone relies on it. A tribunal has held a company responsible for what its chatbot said (Moffatt v. Air Canada).
From finding to fix
Trace each incident as a chain of steps, and break it where one control does the most: checking output before it's published, keeping the prompt free of secrets, answering policy questions from approved text, testing every release.
Practise it
Further reading
Built with Llama