Designing an AI system to fail safely

Draw the boundaries

List who can write to each input (users, documents, the web, other agents), what the model can reach (data, tools, credentials) and where its output goes. Most AI incidents sit on one of those lines.

Work back from outcomes

For each outcome you can't accept (a secret leaving, an unwanted action, a cost spike, a false promise), ask which control removes it, which only reduces it, and what each costs. Controls compound, and none reaches zero.

Some options aren't available: an email agent has to read sensitive mail, so the defence has to sit on what it can do rather than what it can see.

What holds

Remove the asset or the capability where you can, scope access to the user, add confirmations and limits, and keep prompt-level controls as the last layer, not the first.