Red-team lab / Learn / Defences that raise the cost, and defences that hold
Defences that raise the cost, and defences that hold
Filters, judges and prompt rules versus architectural controls. · 3 min read
Probabilistic layers
Prompt rules, keyword filters, output scanners and LLM judges all work by recognising something bad. They catch the cases their authors anticipated and miss the rest; a judge is itself a model that can be persuaded. They are worth having, because they make attacks slower and noisier.
Test them like code: a fixed suite of attacks, run on every change, and expect scores to move between runs.
Deterministic controls
Other controls don't depend on recognising anything: a secret that isn't in the context can't leak, a tool the agent doesn't hold can't be misused, an image the chat won't render can't make a request. These remove whole classes of outcome.
Putting them together
Start from what an attacker could achieve, remove what you can by design, then add probabilistic layers where design can't reach. Spend where one control removes a whole attack, which is exactly how the architecture challenges score.
Practise it
Further reading
Built with Llama