Red-team lab / Learn / Defences that raise the cost, and defences that hold

Defences that raise the cost, and defences that hold

Filters, judges and prompt rules versus architectural controls. · 3 min read

Probabilistic layers

Prompt rules, keyword filters, output scanners and LLM judges all work by recognising something bad. They catch the cases their authors anticipated and miss the rest; a judge is itself a model that can be persuaded. They are worth having, because they make attacks slower and noisier.

Test them like code: a fixed suite of attacks, run on every change, and expect scores to move between runs.

Deterministic controls

Other controls don't depend on recognising anything: a secret that isn't in the context can't leak, a tool the agent doesn't hold can't be misused, an image the chat won't render can't make a request. These remove whole classes of outcome.

Putting them together

Start from what an attacker could achieve, remove what you can by design, then add probabilistic layers where design can't reach. Spend where one control removes a whole attack, which is exactly how the architecture challenges score.

Practise it

Further reading

Next concept · 3 minAgents, tools and excessive agency

Built with Llama