Red-team lab / Learn / Output, rendering and trust

Output, rendering and trust

What happens to a model's answer: rendered, executed, or believed. · 3 min read

Output is input to something else

A model's answer is rarely the end. A browser renders it, a shell runs it, another agent reads it, or a person acts on it. Each of those is a place where a manipulated answer turns into harm.

Rendering is the classic case: a chat that displays images makes the browser fetch whatever URL the model wrote, with no click. Put data in that URL and rendering becomes a leak.

When the target is the person

People trust tools that are usually right. An instruction that reaches someone through their assistant borrows its credibility, so a human check only helps if the human gets independent information.

What holds

Treat model output as untrusted: sanitise rendered Markdown, never execute it unreviewed, and show where each claim or recommendation comes from. Keep verification steps (a bank change, a policy answer) outside the assistant.

Practise it

Further reading

Next concept · 3 minDesigning an AI system to fail safely

Built with Llama