Red-team lab / Failure patterns
One user's data reached another user
Shared indexes, caches and sessions mix data from different people. Unless permissions are enforced at retrieval, the model sees whatever matched the query, and will repeat it to whoever asked.
Design it out
Enforce permissions before the model sees the data: per-tenant namespaces, access-control filtering at retrieval, and permissions kept in sync with the source.
- User-scoped access Tools act with the signed-in user's own permissions, never a shared service account.
- Permission filter at query time Filter retrieved chunks by the requesting user's permissions before they reach the model.
- Least-privilege tools Each tool does one narrow thing, with only the scopes the task needs.
- Output secret scanner Block replies that contain known secrets in common encodings.
- Permission sync and re-indexing Changes to access rights in the source systems are pushed to the index promptly.
- Tenant-isolated indexes Each customer's embeddings live in a separate index or namespace that queries can't cross.
- Trusted-source ingestion The knowledge base only ingests documents from approved, accountable sources.
The agent design check flags it as: Public users could ask it for data that isn't theirs
; Its search could answer one person with another person's data
.
Practise it
Where it happened
- Other users' chat titles and billing detailsShared infrastructure around a model (caches, queues, indexes) needs the same per-user isolation as the database.
Risk codes
MITRE ATLAS AML.T0024AML.T0136AML.T0077AML.T0053AML.T0086AML.T0101AML.T0070AML.T0071AML.T0064
OWASP Top 10 for LLM and Agentic Applications (2026) and MITRE ATLAS 2026.10, as mapped on By OWASP risk. My own mapping, not endorsed by OWASP or MITRE.
Built with Llama