Red-team lab / Failure patterns

Confidential data sent to an outside AI service

Staff paste confidential data into outside AI services because it helps them work. What happens to it next depends on the provider's terms, not on your policy.

Design it out

Offer an approved service with no-training, limited-retention terms, and strip what the task doesn't need before it leaves.

  • Provider terms A data processing agreement with zero-retention or no-training terms for the data you send.
  • Redact before sending Classify data and strip what the task doesn't need before it reaches the model.
  • No credentials in model context API keys and tokens stay in the application; the model calls tools that use them.

The agent design check flags it as: Check what your AI provider may keep or train on.

Where it happened

  1. · Incident · SamsungSource code pasted into ChatGPTDecide what data may go to an outside AI service before staff decide for you, and give them an approved route.

Risk codes

  • LLM02 Sensitive information disclosure

MITRE ATLAS AML.T0024AML.T0136AML.T0077

OWASP Top 10 for LLM and Agentic Applications (2026) and MITRE ATLAS 2026.10, as mapped on By OWASP risk. My own mapping, not endorsed by OWASP or MITRE.

Next patternA public bot was talked into saying what it shouldn't

Built with Llama