Red-team lab / Case files / The bot that swore

The bot that swore

DPD (UK parcel delivery) · January 2024

Case file

Musician Ashley Beauchamp couldn't find his parcel, and DPD's support chatbot couldn't help him or put him through to a person. So he tested it instead. Asked to swear in its future answers and disregard any rules, it swore. Asked for a poem about a useless chatbot for a parcel delivery firm, it wrote one, and it called DPD the worst delivery firm in the world. He posted the screenshots on 18 January 2024 and they spread quickly.

How it was built:

  • A support chatbot with an AI component on top of scripted flows
  • No access to the customer's parcel, and no route to a person
  • Replies go straight to the customer

What happened next: DPD said an error had occurred after a system update. It disabled the AI part of the chat straight away while it was updated, noting the AI element had worked well for years.

Sources:

Phase 1Spot it

Flag every message that pushes the bot outside its job or its rules. An angry customer isn't an attacker.

Support chat, one afternoonReconstructed
0 flagged

Phase 2Stop it

Each chain below shows how the incident unfolded. Break every chain at some step, within a budget of 4, and keep the service running.

The bot swears at a customer

  1. An update loosens the bot's guardrails, and nobody notices
  2. A customer asks it to drop its rules
  3. The reply goes out unchecked

The bot runs down its own company

  1. An update loosens the bot's guardrails, and nobody notices
  2. A customer asks for poems and opinions
  3. The reply goes out unchecked
Cost 0 / 4

Logs are reconstructions: names, times and most wording are invented to match the documented pattern, except lines marked "as reported". Which controls stop which step is a teaching model based on common judgement, not a measurement. Points: 100 when both phases pass first time, 10 less per failed submission.

Built with Llama