At-least-once delivery means at least once

The worst bug I ever shipped ran perfectly. It was a queue consumer that was not idempotent, and the fix is the same every time: give every consumer an idempotency key and check it before doing any work.

The bug that did not crash

It was a payment notification consumer. The queue delivers a message, the consumer sends the notification. Tested, deployed, worked.

Then a network blip made the queue redeliver a batch. Every affected user got the same payment notification twice. Some got it three times. Nothing crashed. No errors. The system did exactly what I built it to do, twice.

That week I learned the phrase that now lives rent free in my head: at-least-once delivery means at least once.

The boring fix

Every consumer gets an idempotency key. Before doing the work, check if the key was already processed. A Redis SETNX with a TTL covers most cases in a few lines:

async function handle(message: { id: string; payload: unknown }) {
  const key = `processed:${message.id}`;
  // SET key NX EX: only succeeds if the key does not exist yet
  const firstTime = await redis.set(key, "1", "EX", 60 * 60 * 24, "NX");
  if (!firstTime) return; // already handled, skip
 
  await sendNotification(message.payload);
}

The key has to come from the message itself, something stable like an event ID, not something generated when the consumer runs. Otherwise every redelivery looks new.

Duplicates come from everywhere

What surprised me is how deep the rule goes. Retries, replays, manual reprocessing after an incident, a colleague running a backfill script. Duplicates come from everywhere. The queue is just the messenger.

The one review question

Now when I review a consumer, I ask one question first: what happens if this runs twice? If the answer is "that can't happen", the review is over. It can.

Takeaways

• At-least-once delivery guarantees duplicates eventually, not just in theory.
• Give every consumer an idempotency key derived from the message, and check it before doing work.
• A Redis SET ... NX with a TTL handles most cases.
• Retries, replays and backfills cause duplicates too, not only the queue.
• In review, ask first: what happens if this runs twice?