Skip to content
Star16Hire me
01 / StartXP 0%
Sep 1, 2026 · 2 min · by Ahmed Mamdouh

At-least-once delivery means at least once

#nodejs#distributed-systems#queues#redis#backend

The worst bug I ever shipped ran perfectly. It was a queue consumer that wasn't idempotent, and the fix is the same every time: give every consumer an idempotency key and check it before doing any work.

The bug that did not crash

It was a payment notification consumer. The queue delivers a message, the consumer sends the notification. I tested it, deployed it, and it worked.

Then a network blip made the queue redeliver a batch. Every affected user got the same payment notification twice, and some got it three times. Nothing crashed and nothing logged an error, because the system did exactly what I built it to do. It just did it twice.

That week I learned the phrase that has lived rent free in my head ever since: at-least-once delivery means at least once.

The boring fix

Every consumer gets an idempotency key. Before doing the work, it checks whether that key was already processed. A Redis SETNX with a TTL covers most cases in a few lines:

async function handle(message: { id: string; payload: unknown }) {
  const key = `processed:${message.id}`;
  // SET key NX EX: only succeeds if the key does not exist yet
  const firstTime = await redis.set(key, "1", "EX", 60 * 60 * 24, "NX");
  if (!firstTime) return; // already handled, skip
 
  await sendNotification(message.payload);
}

The key has to come from the message itself, something stable like an event ID. If you generate it when the consumer runs, every redelivery looks new and the check does nothing.

Duplicates come from everywhere

What surprised me is how far the rule reaches. Retries, replays, manual reprocessing after an incident, a colleague running a backfill script: all of them send the same work through twice. The queue was only the first place I noticed it.

The one review question

When I review a consumer now, the first thing I ask is what happens if it runs twice. If the answer is "that can't happen", I already know what the next comment is going to say. It can, and eventually it will, usually during the one week you'd least like it to.

Building something like this?

I'm Ahmed Mamdouh, a senior full-stack and AI engineer. I reply within one working day.

More posts