The worst bug I ever shipped ran perfectly. It was a queue consumer that wasn't idempotent, and the fix is the same every time: give every consumer an idempotency key and check it before doing any work.
The bug that did not crash
It was a payment notification consumer. The queue delivers a message, the consumer sends the notification. I tested it, deployed it, and it worked.
Then a network blip made the queue redeliver a batch. Every affected user got the same payment notification twice, and some got it three times. Nothing crashed and nothing logged an error, because the system did exactly what I built it to do. It just did it twice.
That week I learned the phrase that has lived rent free in my head ever since: at-least-once delivery means at least once.
The boring fix
Every consumer gets an idempotency key. Before doing the work, it checks whether that key was already processed. A Redis SETNX with a TTL covers most cases in a few lines:
async function handle(message: { id: string; payload: unknown }) {
const key = `processed:${message.id}`;
// SET key NX EX: only succeeds if the key does not exist yet
const firstTime = await redis.set(key, "1", "EX", 60 * 60 * 24, "NX");
if (!firstTime) return; // already handled, skip
await sendNotification(message.payload);
}The key has to come from the message itself, something stable like an event ID. If you generate it when the consumer runs, every redelivery looks new and the check does nothing.
Duplicates come from everywhere
What surprised me is how far the rule reaches. Retries, replays, manual reprocessing after an incident, a colleague running a backfill script: all of them send the same work through twice. The queue was only the first place I noticed it.
The one review question
When I review a consumer now, the first thing I ask is what happens if it runs twice. If the answer is "that can't happen", I already know what the next comment is going to say. It can, and eventually it will, usually during the one week you'd least like it to.
Building something like this?
I'm Ahmed Mamdouh, a senior full-stack and AI engineer. I reply within one working day.

Watch queue depth trend, not the current number
Track queue depth over time from day one. The current number says little; the slope tells you whether you have twenty minutes or two.

Scaling 100k WebSocket connections: the reconnect storm
At 100k+ concurrent sockets the count is easy. The reconnect storm is what breaks, and jittered backoff, load shedding and resumable sessions fix it.

npm v12 blocks install scripts by default
npm v12 no longer runs preinstall, install or postinstall scripts unless you approve them, closing a common supply chain attack path.
