Dead Letter Queues (DLQ).

🎬The scenario

It's 2am. Your payment-processing worker pulls jobs off an SQS/RabbitMQ/Kafka queue. One message has a malformed amount field. The worker throws, the broker doesn't get an ack, so it redelivers the same message. Worker throws again. Redeliver. Again.

That single bad message is now stuck at the head of the queue, retried forever, burning CPU and blocking the 40,000 healthy payments behind it. This is a "poison message," and it just turned one bug into an outage.

🩹The fix

A Dead Letter Queue is a second queue where messages go after they've failed too many times. You set a max-receive-count (say 5). On the 6th delivery attempt, the broker stops redelivering and moves the message to the DLQ instead. The main queue keeps flowing; the poison is quarantined for a human to inspect.

⚙️How it actually works

• Every delivery increments a receive/redelivery counter on the message.
• Failure = no ack (exception, timeout, or explicit nack/reject).
• Threshold crossed → broker routes the message to the configured DLQ.
• The DLQ is just a normal queue — you can read it, alert on its depth, and replay from it after a fix.

SQS example (Terraform-ish):

redrive_policy = {
  deadLetterTargetArn = aws_sqs_queue.payments_dlq.arn
  maxReceiveCount     = 5
}

🔍The nuance intermediates get wrong

1. A DLQ is NOT a retry mechanism. Retries happen on the MAIN queue (often with exponential backoff). The DLQ is the terminal state after retries are exhausted. Don't confuse "retry" with "dead-letter."
2. maxReceiveCount = 1 defeats the point — one transient blip (a 500 from a downstream API) and you dead-letter a perfectly good message. Too high, and a truly poison message wastes 100 attempts. 3–5 is the usual sweet spot.
3. The DLQ is a monitoring surface, not a graveyard. If nobody alerts on DLQ depth, messages silently rot there. A rising DLQ count is one of the highest-signal "something is broken" metrics you have — wire it to an alarm.
4. Redriving needs care. Before you replay DLQ messages back to the main queue, make sure your consumer is idempotent (remember idempotency keys?). Otherwise replaying a half-processed payment charges the customer twice.

📊A number to anchor it

If your consumer takes 2s per attempt and retries a poison message 5 times before dead-lettering, that's 10s of wasted work per bad message — cheap. Without a DLQ, that same message retries indefinitely: at 2s each, it consumes an entire worker thread forever. DLQ turns "unbounded" into "10 seconds."

🧠Mental model

Main queue = the conveyor belt. DLQ = the reject bin at the end. You don't stop the belt to stare at one broken widget — you toss it in the bin and keep the line moving, then a human sorts the bin later.

🏋️Your exercise

You have a DLQ slowly filling up with ~20 messages/hour. Two questions:

1. What's the FIRST piece of data you'd pull from a dead-lettered message to triage it — and where does it usually live (hint: not the message body)?
2. You fix the bug and want to replay the 300 messages sitting in the DLQ. What one property must your consumer have before you hit "redrive," and why?