Dead Letter Queue

What is a dead letter queue?

A dead letter queue (DLQ) is where messages go when normal processing has given up on them. In asynchronous systems a consumer reads a message, tries to handle it and acknowledges it on success. If handling fails, the message becomes visible again and is retried. Some messages will never succeed: malformed payloads, references to records that no longer exist, or a bug that only certain inputs trigger. Without a DLQ, such a poison message is retried forever, wastes capacity and, in ordered queues, blocks everything behind it.

A DLQ breaks that loop. After a defined number of failed attempts, or when a message is rejected or expires, the broker or the consumer moves it aside. The main queue keeps flowing, and the failed message stays available for analysis and replay.

How a dead letter queue works in common brokers

Amazon SQS

In SQS a DLQ is an ordinary queue referenced from the source queue's redrive policy. The policy sets deadLetterTargetArn and maxReceiveCount: once a message has been received that many times without being deleted, SQS moves it to the DLQ. The DLQ must be of the same type as the source (standard or FIFO) and live in the same account and region. A redrive allow policy on the DLQ controls which source queues may use it, and a DLQ redrive moves messages back to the source queue after a fix. For standard queues the original enqueue timestamp is kept, so the DLQ's retention period should be longer than the source queue's.

Apache Kafka

Kafka brokers have no built-in DLQ. A Kafka topic is a log, and consumers track offsets rather than acknowledging single messages. The dead letter pattern is implemented in the client or framework: on a failure the consumer publishes the record, often with error details in headers, to a separate topic such as orders.DLT, then commits the offset and moves on. Kafka Connect supports this for sink connectors via errors.tolerance=all and errors.deadletterqueue.topic.name, and frameworks such as Spring for Apache Kafka provide a dead letter publishing recoverer. Many teams add intermediate retry topics with increasing delays before the final dead letter topic.

RabbitMQ

RabbitMQ uses dead letter exchanges (DLX), configured with the x-dead-letter-exchange queue argument or, preferably, a policy. A message is dead-lettered when a consumer rejects or nacks it with requeue=false, when its TTL expires, when the queue length limit is exceeded, or when a quorum queue's delivery limit is reached. Since RabbitMQ 4.0, quorum queues have a default delivery limit of 20, so messages that exceed it are dropped unless a dead letter target is configured. RabbitMQ records the history in the x-death header.

Why DLQs matter for reliability

• Isolation: one bad message does not stall a whole partition or consumer group.
• No silent data loss: failed orders, payments or events are preserved instead of dropped.
• Debuggability: the DLQ is a curated list of exactly the inputs your code cannot handle.
• Backpressure control: combined with exponential backoff between retries, a DLQ caps how much load failing messages generate.

Example: an SQS redrive policy

{
  "deadLetterTargetArn": "arn:aws:sqs:eu-central-1:123456789012:orders-dlq",
  "maxReceiveCount": "5"
}

In this illustrative setup each order message gets five attempts. With a visibility timeout of 60 seconds, a permanently failing message reaches orders-dlq after roughly five minutes, and the alarm on the DLQ fires instead of the consumer retrying for days.

Best practices for monitoring a DLQ

• Alert when the DLQ is not empty, or when its message count grows. A healthy DLQ is usually empty, so any queue depth above zero is a signal, and the age of the oldest message shows how long a problem has gone unnoticed.
• Attach context. Store the error message, stack trace reference, attempt count and trace ID with the message, for example in headers or attributes.
• Separate transient from permanent errors. Timeouts deserve retries, validation errors can go to the DLQ immediately.
• Have a replay procedure. Document how to inspect, fix and redrive messages, and make consumers idempotent so replays are safe.
• Set retention deliberately so messages do not expire from the DLQ before someone looks at them.

Common mistakes

• Creating a DLQ but never alerting on it, which turns it into a silent graveyard.
• A maxReceiveCount of 1, which sends messages to the DLQ on the first transient hiccup.
• Redriving everything without fixing the cause, so messages bounce straight back.
• Losing the link between a dead-lettered message and the trace that shows why it failed.

How Obseria helps

Obseria monitors queue and topic metrics from brokers such as Amazon SQS and Kafka alongside consumer traces, so a growing DLQ can be tied to the error and deployment that caused it. See queue monitoring and Kafka monitoring for consumer lag, throughput and dead letter topics in one view.