Queue monitoring for Kafka, RabbitMQ, SQS and Pub/Sub

What is queue monitoring?

Queue monitoring is the observation of message brokers and event streaming platforms, such as Apache Kafka, RabbitMQ, Amazon SQS and Google Cloud Pub/Sub, together with the producers and consumers that use them. It answers whether messages are flowing, how quickly they are processed and whether any part of the pipeline is falling behind. The central metrics are consumer lag, throughput, message age and error rates.

Queues and streams decouple services. A producer can publish an order event without waiting for billing, shipping and analytics to finish their work. That decoupling makes systems more resilient, but it also hides problems: when a consumer slows down, nothing fails immediately. Messages simply pile up until delays become visible to users or downstream systems.

Why queue monitoring matters

In event-driven architectures, many business processes run asynchronously: payments, notifications, inventory updates, data pipelines. A growing backlog in one consumer group can mean emails sent hours late, stock levels that are wrong or dashboards that show stale data. Because the producer keeps succeeding, classic request-based monitoring does not notice anything.

Diagnosing these issues is also harder than with synchronous calls. The cause of a backlog might be a slow database behind the consumer, a rebalance storm in Kafka, a poison message that is retried endlessly, or simply more traffic than the consumers can handle. Queue monitoring combines broker metrics with consumer-side traces so you can tell these cases apart quickly.

Key queue and streaming metrics

Metric	Why it matters
Consumer lag	The number of messages a consumer group is behind the latest offset; the primary signal for Kafka backlogs.
Age of oldest message	How long the oldest unprocessed message has been waiting; directly tied to user-visible delay.
Throughput in and out	Publish and consume rates; a widening gap means the backlog is growing.
Processing duration	Time consumers need per message or batch, measured from traces.
Errors and redeliveries	Failed processing, negative acknowledgements and retries that point to poison messages or failing dependencies.
Dead-letter queue depth	Messages that could not be processed and need attention.
Broker health	Under-replicated or offline partitions, disk and memory alarms, connection counts.

How Obseria handles queue monitoring

Obseria collects broker metrics from self-hosted Kafka and RabbitMQ clusters through the Obseria Agent or OpenTelemetry Collector receivers, including clusters running in Kubernetes. Amazon SQS, Google Cloud Pub/Sub and other managed messaging services are monitored agentless as part of Obseria's coverage of 300+ cloud services. All brokers end up in one data model, so dashboards and alerts look the same regardless of the technology.

On the application side, OpenTelemetry messaging instrumentation injects trace context into message headers. Obseria links publish and process spans, so a trace shows the HTTP request that produced an event, the time the message spent in the queue and the work the consumer did afterward. Obseria TopoGraph draws producers, topics, queues and consumers as a dependency map. When lag grows, Obseria Intelligence checks consumer errors, processing time, downstream dependencies and recent deployments, and returns a root cause with its chain of evidence.

You can query lag and throughput with OQL or PromQL:

fetch metrics
| filter messaging.system == "kafka"
| summarize max(kafka.consumer_group.lag), by:{consumer_group, topic}

Queue monitoring best practices

1. Alert on lag trends and message age. A stable lag is often fine; a steadily growing one is not. Age reflects user impact better than counts.
2. Define SLOs for asynchronous work. For example, 99% of orders processed within 60 seconds of being published.
3. Propagate trace context in headers. Without it, every queue becomes a blind spot in your traces.
4. Watch dead-letter queues. Treat growth as a signal, not as a place where failures go to be forgotten.
5. Correlate with consumer scaling. Autoscaling, rebalances and deployments often explain sudden lag changes.