Apache Kafka monitoring: brokers, partitions and consumer lag
See consumer lag per group and partition, broker and replication health and the services behind every topic. Obseria collects Kafka metrics via OpenTelemetry and links them to traces.
Why monitor Apache Kafka with Obseria
Consumer lag you can act on
Lag per consumer group, topic and partition, linked to the consuming service in TopoGraph, so you know who is falling behind and why.
Replication health at a glance
In-sync replicas, under-replicated and offline partitions per broker reveal durability risks before a broker failure turns into data loss.
End-to-end message traces
OpenTelemetry context in Kafka headers connects producer and consumer spans, so a slow checkout shows the exact topic it waited on.
Kafka components Obseria monitors
- Brokers. Broker count, request latency and JMX metrics such as UnderReplicatedPartitions and ActiveControllerCount.
- Topics and partitions. Partitions per topic, current and oldest offsets, replicas and in-sync replicas per partition.
- Consumer groups. Members, committed offsets and lag per group, topic and partition, plus lag summed per topic.
- Producers and consumers. Client spans and metrics from the OpenTelemetry Java agent and other SDKs with Kafka instrumentation.
- Kafka Connect and Streams. JVM and JMX metrics of Connect workers and Streams applications through the Obseria Agent or Java agent.
- Managed Kafka. Amazon MSK and other managed clusters through the kafka_metrics receiver and agentless cloud integrations.
The Apache Kafka signals that matter
Obseria collects these values automatically and shows them in ready-made dashboards and alerts.Processing backlog
Shows how far your consuming applications are behind incoming data, a direct indicator of delayed orders or updates.
Consumer activity
Tells you how many processing instances are active and whether they keep moving forward through the data.
Data safety
Reveals whether every piece of data is safely copied to several servers, so nothing is lost if one fails.
Cluster health and growth
Shows how many servers are online, whether the cluster is properly coordinated and how data volume is growing.
Show all 10 metrics in detail
| Metric | Unit | What it tells you |
|---|---|---|
kafka.consumer_group.lag | 1 | Messages a consumer group is behind the latest offset on one partition. |
kafka.consumer_group.lag_sum | 1 | Total lag of a consumer group across all partitions of a topic. |
kafka.consumer_group.members | {members} | Active members per group; drops point to crashed consumers or rebalances. |
kafka.consumer_group.offset | 1 | Committed offset per partition; a flat line with growing lag means a stuck consumer. |
kafka.partition.current_offset | 1 | Latest offset per partition, the basis for producer throughput per topic. |
kafka.partition.replicas_in_sync | {replicas} | In-sync replicas per partition; fewer than the replication factor means risk. |
kafka.brokers | {brokers} | Number of brokers in the cluster as seen by the client. |
kafka.topic.partitions | {partitions} | Partitions per topic, which caps consumer parallelism. |
UnderReplicatedPartitions | {partitions} | JMX gauge (kafka.server:type=ReplicaManager) of partitions missing replicas; should be 0. |
ActiveControllerCount | 1 | JMX gauge (kafka.controller:type=KafkaController); exactly one broker should report 1. |
Set up Apache Kafka monitoring in minutes
- Step 1
Point the Collector at your brokers
Add the kafka_metrics receiver with your bootstrap brokers and enable the brokers, topics and consumers scrapers.
- Step 2
Add broker JMX metrics
Install the Obseria Agent on broker hosts or attach the OpenTelemetry Java agent to collect JVM and Kafka JMX metrics.
- Step 3
Instrument producers and consumers
Use OpenTelemetry SDKs or the Java agent in your services so spans carry trace context through Kafka headers.
receivers:
kafka_metrics:
cluster_alias: kafka-prod
brokers: ["kafka-0:9092", "kafka-1:9092", "kafka-2:9092"]
protocol_version: 3.0.0
scrapers: [brokers, topics, consumers]
collection_interval: 30s
exporters:
otlphttp:
endpoint: ${env:OBSERIA_OTLP_ENDPOINT}
headers:
Authorization: "Bearer ${env:OBSERIA_TOKEN}"
service:
pipelines:
metrics:
receivers: [kafka_metrics]
exporters: [otlphttp]
Out of the box
Dashboards, alerts and SLOs that are ready the moment data arrives.
- Consumer lag by group, topic and partition
- Broker and replication health overview
- Topic throughput from partition offsets
- Alert: consumer lag growing for 10 minutes
- Alert: under-replicated or offline partitions above 0
Apache Kafka monitoring: what matters
Why Kafka monitoring matters
Apache Kafka sits between the services that produce events and the services that act on them. When Kafka is healthy, nobody notices it. When a broker loses its replicas or a consumer group falls behind, orders are processed late, fraud checks miss their window and dashboards show stale data. Because producers keep succeeding while consumers lag, these problems rarely surface as errors. Kafka monitoring has to make them visible as numbers: lag, replication state and throughput.
Obseria collects Kafka metrics with the OpenTelemetry Collector and the Obseria Agent, links each consumer group to the service that runs it, and follows messages through traces from producer to consumer.
The Kafka metrics that matter
Consumer lag
Kafka consumer lag is the difference between the latest offset in a partition and the offset a consumer group has committed. kafka.consumer_group.lag shows it per partition, kafka.consumer_group.lag_sum per topic. Absolute lag alone is misleading: 50,000 messages can be seconds of work for one service and hours for another. Watch the trend, and alert when lag grows steadily over several minutes.
Broker and replication health
Every broker exposes JMX metrics. UnderReplicatedPartitions should be 0 on every broker, OfflinePartitionsCount should be 0 on the controller, and exactly one broker should report ActiveControllerCount as 1. From the client side, kafka.partition.replicas_in_sync compared with kafka.partition.replicas shows which partitions are at risk.
Throughput and partitions
The rate of kafka.partition.current_offset is the produce rate per partition. Uneven rates across partitions of one topic point to a skewed message key. kafka.topic.partitions limits how many consumers in a group can work in parallel.
Common Kafka problems and how to detect them
- Stuck consumer: committed offset stays flat while lag grows. Often a poison message or a blocked downstream call; the consumer span in the trace shows which.
- Rebalance storms:
kafka.consumer_group.memberskeeps changing and lag jumps after each change. Typical causes are long processing times that exceedmax.poll.interval.msor frequent pod restarts. - Hot partitions: one partition carries most of the traffic and its consumer lags while others idle.
- Replication falling behind: under-replicated partitions after a broker restart that do not recover point to disk or network saturation on the follower.
- Retention surprises: if lag approaches the oldest offset (
kafka.partition.oldest_offset), messages are deleted before they are consumed.
Best practices for Kafka monitoring
- Alert on lag growth and time to catch up, not on a fixed lag number.
- Keep replication factor 3 and
min.insync.replicas2 for critical topics, and alert when in-sync replicas drop below that. - Monitor clients as well as brokers: producer error rates and consumer processing time explain most lag.
- Propagate OpenTelemetry trace context in message headers so asynchronous flows stay traceable.
- Tag metrics with a cluster alias so multiple clusters stay apart in dashboards.
How Obseria helps with Kafka monitoring
Obseria brings broker, topic and consumer metrics, JVM metrics and client traces into one place. TopoGraph shows which services produce to and consume from each topic, so a lag alert names the owning team. Obseria Intelligence correlates lag with deployments, consumer errors and downstream latency, and explains the likely root cause with a chain of evidence. The queue monitoring views, dashboards and alerts are included, and you can query everything with OQL, PromQL or SQL.
A practical starting point is a single Collector per cluster with the kafka_metrics receiver, plus the Obseria Agent on each broker host for JMX and host metrics such as disk usage and network throughput. Once lag and replication dashboards are in place, instrument the two or three most critical producer and consumer services with OpenTelemetry. From then on, every lag alert comes with the traces of the messages that were waiting and the deployment history of the consuming service, which shortens the path from alert to fix considerably.
Related integrations
Related capabilities
Queue Monitoring
Watch consumer lag, throughput and message age across every broker and topic, and trace messages from producer to consumer to find where processing stalls.
Learn moreDistributed Tracing
Follow each request across services, queues and databases with OpenTelemetry-native distributed tracing, and find the slow span or failing dependency in seconds.
Learn moreAnomaly Detection
Obseria Intelligence learns the normal rhythm of every service, flags real deviations instead of threshold noise and groups related anomalies into one problem with a probable root cause.
Learn moreLearn more
The four golden signals: latency, traffic, errors and saturation
The four golden signals explained: latency, traffic, errors and saturation, how they relate to the RED and USE methods, and how to measure them with OTel.
What is root cause analysis (RCA)?
What is root cause analysis? The RCA process for software incidents, 5 Whys, fishbone and fault tree methods, a worked example and how AI speeds up RCA.
Apache Kafka monitoring FAQ
The kafka_metrics receiver of the OpenTelemetry Collector reads the latest partition offsets and the committed offsets of each consumer group and reports the difference as kafka.consumer_group.lag per partition and kafka.consumer_group.lag_sum per topic.
Consumer lag and partition metrics work without JMX. For broker internals such as UnderReplicatedPartitions or request latency, install the Obseria Agent on the broker hosts or attach the OpenTelemetry Java agent.
Yes. OpenTelemetry instrumentation for Kafka clients writes trace context into message headers, so producer and consumer spans appear in the same trace in Obseria.
Yes. Point the kafka_metrics receiver at the MSK bootstrap brokers with TLS or SASL settings, and use the agentless AWS integration for the service metrics AWS publishes.
Start with lag growing for 10 minutes per consumer group, under-replicated partitions above 0, offline partitions above 0, and an active controller count other than 1 across the cluster.
Start monitoring Apache Kafka today
Connect Apache Kafka in minutes with OpenTelemetry or the Obseria Agent. Free for 14 days.