Kafka Monitoring and Operations: A Practical Guide

Operating Kafka Is a Different Skill Than Using Kafka

Producing and consuming messages is the easy part. Keeping a Kafka cluster healthy, performant, and reliable in production is where the real work begins. A Kafka cluster in production is a living system — brokers join and leave, partitions rebalance, disks fill up, consumers fall behind. Without proper monitoring, you are flying blind.

Think of operating Kafka like maintaining an airplane. The passengers (your applications) only care that the flight is smooth. But the pilots (your operations team) need dozens of instruments showing altitude, fuel, engine temperature, and air speed. They need alarms that go off before something fails, not after. And they need to know which alarms mean “land immediately” versus “check this at the next maintenance window.”

This article covers the metrics that matter, the tools to collect them, and the alerting strategies that keep your team sane.

Broker Metrics: The Health of Your Cluster

Brokers are the heart of your Kafka cluster. When brokers are healthy, everything else works. When brokers struggle, everything suffers. These are the metrics every Kafka operator must watch.

Under-Replicated Partitions (URP)

This is the single most important broker metric. An under-replicated partition is one where the number of in-sync replicas (ISR) is less than the configured replication factor. If you have a replication factor of 3 and one replica falls behind, you have an under-replicated partition. If two replicas fall behind, you are one broker failure away from data loss.

A healthy cluster has zero under-replicated partitions at all times. If URPs spike, something is wrong. Common causes include:

• A broker is overloaded (CPU, disk, or network saturation)
• A broker is down or unreachable
• Network issues between brokers
• A disk is failing, causing slow writes on one broker

# Check under-replicated partitions via kafka-topics command
kafka-topics.sh --bootstrap-server kafka-1:9092 \
  --describe --under-replicated-partitions

Alert threshold: Page immediately if URPs are greater than zero for more than 5 minutes. This is a “wake someone up at 3 AM” alert.

Request Latency

Kafka brokers handle two primary request types: produce requests (writes) and fetch requests (reads). Monitoring the latency of both tells you how responsive your cluster is.

The key metrics are RequestLatencyMean and RequestLatencyP99 for both Produce and FetchConsumer request types. A healthy broker typically shows produce latency under 5ms and fetch latency under 10ms. If latency suddenly increases, it usually indicates disk I/O problems, network saturation, or an overloaded broker.

Metric	Healthy Range	Warning	Critical
Produce P99 latency	< 10ms	< 50ms	> 100ms
Fetch P99 latency	< 15ms	< 50ms	> 100ms
Under-replicated partitions	0	> 0 for 2min	> 0 for 5min
Active controller count	1 per cluster	0 or > 1	Persists > 1min
Offline partitions	0	> 0	Any

Disk Usage and Log Segment Health

Kafka stores all messages on disk. When a disk fills up, the broker stops accepting writes for partitions on that disk. This is a hard failure with no graceful degradation.

Monitor disk usage as a percentage and project when it will be full based on current ingestion rates. A topic with 7-day retention that ingests 100 GB per day needs at least 700 GB of disk space, plus headroom for compaction, segment rolling, and temporary spikes.

# Check disk usage on each broker
df -h /var/kafka-logs

# Check log directory sizes per topic
du -sh /var/kafka-logs/*/

Alert threshold: Ticket at 70% disk usage. Page at 85%. At 90%, you are one traffic spike away from an outage.

Active Controller Count

Every Kafka cluster has exactly one controller broker that manages partition leadership, broker registration, and administrative operations. If no broker is the controller, the cluster cannot perform leader elections and partitions become unavailable when brokers fail. If multiple brokers think they are the controller (a “split brain”), you have a serious metadata inconsistency.

Alert threshold: Page if the count is anything other than 1 for more than 60 seconds.

Producer Metrics: Are Your Writes Healthy?

Producer metrics tell you whether your applications are successfully writing data to Kafka. Problems here usually manifest as data delays or data loss.

Record Send Rate and Error Rate

The record-send-rate metric shows how many records per second the producer is sending. The record-error-rate shows how many sends are failing. In a healthy producer, the error rate should be zero or near zero. If errors spike, check broker availability, network connectivity, and whether the producer is hitting serialization errors or authorization failures.

from confluent_kafka import Producer

def delivery_callback(err, msg):
    if err:
        print(f"Delivery failed: {err}")
        # Increment your error counter metric here
    else:
        print(f"Delivered to {msg.topic()} [{msg.partition()}] @ {msg.offset()}")
        # Increment your success counter metric here

producer = Producer({
    "bootstrap.servers": "kafka-1:9092",
    "statistics.interval.ms": 5000,  # Emit internal stats every 5 seconds
})

# Use delivery callbacks to track success/failure rates
producer.produce("events", value=b"test", callback=delivery_callback)
producer.flush()

Batch Size and Compression Ratio

If your producer’s average batch size is much smaller than batch.size, messages are being sent before the batch fills up. This means linger.ms is too low or message volume is too low to fill batches. Small batches waste network overhead.

The compression ratio shows how effectively your messages are being compressed. A ratio of 0.5 means data is compressed to half its original size. If compression ratio is close to 1.0, either compression is disabled or your messages are not compressible (already compressed data, random bytes, encrypted payloads).

Consumer Metrics: The Most Important Section

Consumer metrics are where most operational problems surface. A healthy producer and a healthy broker do not help if consumers cannot keep up with the data being produced.

Consumer Lag: The One Metric to Rule Them All

Consumer lag is the difference between the latest offset in a partition (the log-end offset) and the last committed offset for a consumer group. It tells you how far behind a consumer is from the latest data.

Think of consumer lag like a line at a grocery store. The log-end offset is the last person who joined the line. The consumer’s committed offset is the person currently being served. Lag is the number of people waiting. If the line grows faster than the cashier can serve customers, the wait time increases indefinitely. If the cashier is faster than new arrivals, the line shrinks to zero.

A consumer with zero lag is fully caught up — every message is processed shortly after it is produced. A consumer with growing lag is falling behind. Left unchecked, growing lag means data is getting staler and staler, and eventually the consumer might fall so far behind that messages are deleted by retention before being consumed.

# Check consumer lag for a specific consumer group
kafka-consumer-groups.sh --bootstrap-server kafka-1:9092 \
  --describe --group my-consumer-group

# Output shows lag per partition:
# TOPIC     PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG
# orders    0          1000            1050             50
# orders    1          2000            2000             0
# orders    2          1500            1800             300

Alert threshold: This depends entirely on your use case. A real-time fraud detection system should page if lag exceeds 1000 messages (seconds of delay). A nightly batch analytics consumer might tolerate lag of millions of messages during the day as long as it catches up overnight. Define your SLA and alert accordingly.

Common Causes of Consumer Lag

When lag grows, the question is: why is the consumer not keeping up? Here are the usual suspects:

Slow processing logic. The consumer spends too long processing each message. If processing takes 100ms per message and you receive 100 messages per second, a single consumer cannot keep up. Solutions: optimize processing logic, increase partitions and consumers for parallelism, or move expensive processing to an async worker pool.

Frequent rebalances. When consumers join or leave a group, Kafka reassigns partitions. During a rebalance, all consumers in the group stop processing. If rebalances happen frequently (due to consumers crashing, deploying, or exceeding max.poll.interval.ms), effective throughput drops.

Insufficient consumers. If you have 12 partitions and only 2 consumers, each consumer reads from 6 partitions. Adding more consumers (up to 12) distributes the load.

GC pauses or resource contention. Long garbage collection pauses can cause a consumer to miss heartbeats, triggering a rebalance and further increasing lag.

Tools: Building Your Monitoring Stack

JMX: Kafka’s Native Metric Source

Kafka exposes all its internal metrics through JMX (Java Management Extensions). Every broker, producer, and consumer metric mentioned in this article is available as a JMX MBean. To access JMX metrics, you need to enable the JMX port when starting the broker:

# Enable JMX on broker startup
export JMX_PORT=9999
export KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote \
  -Dcom.sun.management.jmxremote.authenticate=false \
  -Dcom.sun.management.jmxremote.ssl=false \
  -Dcom.sun.management.jmxremote.port=9999"

# Start the broker with JMX enabled
bin/kafka-server-start.sh config/server.properties

Prometheus + Grafana: The Industry Standard

The most common production setup is to export JMX metrics to Prometheus using the JMX Exporter, then visualize them in Grafana dashboards.

The JMX Exporter runs as a Java agent alongside each broker, converting JMX MBeans into Prometheus-compatible metrics on an HTTP endpoint. Prometheus scrapes these endpoints at regular intervals and stores the time-series data. Grafana connects to Prometheus and provides dashboards and alerting.

# prometheus-jmx-exporter-config.yml
# Mount this config alongside the JMX exporter agent
rules:
  # Broker: under-replicated partitions
  - pattern: "kafka.server<type=ReplicaManager, name=UnderReplicatedPartitions><>Value"
    name: kafka_server_under_replicated_partitions
    type: GAUGE

  # Broker: active controller count
  - pattern: "kafka.controller<type=KafkaController, name=ActiveControllerCount><>Value"
    name: kafka_controller_active_controller_count
    type: GAUGE

  # Broker: request latency
  - pattern: "kafka.network<type=RequestMetrics, name=RequestsPerSec, request=(.+), version=(.+)><>Count"
    name: kafka_network_requests_per_sec
    type: COUNTER
    labels:
      request: "$1"
      version: "$2"

  # Broker: bytes in/out per second
  - pattern: "kafka.server<type=BrokerTopicMetrics, name=(.+)PerSec, topic=(.+)><>Count"
    name: kafka_server_broker_topic_metrics_$1_per_sec
    type: COUNTER
    labels:
      topic: "$2"

  # Consumer: lag (via consumer group metrics)
  - pattern: "kafka.server<type=FetcherLagMetrics, name=ConsumerLag, clientId=(.+), topic=(.+), partition=(.+)><>Value"
    name: kafka_consumer_lag
    type: GAUGE
    labels:
      client_id: "$1"
      topic: "$2"
      partition: "$3"

# Start broker with JMX exporter as a Java agent
export KAFKA_OPTS="-javaagent:/opt/jmx-exporter/jmx_prometheus_javaagent.jar=7071:/opt/jmx-exporter/config.yml"
bin/kafka-server-start.sh config/server.properties

Burrow: Purpose-Built Consumer Lag Monitoring

Burrow, developed by LinkedIn, is a dedicated consumer lag monitoring tool. Unlike simple lag checks that only report the current lag number, Burrow evaluates the trend of consumer lag over time. It classifies consumer groups into states:

• OK: Lag is stable or decreasing.
• WARNING: Lag is increasing but offsets are still moving (consumer is working but falling behind).
• ERROR: Offsets are not moving at all (consumer is stuck or dead).

This is a critical distinction. A consumer with lag of 50,000 that is steadily decreasing is recovering from a temporary spike and does not need attention. A consumer with lag of only 100 where offsets have not moved in 5 minutes has a real problem. Burrow catches the second case even though the absolute lag number is small.

Confluent Control Center and Kafka UI

For teams that prefer a graphical interface, Confluent Control Center provides a comprehensive dashboard for monitoring brokers, topics, consumer groups, and schema registries. It is a commercial product included with Confluent Platform.

For open-source alternatives, tools like Kafka UI (formerly known as Kafka-UI by Provectus) provide a web interface for browsing topics, viewing consumer groups, and monitoring basic cluster health. These are not replacements for a full Prometheus + Grafana stack, but they are invaluable for ad-hoc debugging and exploration.

Alerting Strategies: What to Page On vs What to Ticket

Not every metric anomaly deserves a 3 AM phone call. Effective alerting requires classifying issues by severity and urgency. Over-alerting leads to alert fatigue, where operators start ignoring pages because most of them are noise. Under-alerting means real problems go unnoticed until they cause an outage.

Page-Worthy Alerts (Wake Someone Up)

These indicate an active incident or imminent data loss:

• Under-replicated partitions > 0 for more than 5 minutes. Data durability is at risk.
• Offline partitions > 0. Some data is completely unavailable.
• Active controller count is not 1. The cluster cannot perform leader elections.
• Broker process down. A broker has stopped responding.
• Consumer lag growing exponentially. A critical consumer is falling behind with no sign of recovery.
• Disk usage > 90% on any broker. The broker will stop accepting writes when the disk is full.

Ticket-Worthy Alerts (Fix During Business Hours)

These indicate problems that need attention but are not emergencies:

• Disk usage > 70%. Plan capacity expansion.
• Consumer lag above SLA threshold. The consumer is behind but not critically.
• Producer error rate > 1%. Some messages are failing, but the producer is retrying.
• Rebalance frequency > 3 per hour. Something is causing frequent consumer group instability.
• ISR shrink rate elevated. Replicas are falling in and out of sync more often than normal.

# Example Prometheus alerting rules
groups:
  - name: kafka_critical
    rules:
      - alert: KafkaUnderReplicatedPartitions
        expr: kafka_server_under_replicated_partitions > 0
        for: 5m
        labels:
          severity: page
        annotations:
          summary: "Under-replicated partitions detected"
          description: "Broker {{ $labels.instance }} has {{ $value }} URPs"

      - alert: KafkaOfflinePartitions
        expr: kafka_controller_offline_partitions_count > 0
        for: 1m
        labels:
          severity: page
        annotations:
          summary: "Offline partitions detected -- data unavailable"

  - name: kafka_warning
    rules:
      - alert: KafkaDiskUsageHigh
        expr: (node_filesystem_size_bytes{mountpoint="/var/kafka-logs"} -
               node_filesystem_free_bytes{mountpoint="/var/kafka-logs"})
               / node_filesystem_size_bytes{mountpoint="/var/kafka-logs"} > 0.7
        for: 15m
        labels:
          severity: ticket
        annotations:
          summary: "Kafka disk usage above 70%"

      - alert: KafkaConsumerLagGrowing
        expr: delta(kafka_consumer_lag[10m]) > 10000
        for: 15m
        labels:
          severity: ticket
        annotations:
          summary: "Consumer lag is growing steadily"

Capacity Planning: When to Add Brokers

Adding brokers to a Kafka cluster is a significant operational event. Partitions must be reassigned, data must be rebalanced, and the cluster runs in a degraded state during the migration. You do not want to do it reactively during an outage. Proactive capacity planning prevents this.

Key Signals That You Need More Brokers

Network throughput saturation. If brokers consistently use more than 70% of available network bandwidth, adding brokers spreads the load. Remember that Kafka’s network traffic includes producer writes, consumer reads, and inter-broker replication. A broker with a replication factor of 3 uses 3x the producer write bandwidth for replication.

Disk I/O saturation. If disk write latency is increasing or I/O utilization is consistently above 80%, the broker is bottlenecked on storage. Before adding brokers, consider upgrading to faster disks (SSDs if still on HDDs) or adding more disks in JBOD configuration.

CPU saturation. This is less common because Kafka is not CPU-intensive, but if you use heavy compression (zstd or gzip) or have SSL/TLS enabled, CPU can become the bottleneck.

Partition count per broker exceeding 4,000. While Kafka can technically handle more, broker startup time, leader election time, and memory usage increase with partition count. Spreading partitions across more brokers keeps each broker manageable.

The Math: A Simple Capacity Model

A simple capacity model considers three resources: network, disk throughput, and disk space.

Incoming data rate: 500 MB/s
Replication factor: 3
Total write throughput: 500 * 3 = 1,500 MB/s (producers + replication)
Consumer read throughput: 500 MB/s (assuming consumers read everything)
Total throughput per broker cluster: 2,000 MB/s

If each broker can handle 200 MB/s of sustained throughput:
Minimum brokers needed: 2,000 / 200 = 10 brokers

Add 30% headroom for spikes and rolling upgrades:
Recommended brokers: 13 brokers

Retention: 7 days
Daily data: 500 MB/s * 86400 = 43 TB * 3 replicas = 129 TB
Per-broker storage: 129 TB / 13 brokers = ~10 TB each

This is a back-of-envelope calculation, but it gives you a starting point. Refine it with actual measurements from your production cluster as you gather operational experience.

Next Steps

Monitoring is not a one-time setup. As your cluster grows, your monitoring needs evolve. Start with the basics — consumer lag, under-replicated partitions, and disk usage — and expand your dashboards as you learn what questions your team asks most often during incidents.

• Kafka Performance Tuning — optimize the settings your monitoring reveals as bottlenecks
• Building Data Pipelines — monitor the health of end-to-end data pipelines through Kafka