Kafka Performance Tuning: Producers, Consumers, and Brokers

Kafka Is Fast by Default — Tuning Is About Making Trade-offs

Out of the box, Kafka can handle hundreds of thousands of messages per second on modest hardware. It was designed from the ground up for high throughput: sequential disk writes, zero-copy data transfer, batching at every layer. Most applications will never hit Kafka’s limits with default settings.

So why does performance tuning matter? Because “fast” means different things in different contexts. A real-time fraud detection system needs every message delivered in under 5 milliseconds — it will sacrifice throughput for latency. A data pipeline ingesting clickstream events cares about processing 2 million events per second — it will sacrifice latency for throughput. A financial audit log needs every single message written to three replicas before acknowledging — it will sacrifice both speed metrics for durability.

Performance tuning in Kafka is not about making things faster in general. It is about understanding these trade-offs and dialing in the right balance for your specific use case. Every setting you change helps one dimension at the expense of another. The art is knowing which trade-offs matter for your system.

Partition Strategy: The Most Impactful Decision

Partitions are the fundamental unit of parallelism in Kafka. Each partition is an independent, ordered log that can be read by exactly one consumer within a consumer group. If you have 12 partitions and 12 consumers in a group, each consumer reads from one partition, giving you 12x parallel processing. If you have 12 partitions but only 3 consumers, each consumer reads from four partitions. If you have 12 partitions and 20 consumers, 8 consumers sit idle because there are no partitions left to assign.

This means the partition count directly controls how much parallelism your consumers can achieve. But more partitions are not always better, and this is where the trade-off thinking begins.

The Trade-off: More Parallelism vs More Overhead

Every partition has a cost. On the broker side, each partition requires open file handles (for the log segment and index files), memory buffers, and a replication thread. A broker with 10,000 partitions will use significantly more memory and file descriptors than one with 100 partitions. During leader election (when a broker fails), Kafka must elect a new leader for every partition that was led by the failed broker. With 5,000 partitions to reassign, this can take minutes, during which those partitions are unavailable.

On the producer side, each partition gets its own send buffer. More partitions means more memory allocated to buffering. On the consumer side, rebalancing (when consumers join or leave the group) takes longer with more partitions because each partition must be reassigned.

Here is a practical rule of thumb. A single partition typically handles 10-50 MB/s depending on message size, compression, and hardware. Work backward from your throughput needs:

• Low volume (under 10 MB/s): 6 partitions. This gives you room to scale consumers to 6 without repartitioning, while keeping overhead minimal.
• Medium volume (10-100 MB/s): 12-30 partitions. Enough parallelism for a small cluster of consumers, manageable overhead.
• High volume (100+ MB/s): 50-200 partitions. You need this level of parallelism, and you accept the operational overhead that comes with it.

One critical constraint: once created, partition counts can only increase, never decrease. Start conservative. It is easy to add partitions later, but you cannot remove them without recreating the topic and losing ordering guarantees for existing keys.

Producer Tuning: The Speed vs Safety Dial

Think of the Kafka producer as having a dial that goes from “fast but risky” on one end to “safe but slower” on the other. The default settings sit somewhere in the middle, and your job is to move the dial to match your application’s needs.

The Batching Trade-off

When you call producer.produce(), the message does not immediately go to the broker. Instead, it goes into an internal buffer. The producer accumulates messages and sends them as a batch. Larger batches mean fewer network round trips (better throughput) but higher latency for individual messages (they wait in the buffer). Smaller batches mean lower latency but more network overhead.

Two settings control this behavior. batch.size sets the maximum size of a batch in bytes. linger.ms sets how long the producer will wait to fill the batch before sending whatever it has. If linger.ms is 0, the producer sends immediately even if the batch is nearly empty. If linger.ms is 50, the producer waits up to 50 milliseconds to accumulate more messages, resulting in fuller batches and better throughput.

The Acknowledgment Trade-off

The acks setting controls how many brokers must confirm a write before the producer considers it successful. This is the most direct speed-vs-safety trade-off in Kafka:

• acks=0: The producer does not wait for any confirmation. Fire and forget. Maximum speed, but if the broker crashes, messages are silently lost.
• acks=1: The producer waits for the partition leader to confirm. The message is on one broker’s disk. If the leader crashes before replication, the message is lost.
• acks=all: The producer waits for all in-sync replicas to confirm. The message is on multiple brokers’ disks. No data loss unless multiple brokers fail simultaneously. This is the slowest but safest option.

Here is a high-throughput configuration for a scenario where you are ingesting web analytics events. Losing a few events is acceptable, but you need to handle 500,000 events per second:

from confluent_kafka import Producer

high_throughput = Producer({
    "bootstrap.servers": "kafka-1:9092,kafka-2:9092,kafka-3:9092",
    "batch.size": 65536,             # 64 KB batches (default is 16 KB)
    "linger.ms": 50,                 # Wait up to 50ms to fill batches
    "compression.type": "lz4",       # Compress batches for less network I/O
    "acks": "1",                     # Leader-only ack for speed
    "queue.buffering.max.messages": 100000,
})

And here is a low-latency configuration for a real-time alerting system where every millisecond matters and no message can be lost:

low_latency = Producer({
    "bootstrap.servers": "kafka-1:9092,kafka-2:9092,kafka-3:9092",
    "batch.size": 1,                 # Send immediately, no batching
    "linger.ms": 0,                  # Zero wait time
    "compression.type": "none",      # Skip compression to save CPU time
    "acks": "all",                   # Wait for all replicas
    "enable.idempotence": True,      # Prevent duplicates on retry
    "request.timeout.ms": 5000,      # Fail fast if broker is slow
})

Notice how every setting in the high-throughput config is the opposite of the low-latency config. Large batches vs tiny batches. Compression vs no compression. Relaxed acks vs strict acks. This is the trade-off dial in action.

Consumer Tuning: Sipping vs Gulping

Consumers also have a fundamental trade-off, but it is about how much data they fetch at a time. Think of it as the difference between sipping water from a glass and gulping it from a pitcher.

A “sipping” consumer fetches small amounts of data frequently. Each poll() call returns quickly with a few messages, processes them, and polls again. This gives low latency because each message is processed shortly after arriving. But it means more frequent network requests, which wastes network overhead.

A “gulping” consumer fetches large amounts of data in each request. It tells the broker, “do not bother responding until you have at least 1 MB of data for me, and I am willing to wait 500ms for that much to accumulate.” This is more efficient because fewer, larger network requests are cheaper than many small ones. But it means individual messages may wait longer before being fetched.

The key settings are fetch.min.bytes (the minimum amount of data the broker should accumulate before responding to a fetch request) and fetch.wait.max.ms (how long the broker will wait to reach that minimum). For a high-throughput analytics consumer that processes data in batch:

from confluent_kafka import Consumer

gulping_consumer = Consumer({
    "bootstrap.servers": "kafka-1:9092,kafka-2:9092,kafka-3:9092",
    "group.id": "analytics-consumer",
    "fetch.min.bytes": 1048576,          # Wait for 1 MB of data
    "fetch.wait.max.ms": 500,            # But no more than 500ms
    "max.partition.fetch.bytes": 1048576, # Up to 1 MB per partition
    "enable.auto.commit": True,
    "auto.commit.interval.ms": 5000,
    "session.timeout.ms": 45000,
    "heartbeat.interval.ms": 15000,
})

For CPU-intensive processing where each message takes significant time (like image analysis or ML inference), the bottleneck is not Kafka but your processing logic. In this case, decouple fetching from processing using a thread pool. The consumer thread fetches messages quickly while a pool of worker threads processes them in parallel:

import concurrent.futures
from confluent_kafka import Consumer

consumer = Consumer({
    "bootstrap.servers": "kafka-1:9092",
    "group.id": "parallel-processor",
    "enable.auto.commit": False,
    "max.poll.interval.ms": 600000,  # 10 min to allow slow processing
})
consumer.subscribe(["events"])

executor = concurrent.futures.ThreadPoolExecutor(max_workers=8)

def process_message(msg):
    """Your CPU-intensive processing logic."""
    data = msg.value().decode("utf-8")
    # ... heavy computation ...
    return msg

try:
    while True:
        messages = consumer.consume(num_messages=100, timeout=1.0)
        if not messages:
            continue

        futures = [executor.submit(process_message, msg)
                   for msg in messages if not msg.error()]
        concurrent.futures.wait(futures)

        # Commit only after all messages are successfully processed
        consumer.commit(asynchronous=False)
except KeyboardInterrupt:
    pass
finally:
    executor.shutdown(wait=True)
    consumer.close()

The critical detail here is enable.auto.commit=False combined with manual commit() after processing completes. If you used auto-commit, Kafka might mark messages as consumed before your thread pool finishes processing them. If the consumer crashes, those messages would be lost.

Broker Tuning: Where the Heavy Lifting Happens

Broker tuning is about matching Kafka’s internal thread pools and memory allocation to your hardware. The most important insight is that Kafka relies heavily on the operating system’s page cache for read performance, not the JVM heap. This is counterintuitive for Java developers who are used to tuning heap sizes.

When a producer writes a message, Kafka appends it to a log file on disk. The OS page cache automatically keeps recently written data in RAM. When a consumer reads recent messages, it reads directly from the page cache without touching the disk at all. This means the more RAM you leave for the OS (by giving less to the JVM), the more data can be served from cache.

A broker with 32 GB of RAM should give about 6 GB to the JVM heap and leave 26 GB for the page cache. If you give 24 GB to the JVM and leave 8 GB for the OS, your consumers will hit disk much more often, dramatically reducing read throughput.

# JVM settings: conservative heap, aggressive GC tuning
export KAFKA_HEAP_OPTS="-Xms6g -Xmx6g"
export KAFKA_JVM_PERFORMANCE_OPTS="-XX:+UseG1GC \
  -XX:MaxGCPauseMillis=20 \
  -XX:InitiatingHeapOccupancyPercent=35 \
  -XX:G1HeapRegionSize=16M"

For thread pool sizing, the rules are simple. Network threads handle reading requests from the socket and writing responses back. Set this to roughly one thread per 2 Gbps of network bandwidth. I/O threads handle the actual work of appending to log files and reading from disk. Set this to match the number of physical disks. If you have 8 SSDs in a JBOD configuration, set num.io.threads=8:

num.network.threads=8
num.io.threads=16
num.replica.fetchers=4

Replication: The Durability Trade-off

Replication settings control how Kafka balances data safety against availability. The golden rule is: set min.insync.replicas to replication.factor - 1. For a replication factor of 3, set min.insync.replicas=2. This means the producer (with acks=all) waits for 2 out of 3 replicas to acknowledge. One broker can fail without losing data or availability. If two brokers fail, writes will be rejected (preserving data integrity) rather than proceeding with only one copy.

min.insync.replicas=2
default.replication.factor=3
unclean.leader.election.enable=false
replica.lag.time.max.ms=30000

The unclean.leader.election.enable=false setting is critical for data safety. When set to true, Kafka can elect a replica that is behind (not fully caught up) as the new leader, which means some messages may be lost. Setting it to false means Kafka will wait for a fully caught-up replica, which may cause temporary unavailability but never data loss.

Compression: Shrinking Messages Before They Travel

Compression in Kafka works at the batch level, not the individual message level. The producer compresses an entire batch of messages, sends the compressed batch to the broker, and the broker stores it compressed on disk. The consumer fetches the compressed batch and decompresses it. This means compression saves network bandwidth (between producer and broker, between broker and consumer, and between brokers during replication) and disk space.

Think of it like shipping boxes. Without compression, you ship each item in its own oversized box with lots of packing peanuts. With compression, you carefully pack items tightly together, reducing the number of trips the delivery truck needs to make and the warehouse space required.

Different compression algorithms make different trade-offs between compression ratio (how small the data gets) and CPU cost (how much processing time it takes to compress and decompress):

Algorithm	Compression Ratio	CPU (Compress)	CPU (Decompress)	Best For
none	1.0x	-	-	Low latency, small messages
snappy	2.0-2.5x	Low	Very Low	General purpose, balanced
lz4	2.0-2.5x	Low	Very Low	High throughput (recommended)
zstd	2.5-3.5x	Medium	Low	Best ratio with acceptable CPU
gzip	2.5-3.0x	High	Medium	Maximum compression, batch jobs

For most workloads, lz4 is the right choice. It compresses data to roughly half its original size while adding negligible CPU overhead. The decompression is particularly fast, which matters because every consumer must decompress every batch it reads. Switch to zstd if network bandwidth is your bottleneck and you can afford higher CPU usage on producers. Avoid gzip in latency-sensitive paths because its compression step is significantly slower than the alternatives.

To put concrete numbers on this: a JSON message averaging 500 bytes will compress to roughly 200-250 bytes with lz4, saving about 50% of network and disk I/O. If you produce 1 million messages per second, that is 250 MB/s saved in network traffic, which at cloud networking prices adds up quickly.

Kafka vs Alternatives: Choosing the Right Tool

Kafka is not the only messaging system, and it is not always the best choice. Understanding where Kafka excels and where other tools are a better fit helps you make informed architecture decisions.

Kafka vs RabbitMQ is the most common comparison, and the two systems are designed for fundamentally different use cases. RabbitMQ is a traditional message broker built around the idea of message queues: a producer sends a message, a consumer picks it up, and the message is deleted. It excels at task distribution, request-reply patterns, and scenarios where messages are transient. RabbitMQ delivers sub-millisecond latency and has sophisticated routing with exchanges and bindings. But it does not retain messages after consumption, so you cannot replay history or reprocess data. Kafka, by contrast, is a distributed log. Messages are retained for days or weeks regardless of whether they have been consumed. Multiple consumers can read the same data independently, and you can replay from any point in time. This makes Kafka ideal for event streaming, audit trails, and data pipelines where downstream systems need to catch up or reprocess.

Kafka vs Apache Pulsar is a comparison of two systems built for similar goals but with different architectural philosophies. Pulsar separates compute (brokers) from storage (Apache BookKeeper), which enables features like infinite storage tiering and easier scaling. Pulsar also has built-in multi-tenancy and geo-replication that Kafka requires additional tooling to achieve. However, Pulsar’s operational complexity is significantly higher. Running BookKeeper clusters alongside Pulsar brokers means more moving parts to manage, monitor, and debug. If you need multi-tenant streaming with fine-grained isolation, Pulsar may be worth the complexity. For most single-tenant streaming use cases, Kafka’s simpler architecture is easier to operate.

Kafka vs Redis Streams is a comparison of a heavyweight and a lightweight option. Redis Streams provides basic stream processing with sub-millisecond latency and is trivial to set up if you already run Redis. But Redis Streams stores data in memory, which limits retention to what fits in RAM, and its consumer group implementation is simpler than Kafka’s. Redis Streams is excellent for lightweight streaming needs, ephemeral data, and use cases where you need blazing speed and already have Redis infrastructure. For anything requiring durable storage, high throughput, or sophisticated stream processing, Kafka is the better fit.

Real-World Architecture Patterns

Kafka is more than a message queue — it is an infrastructure primitive that enables several powerful architectural patterns. Each pattern solves a specific business problem, and understanding the problem is more important than memorizing the implementation.

Event Sourcing: Remembering Everything That Ever Happened

Traditional databases store the current state of things. An order’s status is “shipped” — that is all you know. But what if you need to know when it was created, when payment was confirmed, and when it was shipped? What if there is a dispute and you need to audit the full history? What if you discover a bug in your billing logic and need to recalculate charges for the last month?

Event sourcing solves this by storing every state change as an immutable event, rather than overwriting the current state. Kafka is a natural fit because it is already an immutable, append-only log. The topic becomes the source of truth, and the current state is just a derived view that you can rebuild at any time by replaying the events from the beginning.

# Every state change is recorded as a separate event
events = [
    {"type": "OrderCreated", "order_id": "ORD-001", "amount": 99.99},
    {"type": "PaymentReceived", "order_id": "ORD-001", "payment_id": "PAY-001"},
    {"type": "OrderShipped", "order_id": "ORD-001", "tracking": "TRK-123"},
]

# Current state is derived by replaying events in order
def rebuild_order_state(events):
    state = {}
    for event in events:
        if event["type"] == "OrderCreated":
            state = {"order_id": event["order_id"], "amount": event["amount"],
                     "status": "created"}
        elif event["type"] == "PaymentReceived":
            state["status"] = "paid"
        elif event["type"] == "OrderShipped":
            state["status"] = "shipped"
            state["tracking"] = event["tracking"]
    return state

CQRS: Separating Reads from Writes

Many applications have fundamentally different requirements for reading and writing data. A user placing an order needs a fast, validated write path. An analytics dashboard querying order trends needs a denormalized, indexed read path. Trying to serve both from a single database design leads to painful compromises — either writes are slow because of all the indexes, or reads are slow because the schema is normalized.

CQRS (Command Query Responsibility Segregation) solves this by separating the write model (optimized for accepting commands) from the read model (optimized for answering queries). Kafka sits in the middle as the event bus. Commands write to the command service, which validates and persists them, then publishes events to Kafka. One or more read model builders consume those events and project them into read-optimized data stores (Elasticsearch for search, a denormalized Postgres table for dashboards, a Redis cache for fast lookups).

Commands --> Command Service --> Kafka Topic --> Read Model Builder --> Read DB
                                            \-> Analytics Service
                                            \-> Notification Service

Change Data Capture: Turning Databases into Event Streams

Many organizations have valuable data locked inside databases that other systems need access to. The traditional approach is batch ETL: every night, extract data from the database, transform it, and load it into a data warehouse. But this means downstream systems are always working with stale data, sometimes up to 24 hours old.

Change Data Capture (CDC) solves this by streaming database changes in real-time. Tools like Debezium read the database’s internal change log (the write-ahead log in Postgres, the binlog in MySQL) and publish each insert, update, and delete as a Kafka message. Downstream systems consume these events and stay synchronized with the source database in near real-time. This is how modern data platforms keep data warehouses, search indexes, and caches up to date without batch jobs.

# Debezium connector captures every change to the orders table
curl -X POST http://localhost:8083/connectors -H "Content-Type: application/json" -d '{
  "name": "postgres-orders-cdc",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname": "postgres-host",
    "database.port": "5432",
    "database.user": "debezium",
    "database.password": "secret",
    "database.dbname": "orders_db",
    "table.include.list": "public.orders",
    "topic.prefix": "cdc",
    "plugin.name": "pgoutput"
  }
}'

The Saga Pattern: Coordinating Distributed Transactions

When a single business operation spans multiple microservices, you have a distributed transaction problem. Placing an order might require charging a credit card (payment service), reserving inventory (inventory service), and scheduling delivery (shipping service). If the payment succeeds but inventory reservation fails, you need to refund the payment. Traditional two-phase commit does not work well across microservices because it requires tight coupling and is fragile in the face of network failures.

The Saga pattern solves this by breaking the distributed transaction into a sequence of local transactions, each with a compensating action. Kafka is the coordination layer. An orchestrator service sends commands to each participating service via Kafka topics and listens for their responses. If any step fails, the orchestrator sends compensating commands to all previously completed steps, effectively “undoing” the transaction.

SAGA_STEPS = [
    {"command_topic": "payment.commands",
     "compensate_topic": "payment.compensate"},
    {"command_topic": "inventory.commands",
     "compensate_topic": "inventory.compensate"},
    {"command_topic": "shipping.commands",
     "compensate_topic": "shipping.compensate"},
]

def execute_saga(order_id, steps):
    completed = []
    for step in steps:
        producer.produce(step["command_topic"], key=order_id,
                        value=json.dumps({"order_id": order_id}).encode())
        producer.flush()

        reply = wait_for_reply(order_id)  # Listen for success/failure

        if reply["status"] == "success":
            completed.append(step)
        else:
            # Something failed -- undo everything we already did
            for completed_step in reversed(completed):
                producer.produce(completed_step["compensate_topic"],
                               key=order_id,
                               value=json.dumps({"order_id": order_id}).encode())
            producer.flush()
            return {"status": "failed", "failed_at": step["command_topic"]}

    return {"status": "completed"}

Dead Letter Queue: Handling Poison Messages

In any message processing system, some messages will fail to process. Maybe the message is malformed, maybe it references data that does not exist yet, or maybe a downstream service is temporarily unavailable. Without a strategy for handling these failures, a single bad message can block the entire partition — the consumer retries it forever, and all messages behind it are stuck.

A Dead Letter Queue (DLQ) solves this by routing persistently failing messages to a separate topic for later investigation. The consumer tries processing each message up to a configured number of times. If it still fails after all retries, it sends the message to the DLQ topic along with metadata about why it failed, then moves on to the next message. An operator can later inspect the DLQ, fix the underlying issue, and replay the failed messages.

from confluent_kafka import Consumer, Producer

consumer = Consumer({
    "bootstrap.servers": "localhost:9092",
    "group.id": "order-processor",
    "enable.auto.commit": False,
})
dlq_producer = Producer({"bootstrap.servers": "localhost:9092"})
consumer.subscribe(["orders"])

MAX_RETRIES = 3

try:
    while True:
        msg = consumer.poll(1.0)
        if msg is None or msg.error():
            continue

        retries = 0
        success = False
        last_error = None

        while retries < MAX_RETRIES:
            try:
                process_order(msg.value())
                success = True
                break
            except Exception as e:
                retries += 1
                last_error = e

        if not success:
            # Move to DLQ with diagnostic metadata
            headers = [
                ("original-topic", b"orders"),
                ("failure-reason", str(last_error).encode()),
                ("retry-count", str(MAX_RETRIES).encode()),
            ]
            dlq_producer.produce("orders.dlq", key=msg.key(),
                                value=msg.value(), headers=headers)
            dlq_producer.flush()

        consumer.commit(message=msg)
except KeyboardInterrupt:
    pass
finally:
    consumer.close()

Next Steps

Performance tuning is an iterative process. Start with default settings, measure your actual throughput and latency, identify the bottleneck, adjust one setting at a time, and measure again. Resist the urge to copy-paste a “high performance” configuration from the internet without understanding what each setting does to your specific workload.

The patterns in this article — event sourcing, CQRS, CDC, sagas, and DLQs — are not theoretical exercises. They are battle-tested solutions to real problems that emerge in every event-driven architecture. Start with the simplest pattern that solves your immediate problem, and add complexity only when the business requirements demand it.

• What is Apache Kafka? — revisit the fundamentals with fresh eyes
• Kafka Security: SSL and SASL — secure your tuned cluster before going to production