KRaft Mode and Kafka Cluster Management

The ZooKeeper Problem: Why Kafka Is Removing It

For over a decade, every Kafka cluster required a separate ZooKeeper ensemble to function. ZooKeeper stored Kafka’s metadata: which brokers are alive, which broker leads each partition, what topics exist, and what their configurations are. Kafka could not start, elect leaders, or perform any administrative operation without ZooKeeper.

This was a pragmatic choice when Kafka was first built at LinkedIn in 2010. ZooKeeper was an established, battle-tested coordination service. Building a custom consensus layer from scratch would have delayed Kafka’s development significantly. So the Kafka team stood on ZooKeeper’s shoulders and focused on building the log-based messaging system that made Kafka unique.

But over time, the ZooKeeper dependency became Kafka’s biggest operational burden. Here is why.

Two Systems to Operate Instead of One

Running Kafka in production means running two distributed systems: the Kafka cluster and the ZooKeeper ensemble. Each has its own deployment, monitoring, scaling, and failure modes. ZooKeeper needs its own servers (typically 3 or 5 nodes), its own disk I/O tuning, its own JVM configuration, and its own alerting. When troubleshooting a Kafka issue, operators often have to investigate ZooKeeper as well — is the problem in Kafka or in ZooKeeper? Is the connection between them healthy?

This doubles the operational surface area. It is like maintaining two separate engines in a car. Even if each engine is reliable, the coupling between them introduces failure modes that neither would have alone.

Scalability Limits

ZooKeeper stores all metadata in memory and replicates it synchronously across all nodes. As Kafka clusters grow to hundreds of brokers and hundreds of thousands of partitions, the volume of metadata grows proportionally. ZooKeeper was not designed for this scale. Large Kafka deployments hit ZooKeeper session timeouts, slow controller failovers, and metadata update bottlenecks that are difficult to resolve without fundamental architectural changes.

The Controller Bottleneck

In the ZooKeeper-based architecture, one Kafka broker is elected as the controller. The controller is responsible for all administrative operations: partition leader election, ISR changes, topic creation and deletion. The controller reads metadata from ZooKeeper, makes decisions, and pushes updates to all brokers. If the controller fails, a new controller must be elected and must reload all metadata from ZooKeeper before it can start operating. For a cluster with 200,000 partitions, this metadata reload can take minutes, during which no leader elections can happen and affected partitions are unavailable.

KRaft: Kafka Without ZooKeeper

KRaft (Kafka Raft) replaces ZooKeeper with a built-in consensus protocol based on Raft. All metadata is stored inside Kafka itself, in a special internal topic called the metadata log. There is no external dependency. Kafka is now a self-contained system.

The name “KRaft” is a portmanteau of “Kafka” and “Raft.” Raft is a consensus protocol designed to be understandable (unlike its predecessor Paxos, which is notoriously difficult to implement correctly). KRaft adapts Raft to Kafka’s specific needs, particularly the event-driven, log-based nature of Kafka’s architecture.

How KRaft Works: The Metadata Log

In KRaft mode, cluster metadata is stored as a sequence of records in a replicated log — the same pattern Kafka uses for everything else. Instead of ZooKeeper storing metadata as a tree of znodes, KRaft stores metadata as an ordered sequence of events: “Topic X was created with 12 partitions,” “Broker 3 joined the cluster,” “Partition 5 of topic Y changed leader to broker 2.”

This is a natural fit for Kafka because Kafka already knows how to replicate logs efficiently. The metadata log uses the same replication protocol as any other Kafka topic, just with the Raft consensus protocol layered on top to guarantee consistency.

Every broker in the cluster maintains a local copy of the metadata log. When a metadata change occurs, the active controller writes it to the metadata log, and all other brokers apply the change from their local copy. This means metadata is always available locally — a broker never needs to make a network call to look up which partitions it leads or what topics exist.

KRaft Architecture: Controllers and Brokers

In KRaft mode, nodes are configured as one of three roles:

• Controller nodes: Participate in the Raft quorum, store and replicate the metadata log, and handle all metadata operations (topic creation, partition reassignment, broker registration). Typically 3 or 5 nodes for fault tolerance.
• Broker nodes: Serve client requests (produce, fetch), store partition data, and follow the metadata log. These are the same brokers you are used to, but they no longer talk to ZooKeeper.
• Combined nodes: Act as both controller and broker. This is convenient for small clusters but not recommended for large production deployments because the controller workload can interfere with broker performance.

# Controller node configuration (server.properties)
process.roles=controller
node.id=1
controller.quorum.voters=1@controller-1:9093,2@controller-2:9093,3@controller-3:9093
controller.listener.names=CONTROLLER
listeners=CONTROLLER://controller-1:9093
log.dirs=/var/kafka-metadata

# Broker node configuration (server.properties)
process.roles=broker
node.id=101
controller.quorum.voters=1@controller-1:9093,2@controller-2:9093,3@controller-3:9093
listeners=PLAINTEXT://broker-1:9092
advertised.listeners=PLAINTEXT://broker-1:9092
log.dirs=/var/kafka-logs

Leader Election in KRaft

When the active controller fails, the remaining controllers elect a new leader using the Raft protocol. This election is fast — typically under 5 seconds — because:

1. Every controller already has a complete copy of the metadata log. There is no need to reload metadata from an external store.
2. Raft elections have a well-defined protocol: candidates request votes, the first to receive a majority wins. There is no ambiguity.
3. The new leader can start serving immediately because its metadata is already up to date.

Compare this to ZooKeeper-based controller failover, which required the new controller to:

1. Detect that the old controller is gone (session timeout, often 18 seconds).
2. Compete with other brokers to create an ephemeral znode in ZooKeeper.
3. Read all metadata from ZooKeeper (potentially hundreds of thousands of partition records).
4. Rebuild in-memory state from the metadata.
5. Start processing the backlog of state changes that accumulated during the failover.

For large clusters, the ZooKeeper-based failover could take several minutes. KRaft reduces this to seconds.

Migrating from ZooKeeper to KRaft

Kafka provides a migration path from ZooKeeper-based clusters to KRaft. This is a multi-step process that can be performed with zero downtime if done carefully.

The Migration Process

The migration works by running both ZooKeeper and KRaft controllers simultaneously during a transition period. Here is the high-level sequence:

1. Deploy KRaft controllers alongside your existing ZooKeeper-based cluster. The controllers connect to ZooKeeper and synchronize all metadata into the KRaft metadata log.

2. Migrate brokers one at a time. Update each broker’s configuration to use KRaft mode and perform a rolling restart. The broker stops communicating with ZooKeeper and starts following the KRaft metadata log instead.

3. Verify the migration. Confirm that all brokers are running in KRaft mode and that the cluster is healthy. Check that all topics, partitions, and consumer groups are intact.

4. Decommission ZooKeeper. Once all brokers are migrated and the cluster is stable, shut down the ZooKeeper ensemble.

# Step 1: Format storage for KRaft controllers
kafka-storage.sh format \
  --config /etc/kafka/kraft/controller.properties \
  --cluster-id $(kafka-storage.sh random-uuid)

# Step 2: Start KRaft controllers in migration mode
# (controllers connect to ZooKeeper to sync metadata)
# In controller.properties, add:
# zookeeper.connect=zk-1:2181,zk-2:2181,zk-3:2181
# zookeeper.metadata.migration.enable=true

# Step 3: Rolling restart of brokers with KRaft config
# Update each broker's server.properties:
# process.roles=broker
# controller.quorum.voters=1@ctrl-1:9093,2@ctrl-2:9093,3@ctrl-3:9093
# zookeeper.metadata.migration.enable=true

# Step 4: After all brokers migrated, disable migration mode
# Remove zookeeper.* settings from all nodes and restart

# Step 5: Decommission ZooKeeper

Migration Risks and Mitigations

The biggest risk is that the metadata synchronization between ZooKeeper and KRaft falls out of sync during the migration window. To mitigate this:

• Avoid creating or deleting topics during the migration. Metadata changes during migration add complexity.
• Migrate during a low-traffic period. Less activity means fewer metadata changes to synchronize.
• Have a rollback plan. If something goes wrong, you can revert brokers to ZooKeeper mode and shut down the KRaft controllers.
• Test the migration in a staging environment first. Run your full test suite against the migrated cluster before attempting production.

Cluster Management: Day-to-Day Operations

Adding Brokers

Adding a new broker to a KRaft cluster is straightforward. Configure the broker with the correct controller.quorum.voters and a unique node.id, then start it. The broker registers itself with the controller, and the controller begins assigning partitions to it during future topic creation.

However, existing partitions are not automatically moved to the new broker. You must explicitly reassign partitions to balance the load. This is intentional — automatic rebalancing of existing data could cause massive data movement that overwhelms the network.

# Generate a reassignment plan
kafka-reassign-partitions.sh --bootstrap-server kafka-1:9092 \
  --broker-list "101,102,103,104" \
  --topics-to-move-json-file topics.json \
  --generate

# topics.json contains:
# {"topics": [{"topic": "orders"}, {"topic": "events"}], "version": 1}

# Review the generated plan, then execute:
kafka-reassign-partitions.sh --bootstrap-server kafka-1:9092 \
  --reassignment-json-file plan.json \
  --execute

# Monitor progress:
kafka-reassign-partitions.sh --bootstrap-server kafka-1:9092 \
  --reassignment-json-file plan.json \
  --verify

Removing Brokers (Decommissioning)

Removing a broker requires moving all its partitions to other brokers first. This is a data-intensive operation — every partition replica on the decommissioned broker must be copied to another broker over the network. For a broker with 10 TB of data, this can take hours.

The process is:

1. Generate a reassignment plan that excludes the broker being removed.
2. Execute the plan and monitor until all partitions are moved.
3. Verify that the broker has zero partitions assigned.
4. Shut down the broker and remove it from the cluster.

Critical rule: Never shut down a broker before its partitions are moved. If you do, those partitions lose a replica, and if other replicas also fail, you lose data.

Partition Reassignment Best Practices

Moving partitions between brokers transfers data over the network, which consumes bandwidth that would otherwise serve producers and consumers. Here are practices to minimize impact:

• Throttle reassignment bandwidth. Use --throttle to limit how much network bandwidth reassignment uses. A typical throttle is 50-100 MB/s, leaving most bandwidth for production traffic.

kafka-reassign-partitions.sh --bootstrap-server kafka-1:9092 \
  --reassignment-json-file plan.json \
  --execute \
  --throttle 50000000  # 50 MB/s limit

• Reassign during off-peak hours. Even with throttling, reassignment adds I/O load. Doing it during low-traffic periods reduces the impact on latency-sensitive applications.
• Move a few partitions at a time. Rather than reassigning all 500 partitions at once, break it into batches of 50-100. This gives you checkpoints to verify cluster health before continuing.

Multi-Datacenter and Disaster Recovery

Running Kafka across multiple datacenters is essential for disaster recovery, but it introduces significant complexity. The fundamental challenge is the speed of light — data takes time to travel between datacenters, and that latency affects every write and replication operation.

Strategy 1: Active-Passive with MirrorMaker 2

The simplest multi-datacenter strategy is active-passive. One datacenter runs the primary Kafka cluster that handles all production traffic. A secondary cluster in another datacenter receives a real-time copy of all data via MirrorMaker 2 (MM2). If the primary datacenter fails, you fail over to the secondary.

MirrorMaker 2 is a Kafka Connect-based replication tool that replaces the original MirrorMaker. It provides:

• Topic mirroring: Automatically replicates all topics (or a configured subset) from the source cluster to the target cluster.
• Consumer offset translation: Converts consumer group offsets from the source cluster to equivalent offsets on the target cluster, enabling consumers to resume from the correct position after failover.
• Automatic topic creation: When new topics appear on the source, MM2 creates corresponding topics on the target.

# MirrorMaker 2 configuration (connect-mirror-maker.properties)
clusters = primary, dr

primary.bootstrap.servers = primary-kafka-1:9092,primary-kafka-2:9092
dr.bootstrap.servers = dr-kafka-1:9092,dr-kafka-2:9092

# Replicate all topics from primary to DR
primary->dr.enabled = true
primary->dr.topics = .*

# Do not replicate back (active-passive, not active-active)
dr->primary.enabled = false

# Sync consumer offsets for failover
sync.group.offsets.enabled = true
sync.group.offsets.interval.seconds = 10

# Heartbeat and checkpoint intervals
emit.heartbeats.enabled = true
emit.heartbeats.interval.seconds = 5
emit.checkpoints.enabled = true
emit.checkpoints.interval.seconds = 60

# Replication factor for mirrored topics on DR cluster
replication.factor = 3

# Start MirrorMaker 2
bin/connect-mirror-maker.sh config/connect-mirror-maker.properties

Strategy 2: Active-Active (Multi-Region)

In an active-active setup, both datacenters serve production traffic. Each cluster has a subset of topics that are “local” to it, and both clusters replicate to each other using MM2. This provides lower latency for regional users and higher availability but introduces the challenge of conflict resolution — what happens when the same key is written to both clusters simultaneously?

The typical solution is to avoid conflicts by design. Each region owns specific data partitions based on a geographic key (e.g., US orders go to the US cluster, EU orders go to the EU cluster). Cross-region replication ensures that both clusters have a complete view of all data for analytics and failover purposes.

Failover Planning

Having a secondary cluster is only useful if you can actually fail over to it when needed. Failover planning requires:

• Regular DR drills. Actually perform failovers quarterly. Discovering that your failover procedure does not work during a real disaster is the worst possible time.
• Consumer offset synchronization. Verify that consumer offsets on the DR cluster are within an acceptable range of the primary. MM2’s offset translation has a small lag, so consumers may reprocess a few seconds of data after failover.
• DNS or load balancer configuration. Your applications need a way to switch from the primary cluster’s bootstrap servers to the DR cluster’s bootstrap servers. This is typically done through DNS changes or load balancer reconfiguration.
• Runbook documentation. Write a step-by-step failover procedure that anyone on the operations team can follow. Under stress, people do not think clearly. A detailed runbook turns a panic-inducing event into a checklist.

Next Steps

KRaft represents the future of Kafka operations — simpler, faster, and with fewer moving parts. If you are starting a new Kafka deployment, use KRaft from day one. If you are running an existing ZooKeeper-based cluster, plan your migration thoughtfully and test it thoroughly before touching production.

• Monitoring and Operations — monitor the health of your KRaft-based cluster
• Real-World Use Cases — see how large organizations manage Kafka at massive scale