Kafka Installation and Setup: Local, Docker, and CLI

Getting Kafka running on your machine is the essential first step to learning it properly. Reading about topics, partitions, and consumer groups is useful, but nothing cements understanding like actually producing and consuming messages yourself. This guide covers two approaches and explains when each one makes sense.

Choosing your setup approach

Before diving into commands, take a moment to understand which approach fits your situation.

Option 1: Local binary installation is best if you want to understand exactly what Kafka does when it starts up, if you are following along with a course or certification study, or if you need to tweak low-level configuration files. The downside is that you need Java installed, and cleaning up after yourself requires manual steps.

Option 2: Docker Compose is the preferred approach for most developers. It gives you a reproducible, isolated environment that starts with one command and stops with one command. It is also easier to add companion tools like Schema Registry and a web-based UI. If you are building an application that uses Kafka and just need a local cluster to develop against, start here.

Option 3: Managed services (Confluent Cloud, Amazon MSK, Aiven) are what you would use in production. You do not install or operate anything — the cloud provider handles all of that. These are outside the scope of this guide, but they are worth knowing about.

Option 1: Local binary setup with KRaft mode

Since Kafka 3.5, KRaft mode is production-ready and the recommended way to run Kafka. This means you no longer need to install and manage ZooKeeper separately — Kafka handles its own cluster coordination internally.

Step 1: Download Kafka

The first step is downloading the Kafka distribution from the Apache website. This is a compressed archive containing the Kafka server code, configuration files, and all the command-line tools you will use to interact with Kafka. The 2.13 in the filename refers to the Scala version the binaries were compiled with — for most users, this does not matter, but it is good to know what it means.

# Download Kafka 3.7.0 (adjust version as needed)
wget https://downloads.apache.org/kafka/3.7.0/kafka_2.13-3.7.0.tgz

# Extract the archive
tar -xzf kafka_2.13-3.7.0.tgz
cd kafka_2.13-3.7.0

After extracting, you will see a bin/ directory full of shell scripts (the CLI tools) and a config/ directory with configuration templates. The config/kraft/ subdirectory contains the KRaft-specific configuration files.

Step 2: Generate a cluster ID

Every Kafka cluster needs a unique identifier. This ID is used internally to ensure that brokers from different clusters do not accidentally join each other. The command below generates a random UUID that will serve as your cluster’s identity.

KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"
echo $KAFKA_CLUSTER_ID

Step 3: Format the storage directory

Before Kafka can start, it needs to initialize its data directory with the cluster ID. This step creates the metadata files that KRaft uses for its consensus protocol. Think of it as setting up the warehouse before you can start storing goods — you need shelving, an inventory system, and a loading dock.

bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID \
  -c config/kraft/server.properties

If you see a message like “Formatting /tmp/kraft-combined-logs,” the step succeeded. The path shown is where Kafka will store all its data by default.

Step 4: Start Kafka

This command starts Kafka as a foreground process, meaning it will take over your terminal window. You will see a flood of log messages as the broker initializes, elects itself as the controller (since it is the only node), and starts listening for connections on port 9092.

bin/kafka-server-start.sh config/kraft/server.properties

Once you see a line containing “Kafka Server started,” the broker is ready. The default KRaft configuration file sets process.roles=broker,controller, meaning this single node acts as both a data broker and a metadata controller. In a production cluster, these roles would typically be separated across dedicated machines.

Step 5: Verify it works

Open a new terminal window (since the first one is running the Kafka process) and create a test topic. The command below tells Kafka to create a topic called test-topic with 3 partitions and a replication factor of 1 (which is all you can do with a single broker — you need at least 3 brokers for a replication factor of 3).

bin/kafka-topics.sh --create \
  --topic test-topic \
  --bootstrap-server localhost:9092 \
  --partitions 3 \
  --replication-factor 1

Confirm it was created by listing all topics:

bin/kafka-topics.sh --list --bootstrap-server localhost:9092

You should see test-topic in the output. If you do, your local Kafka installation is working correctly.

Option 2: Docker Compose setup

Docker Compose lets you define your entire Kafka environment in a single YAML file. Starting the cluster, stopping it, and wiping it clean to start fresh are all one-command operations.

Single broker setup

The Docker Compose file below creates a single Kafka broker running in KRaft mode inside a container. The key part to understand is the environment variables — they are the Docker equivalent of the server.properties configuration file.

A few variables worth calling out: KAFKA_NODE_ID gives this broker a unique identity. KAFKA_PROCESS_ROLES tells it to act as both broker and controller. KAFKA_ADVERTISED_LISTENERS is the address that clients (your application code running on the host machine) will use to connect — this must be localhost:9092 when running in Docker, not the container’s internal hostname. The CLUSTER_ID is a pre-generated value so that the container can start without the manual format step.

version: '3.8'

services:
  kafka:
    image: apache/kafka:3.7.0
    hostname: kafka
    container_name: kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_LOG_DIRS: /var/lib/kafka/data
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      CLUSTER_ID: "MkU3OEVBNTcwNTJENDM2Qk"
    volumes:
      - kafka-data:/var/lib/kafka/data

volumes:
  kafka-data:

Start the cluster with a single command. The -d flag runs it in the background so you get your terminal back.

docker compose up -d

To check that Kafka started successfully, look for the startup confirmation in the logs:

docker compose logs kafka | grep "Kafka Server started"

If you see the “Kafka Server started” message, you are ready to create topics and start sending messages.

Full setup with Schema Registry and Kafka UI

For a more complete development environment, you can add Confluent Schema Registry (which manages data schemas for your topics) and Kafka UI (a web dashboard for browsing topics, viewing messages, and monitoring consumer groups). Schema Registry is especially useful when you start working with Avro or Protobuf serialization instead of plain strings.

The configuration below adds two more services to the Docker Compose file. Notice that the Kafka service now has an additional listener (PLAINTEXT_HOST) — this is because the Schema Registry and Kafka UI containers need to reach Kafka using the container’s internal hostname (kafka:29092), while your local applications still connect via localhost:9092.

version: '3.8'

services:
  kafka:
    image: apache/kafka:3.7.0
    hostname: kafka
    container_name: kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093,PLAINTEXT_HOST://0.0.0.0:29092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_LOG_DIRS: /var/lib/kafka/data
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      CLUSTER_ID: "MkU3OEVBNTcwNTJENDM2Qk"
    volumes:
      - kafka-data:/var/lib/kafka/data

  schema-registry:
    image: confluentinc/cp-schema-registry:7.6.0
    hostname: schema-registry
    container_name: schema-registry
    depends_on:
      - kafka
    ports:
      - "8081:8081"
    environment:
      SCHEMA_REGISTRY_HOST_NAME: schema-registry
      SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: kafka:29092
      SCHEMA_REGISTRY_LISTENERS: http://0.0.0.0:8081

  kafka-ui:
    image: provectuslabs/kafka-ui:latest
    container_name: kafka-ui
    depends_on:
      - kafka
      - schema-registry
    ports:
      - "8080:8080"
    environment:
      KAFKA_CLUSTERS_0_NAME: local
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka:29092
      KAFKA_CLUSTERS_0_SCHEMAREGISTRY: http://schema-registry:8081

volumes:
  kafka-data:

Start everything with the same command:

docker compose up -d

After all containers are healthy (give it 15-30 seconds), you can access:

• Kafka broker: localhost:9092 (for your applications)
• Schema Registry API: http://localhost:8081
• Kafka UI dashboard: http://localhost:8080

Open the Kafka UI in your browser. It provides a visual interface for browsing topics, viewing individual messages, monitoring consumer group lag, and inspecting schemas. It is an excellent learning tool because you can see the effects of your CLI commands in real time.

Working with topics via CLI

Whether you used the binary or Docker setup, the CLI tools work the same way. If you are using Docker, you will need to prefix commands with docker exec -it kafka /opt/kafka/bin/ instead of bin/.

Creating a topic

The command below creates a topic named user-events with 6 partitions. The --replication-factor 1 is required for a single-broker setup. In a production cluster with 3 brokers, you would set this to 3 so that each partition has two backup copies.

bin/kafka-topics.sh --create \
  --topic user-events \
  --bootstrap-server localhost:9092 \
  --partitions 6 \
  --replication-factor 1

Inspecting a topic

The --describe flag shows you detailed information about a topic, including which broker leads each partition, which brokers hold replicas, and which replicas are in the ISR. This is one of the most useful debugging commands you will use.

bin/kafka-topics.sh --describe \
  --topic user-events \
  --bootstrap-server localhost:9092

Deleting a topic

When you are done experimenting, you can delete a topic to clean up. Be careful with this in production — there is no undo.

bin/kafka-topics.sh --delete \
  --topic user-events \
  --bootstrap-server localhost:9092

Producing and consuming messages via CLI

Kafka ships with command-line producer and consumer tools that are invaluable for testing and debugging. Think of them as the “curl” of the Kafka world — quick, direct ways to interact with the system without writing application code.

Sending messages with the console producer

The console producer opens an interactive session where each line you type becomes a message sent to the specified topic. Press Enter to send each message, and Ctrl+C to exit. Behind the scenes, this tool creates a full Kafka producer client, connects to the cluster, and sends each line as the value of a message with a null key.

bin/kafka-console-producer.sh \
  --topic test-topic \
  --bootstrap-server localhost:9092

Once the prompt appears, type a few messages:

> hello world
> this is a test message
> kafka is working

To send messages with keys (which ensures that messages with the same key always go to the same partition), use the parse.key and key.separator properties. The key goes before the separator, and the value goes after it.

bin/kafka-console-producer.sh \
  --topic test-topic \
  --bootstrap-server localhost:9092 \
  --property parse.key=true \
  --property key.separator=:

> user-1:logged in
> user-2:signed up
> user-1:clicked button

Reading messages with the console consumer

The console consumer subscribes to a topic and prints every message it receives. The --from-beginning flag tells it to start from the earliest available offset rather than only showing new messages that arrive after it connects. Without this flag, you would only see messages produced after the consumer started.

bin/kafka-console-consumer.sh \
  --topic test-topic \
  --bootstrap-server localhost:9092 \
  --from-beginning

For debugging, it is often helpful to see not just the message value but also the key, partition number, and offset. This tells you exactly where each message landed, which helps you verify that key-based partitioning is working correctly.

bin/kafka-console-consumer.sh \
  --topic test-topic \
  --bootstrap-server localhost:9092 \
  --from-beginning \
  --property print.key=true \
  --property print.partition=true \
  --property print.offset=true

The output will look something like this, showing you the full picture of where each message lives:

Partition:0  Offset:0  Key:user-1  Value:logged in
Partition:2  Offset:0  Key:user-2  Value:signed up
Partition:0  Offset:1  Key:user-1  Value:clicked button

Notice that user-1 messages both landed in Partition 0 (same key, same partition), while user-2 ended up in Partition 2. This is key-based partitioning in action.

Consuming with a consumer group

When you add the --group flag, the consumer joins a named consumer group. This matters when you run multiple consumers — Kafka will distribute the partitions among them so each partition is handled by exactly one consumer in the group.

bin/kafka-console-consumer.sh \
  --topic test-topic \
  --bootstrap-server localhost:9092 \
  --group my-test-group \
  --from-beginning

You can check the status of your consumer group, including how far behind it is on each partition:

bin/kafka-consumer-groups.sh --describe \
  --group my-test-group \
  --bootstrap-server localhost:9092

Understanding key broker configuration

A few configuration properties come up repeatedly when setting up and troubleshooting Kafka. Understanding what they do will save you hours of debugging.

listeners vs. advertised.listeners. This is the single most common source of confusion in Kafka setup, especially with Docker. listeners is the network address the broker binds to — it is what the broker listens on. advertised.listeners is the address the broker tells clients to use when connecting. In Docker, the broker might listen on 0.0.0.0:9092 inside the container, but it needs to advertise localhost:9092 so that your application running on the host machine can reach it. Getting this wrong results in “connection refused” errors that are maddeningly hard to debug.

log.dirs is where Kafka stores all partition data on disk. The default is /tmp/kafka-logs, which is fine for development but terrible for production (most operating systems periodically clean /tmp). In production, point this at a dedicated, fast disk.

log.retention.hours controls how long Kafka keeps messages before deleting them. The default is 168 hours (7 days). Set this based on how long your consumers might need to replay data.

Troubleshooting common issues

Here are the problems you are most likely to hit during setup, along with their solutions.

“Connection refused” on port 9092. This almost always means Kafka is not running. Check docker compose ps (Docker) or your process list (binary install). If it was running and crashed, check the logs for errors — the most common cause is a port conflict (another process already using 9092) or insufficient memory.

“Leader not available” when creating or producing to a topic. This typically happens right after topic creation, before the leader has been fully elected. Wait 2-3 seconds and retry. If it persists, check that the broker is healthy.

Client gets the wrong broker address and cannot connect. This is the advertised.listeners problem described above. The broker is telling clients to connect to an address they cannot reach (like the container’s internal hostname instead of localhost). Fix the advertised.listeners configuration.

Messages are not showing up in the consumer. The most common cause is that the consumer group’s offset is already past the messages you produced. Use --from-beginning to start from the beginning, or check the consumer group’s committed offsets with kafka-consumer-groups.sh --describe.

Disk full errors. Your retention period is keeping more data than your disk can hold. Either reduce log.retention.hours, increase disk space, or set log.retention.bytes to cap the total size.

Quick end-to-end verification

Run this sequence to confirm everything works from end to end. If all five messages appear in step 3, your Kafka installation is functioning correctly.

# 1. Create a topic
bin/kafka-topics.sh --create \
  --topic verification-test \
  --bootstrap-server localhost:9092 \
  --partitions 3 \
  --replication-factor 1

# 2. Produce 5 messages
echo -e "msg-1\nmsg-2\nmsg-3\nmsg-4\nmsg-5" | \
  bin/kafka-console-producer.sh \
  --topic verification-test \
  --bootstrap-server localhost:9092

# 3. Consume and count messages
bin/kafka-console-consumer.sh \
  --topic verification-test \
  --bootstrap-server localhost:9092 \
  --from-beginning \
  --max-messages 5

# 4. Clean up
bin/kafka-topics.sh --delete \
  --topic verification-test \
  --bootstrap-server localhost:9092

Next steps

You now have a working Kafka installation and know how to create topics, produce messages, and consume them. Here is where to go from here:

• Topics, Partitions, and Offsets — understand the data model you just interacted with in much greater depth.
• Kafka Architecture Explained — learn how the broker you just started works internally, including replication and failover.