Kafka Security: SSL/TLS Encryption and SASL Authentication

By Default, Anyone on the Network Can Read Your Kafka Messages

Here is a fact that surprises many people: a fresh Kafka installation has zero security. No encryption, no authentication, no authorization. Every message travels across the network in plain text. Any application that knows the broker address can connect, read every topic, produce garbage data, or even delete topics entirely. There are no passwords, no certificates, no access controls.

In a development environment, this is fine. In production, this is a disaster waiting to happen. Imagine your Kafka cluster carries payment data, user credentials, or medical records. Without encryption, anyone with network access (or anyone who compromises a single machine on the same network) can silently capture every message using a basic packet sniffer. Without authentication, a misconfigured test application could accidentally connect to the production cluster and flood it with junk data. Without authorization, a developer with read access to one topic could read every topic in the cluster.

Kafka security is built around three separate concerns, and understanding the distinction between them is crucial. Each one solves a different problem, and you need all three for a secure cluster.

Three Layers: Encryption, Authentication, and Authorization

Think of Kafka security like the security at a corporate office building.

Encryption is like using sealed envelopes instead of postcards. When you send a postcard, anyone who handles it can read the message. When you send a sealed envelope, the contents are hidden from everyone except the intended recipient. SSL/TLS encryption does the same thing for Kafka messages — it scrambles the data in transit so that even if someone intercepts the network traffic, they see only gibberish. Encryption protects the confidentiality of your data.

Authentication is like showing your ID at the front desk. The security guard does not care what you plan to do inside the building — they just need to confirm you are who you claim to be. In Kafka, SASL authentication verifies the identity of every client that connects. It answers the question: “Is this really the orders-service, or is it an imposter?” Without authentication, any application can claim to be any user.

Authorization is like the keycard system inside the building. Even after you have shown your ID, you can only access the floors and rooms your keycard allows. In Kafka, Access Control Lists (ACLs) define exactly what each authenticated user can do. The orders-producer can write to the orders topic but cannot read from the payments topic. The analytics-consumer can read from multiple topics but cannot write to any of them. Authorization enforces the principle of least privilege.

Kafka supports four security protocol combinations that mix these concerns:

Protocol	Encryption	Authentication
PLAINTEXT	No	No
SSL	Yes (TLS)	Optional (mTLS)
SASL_PLAINTEXT	No	Yes (SASL)
SASL_SSL	Yes (TLS)	Yes (SASL)

For production, you should always use SASL_SSL. It gives you both encryption and authentication. SASL_PLAINTEXT provides authentication but sends credentials in plain text, which defeats the purpose. SSL alone provides encryption but does not verify client identity unless you configure mutual TLS, which is more complex to manage than SASL.

SSL/TLS: Sealed Envelopes for Your Data

SSL/TLS encryption ensures that data traveling between producers, consumers, and brokers cannot be read by anyone eavesdropping on the network. It works using the same technology that secures your web browser when you visit an HTTPS website.

The setup requires creating cryptographic certificates. The broker needs a certificate to prove its identity to clients (just like a website proves its identity to your browser), and optionally, clients need certificates to prove their identity to the broker (mutual TLS). All certificates are signed by a Certificate Authority (CA) that both sides trust.

The certificate generation process involves several steps. You create a CA (your own trusted authority), generate a keypair for the broker, have the CA sign the broker’s certificate, and bundle everything into keystores and truststores. The keystore holds the broker’s own certificate and private key. The truststore holds the CA certificate, which tells the broker which certificates to trust.

# Step 1: Create a Certificate Authority (your own trusted signer)
openssl req -new -x509 -keyout ca-key.pem -out ca-cert.pem -days 365 \
  -subj "/CN=Kafka-CA" -nodes

# Step 2: Generate the broker's keypair and create a keystore
keytool -keystore kafka.broker.keystore.jks -alias broker \
  -validity 365 -genkey -keyalg RSA -storepass broker-secret \
  -dname "CN=kafka-broker-1"

# Step 3: Create a signing request and have the CA sign it
keytool -keystore kafka.broker.keystore.jks -alias broker \
  -certreq -file broker-csr.pem -storepass broker-secret

openssl x509 -req -CA ca-cert.pem -CAkey ca-key.pem \
  -in broker-csr.pem -out broker-cert-signed.pem \
  -days 365 -CAcreateserial

# Step 4: Import the CA and signed cert back into the keystore
keytool -keystore kafka.broker.keystore.jks -alias CARoot \
  -import -file ca-cert.pem -storepass broker-secret -noprompt

keytool -keystore kafka.broker.keystore.jks -alias broker \
  -import -file broker-cert-signed.pem -storepass broker-secret

# Step 5: Create a truststore containing the CA cert
keytool -keystore kafka.broker.truststore.jks -alias CARoot \
  -import -file ca-cert.pem -storepass broker-secret -noprompt

Once the certificates exist, you configure the broker to use them. The essential lines in server.properties tell Kafka where to find the keystore and truststore, which TLS version to use, and what listener protocol to expose:

# Tell Kafka to listen on the SASL_SSL protocol
listeners=SASL_SSL://0.0.0.0:9093
advertised.listeners=SASL_SSL://kafka-broker-1:9093

# Point to the certificate stores
ssl.keystore.location=/etc/kafka/ssl/kafka.broker.keystore.jks
ssl.keystore.password=broker-secret
ssl.truststore.location=/etc/kafka/ssl/kafka.broker.truststore.jks
ssl.truststore.password=broker-secret

# Use modern TLS versions only
ssl.enabled.protocols=TLSv1.3,TLSv1.2
ssl.protocol=TLSv1.3

# Encrypt inter-broker traffic too (not just client traffic)
security.inter.broker.protocol=SASL_SSL

A common mistake is encrypting client-to-broker traffic but leaving inter-broker communication on PLAINTEXT. This means that replication data flows unencrypted between brokers. The security.inter.broker.protocol setting closes this gap.

SASL: Showing Your ID at the Door

With encryption in place, the next step is authentication. SASL (Simple Authentication and Security Layer) is Kafka’s framework for verifying client identity. Kafka supports several SASL mechanisms, but SCRAM-SHA-256 is the recommended choice for most deployments. It has two important advantages: it never sends passwords over the wire (it uses a challenge-response protocol instead), and it does not require a Kerberos infrastructure, which is complex to set up and maintain.

SCRAM works like this: when a client connects, the broker sends a random challenge. The client combines this challenge with its password using a one-way hash function and sends back the result. The broker does the same computation with the stored password hash and compares. If they match, the client is authenticated. The actual password never crosses the network.

To set up SCRAM authentication, you first create user credentials. Each application that connects to Kafka should have its own unique credentials — never share passwords between applications, because you need to be able to revoke access to one application without affecting others:

# Create credentials for each application
kafka-configs.sh --zookeeper localhost:2181 \
  --alter --add-config 'SCRAM-SHA-256=[iterations=8192,password=producer-secret]' \
  --entity-type users --entity-name order-producer

kafka-configs.sh --zookeeper localhost:2181 \
  --alter --add-config 'SCRAM-SHA-256=[iterations=8192,password=consumer-secret]' \
  --entity-type users --entity-name order-consumer

kafka-configs.sh --zookeeper localhost:2181 \
  --alter --add-config 'SCRAM-SHA-256=[iterations=8192,password=admin-secret]' \
  --entity-type users --entity-name kafka-admin

The iterations=8192 parameter controls how many times the password is hashed. Higher values make brute-force attacks harder but slow down authentication slightly. 8192 is a reasonable default.

On the broker side, you enable SCRAM and configure the broker’s own credentials with a JAAS configuration file. JAAS (Java Authentication and Authorization Service) is the standard way Java applications handle authentication:

# Add to server.properties
sasl.enabled.mechanisms=SCRAM-SHA-256
sasl.mechanism.inter.broker.protocol=SCRAM-SHA-256

# kafka_server_jaas.conf -- the broker's own credentials
KafkaServer {
    org.apache.kafka.common.security.scram.ScramLoginModule required
    username="kafka-admin"
    password="admin-secret";
};

# Tell Kafka where to find the JAAS config
export KAFKA_OPTS="-Djava.security.auth.login.config=/etc/kafka/kafka_server_jaas.conf"

ACLs: Permission Slips for Every Application

Authentication tells you who the client is. Authorization tells you what they are allowed to do. Without ACLs, an authenticated user can access everything in the cluster. With ACLs, you define granular permissions: which users can read from which topics, which users can write, and which users can perform administrative operations.

Think of ACLs as a permission matrix. Each row is a user, each column is a resource (topic, consumer group, cluster), and each cell says what operations are allowed. The goal is to give each application the minimum permissions it needs and nothing more. If the orders-producer only writes to the orders topic, it should not have permission to read from it, and it certainly should not have permission to delete topics or create new ones.

To set up ACLs, you first enable the authorizer in server.properties and designate a super user who can manage permissions:

authorizer.class.name=kafka.security.authorizer.AclAuthorizer
super.users=User:kafka-admin

Then you grant specific permissions to each application. Here is how to allow a producer to write to a topic and a consumer to read from it:

# Allow the producer to write to the orders topic
kafka-acls.sh --bootstrap-server localhost:9093 \
  --command-config admin.properties \
  --add --allow-principal User:order-producer \
  --operation Write --operation Describe \
  --topic orders

# Allow the consumer to read from the orders topic
kafka-acls.sh --bootstrap-server localhost:9093 \
  --command-config admin.properties \
  --add --allow-principal User:order-consumer \
  --operation Read --operation Describe \
  --topic orders

# Allow the consumer's consumer group
kafka-acls.sh --bootstrap-server localhost:9093 \
  --command-config admin.properties \
  --add --allow-principal User:order-consumer \
  --operation Read \
  --group order-consumer-group

Notice that the consumer needs both topic-level Read permission and consumer group permission. The Describe operation is also needed because producers and consumers must be able to fetch topic metadata (partition count, leader information) to function. Forgetting the consumer group ACL is one of the most common “why is my consumer not working” issues after enabling security.

Putting It All Together: A Secure Python Client

Once the broker is configured with SSL, SASL, and ACLs, your Python clients need to provide credentials and certificate paths. Here is how a producer and consumer connect to a secured cluster. The key lines are the four security-related settings: the protocol, the SASL mechanism, and the username/password pair:

from confluent_kafka import Producer, Consumer

# A producer connecting to a secured cluster
producer = Producer({
    "bootstrap.servers": "kafka-broker-1:9093,kafka-broker-2:9093",
    "security.protocol": "SASL_SSL",          # Use both encryption and auth
    "sasl.mechanism": "SCRAM-SHA-256",         # The auth method
    "sasl.username": "order-producer",         # This app's identity
    "sasl.password": "producer-secret",        # This app's credential
    "ssl.ca.location": "/path/to/ca-cert.pem", # Trust the cluster's CA
    "acks": "all",
})

producer.produce("orders", key="ORD-001", value=b'{"amount": 99.99}')
producer.flush()

# A consumer connecting to the same secured cluster
consumer = Consumer({
    "bootstrap.servers": "kafka-broker-1:9093,kafka-broker-2:9093",
    "security.protocol": "SASL_SSL",
    "sasl.mechanism": "SCRAM-SHA-256",
    "sasl.username": "order-consumer",
    "sasl.password": "consumer-secret",
    "ssl.ca.location": "/path/to/ca-cert.pem",
    "group.id": "order-consumer-group",
    "auto.offset.reset": "earliest",
})

consumer.subscribe(["orders"])

try:
    while True:
        msg = consumer.poll(1.0)
        if msg is None:
            continue
        if msg.error():
            print(f"Error: {msg.error()}")
            continue
        print(f"Received: {msg.key()} -> {msg.value()}")
except KeyboardInterrupt:
    pass
finally:
    consumer.close()

In production, you should never hardcode passwords in your application code. Store them in a secrets manager (HashiCorp Vault, AWS Secrets Manager, or Kubernetes Secrets) and inject them as environment variables at runtime. The ssl.ca.location path should point to a CA certificate bundle that is part of your deployment artifact or mounted as a volume.

Troubleshooting Common Security Issues

When you first enable security, things will break. Here are the two most common problems and how to fix them.

Certificate hostname verification failures happen when the hostname in the broker’s certificate does not match the hostname the client uses to connect. If your certificate has CN=kafka-broker-1 but the client connects to kafka-broker-1.internal.example.com, the TLS handshake will fail. To check what hostnames your certificate supports:

openssl x509 -in broker-cert-signed.pem -text -noout | grep -A1 "Subject Alternative Name"

The fix is to regenerate the certificate with the correct hostname in the Subject Alternative Name (SAN) field, or to ensure your clients connect using the exact hostname in the certificate.

“Unknown user” authentication failures happen when the SCRAM credentials were not created correctly, or when they were created in ZooKeeper but the broker has not loaded them yet. Verify that the credentials exist:

kafka-configs.sh --zookeeper localhost:2181 \
  --describe --entity-type users --entity-name order-producer

If the command returns nothing, the credentials were not created. If they exist but authentication still fails, try restarting the broker — it caches SCRAM credentials and sometimes does not pick up new ones immediately.

Production Security Checklist

Securing a Kafka cluster is not a single configuration change — it is a comprehensive effort that spans encryption, authentication, authorization, network design, and monitoring. Here is what a production-ready security posture looks like, told as a narrative rather than a checklist.

Start with encryption. Every connection in the cluster should use TLS 1.2 or 1.3 — both client-to-broker and broker-to-broker. Older TLS versions and weak cipher suites should be explicitly disabled. Certificates have expiration dates, and expired certificates cause outages. Set up automated monitoring that alerts your team at least 30 days before any certificate expires, and build a certificate rotation process that you practice regularly.

Next, lock down authentication. Every application should have its own unique SCRAM credentials. Never share credentials between applications, because you need the ability to revoke one application’s access without affecting others. Store credentials in a secrets manager, not in config files or environment variables that might appear in logs. Disable anonymous access entirely — if a client cannot authenticate, it should not be able to connect.

Then enforce authorization with ACLs. Follow the principle of least privilege: each application gets exactly the permissions it needs and nothing more. A producer that writes to one topic should not have read access to other topics. Review ACLs quarterly to remove permissions for decommissioned applications. Use deny rules for sensitive topics as an extra layer of protection.

At the network level, Kafka ports should never be exposed to the public internet. Use network segmentation to isolate production Kafka from development environments. Restrict ZooKeeper or KRaft controller ports so that only broker IPs can reach them. Schema Registry should have its own authentication — an unsecured Schema Registry is a backdoor into your message format definitions.

Finally, set up monitoring and alerting for security events. Log and alert on authentication failures (which might indicate brute-force attacks or misconfigured applications). Monitor authorization denials (which might indicate applications trying to access resources they should not). Enable audit logging for all administrative operations so you have a trail of who changed what and when.

Next Steps

Security is a prerequisite for production Kafka, not an afterthought. Once your cluster is locked down, you can focus on making it fast:

• Kafka Performance Tuning — optimize throughput and latency for your secured cluster
• Schema Registry and Avro — manage message schemas centrally