Kafka Exactly-Once Semantics: From Theory to Practice

The Delivery Problem Every Distributed System Faces

Imagine you are sending a letter through the postal service. Three things can happen. The letter might get lost along the way and never arrive — that is at-most-once delivery. You might send the same letter twice to be safe, and the recipient gets two copies — that is at-least-once delivery. Or the letter arrives exactly one time, no more, no less — that is exactly-once delivery.

In everyday life, exactly-once seems trivial. You hand a letter to the mail carrier, they deliver it. Done. But in distributed systems, exactly-once is one of the hardest problems in computer science. The reason is simple: networks are unreliable. When a producer sends a message to a Kafka broker and the network drops the acknowledgment, the producer has no way to know whether the broker received the message or not. Did the message make it? Should the producer retry? If it retries and the broker already has the message, you get a duplicate. If it does not retry and the broker never received it, you lose the message.

This is not a theoretical concern. In a payment processing system, a duplicate message means charging a customer twice. A lost message means a payment goes unrecorded. Both are unacceptable. Understanding delivery guarantees is essential for building systems that handle money, inventory, compliance records, or any data where accuracy matters.

The Three Delivery Guarantees

At-Most-Once: Fire and Forget

At-most-once is the simplest guarantee. The producer sends a message and does not retry if something goes wrong. Messages are never duplicated, but they can be lost. This is what you get when you set acks=0 on the producer or when you commit consumer offsets before processing messages.

Think of it like shouting a message across a noisy room. You say it once. If the other person heard you, great. If not, the message is gone. This is perfectly acceptable for metrics, logs, and telemetry data where losing a few data points does not matter. If you miss one CPU utilization reading out of thousands, nobody cares.

from confluent_kafka import Producer

# At-most-once: no retries, no waiting for acks
fire_and_forget = Producer({
    "bootstrap.servers": "kafka-1:9092",
    "acks": "0",                    # Don't wait for broker confirmation
    "retries": 0,                    # Never retry
    "linger.ms": 5,
})

# Send and move on -- message might be lost
fire_and_forget.produce("metrics", value=b'{"cpu": 72.5}')
fire_and_forget.poll(0)  # Trigger delivery callbacks

At-Least-Once: Retry Until Confirmed

At-least-once guarantees that every message is delivered, but some messages may be delivered more than once. The producer retries failed sends until the broker acknowledges receipt. This is the default behavior in most Kafka client configurations.

Think of it like sending a text message with delivery receipts. If you do not see the “delivered” checkmark, you send it again. The recipient might get the same text twice, but they will definitely get it at least once.

The problem is subtle. Consider a producer that sends a message to the broker. The broker receives it, writes it to the log, and sends back an acknowledgment. But the acknowledgment is lost due to a network glitch. The producer sees no response, assumes the message was lost, and retries. Now the broker has two copies of the same message. The producer did its job correctly — it retried when it did not get confirmation — but the result is a duplicate.

# At-least-once: retries enabled, acks required
reliable_producer = Producer({
    "bootstrap.servers": "kafka-1:9092",
    "acks": "all",                   # Wait for all replicas
    "retries": 5,                    # Retry on failure
    "retry.backoff.ms": 100,         # Wait between retries
})

# This will be delivered, but might be duplicated on retry
reliable_producer.produce("payments", key=b"user-123",
                          value=b'{"amount": 49.99}')
reliable_producer.flush()

Exactly-Once: The Holy Grail

Exactly-once means every message is delivered exactly one time. No losses, no duplicates. For years, many distributed systems engineers considered true exactly-once delivery impossible, or at least impractical. The famous “Two Generals Problem” in distributed computing suggests that you cannot guarantee reliable delivery over an unreliable channel.

Kafka’s approach is pragmatic rather than theoretical. It does not solve the Two Generals Problem in the abstract. Instead, it provides mechanisms that make message processing effectively exactly-once within the Kafka ecosystem. The key insight is that exactly-once is not a single feature — it is a combination of idempotent producers, transactional writes, and careful consumer-side processing working together.

Idempotent Producers: Killing Duplicates at the Source

The first building block of exactly-once is the idempotent producer. An idempotent operation is one that produces the same result no matter how many times you perform it. Setting a thermostat to 72 degrees is idempotent — doing it three times still results in 72 degrees. Adding 5 dollars to an account is not idempotent — doing it three times adds 15 dollars.

When you enable idempotency on a Kafka producer, the broker can detect and discard duplicate messages automatically. Here is how it works internally.

The Mechanism: Producer IDs and Sequence Numbers

When an idempotent producer starts up, it requests a Producer ID (PID) from the broker. This is a unique identifier assigned by the Kafka cluster. Along with the PID, the producer maintains a sequence number for each partition it writes to. Every message sent includes the PID and an incrementing sequence number.

The broker tracks the latest sequence number it has received from each PID for each partition. When a message arrives, the broker checks:

• If the sequence number is exactly one more than the last seen number, it is a new message. Accept it.
• If the sequence number is the same as or less than the last seen number, it is a duplicate. Discard it silently.
• If the sequence number is more than one ahead, something went wrong (messages arrived out of order). Reject it with an error.

This is similar to how TCP handles packet deduplication. Each packet has a sequence number, and the receiver discards duplicates. Kafka applies the same principle at the message level.

# Enable idempotency -- just one setting
idempotent_producer = Producer({
    "bootstrap.servers": "kafka-1:9092",
    "enable.idempotence": True,      # This is the magic switch
    "acks": "all",                   # Required for idempotency
    "retries": 2147483647,           # Effectively infinite retries
    "max.in.flight.requests.per.connection": 5,  # Safe with idempotency
})

# Now retries are safe -- broker deduplicates automatically
for i in range(1000):
    idempotent_producer.produce(
        "orders", key=f"order-{i}".encode(),
        value=f'{{"item": "widget", "qty": {i}}}'.encode()
    )
idempotent_producer.flush()

What Idempotency Does Not Solve

Idempotent producers guarantee that a single producer instance will not create duplicates within a single session. But they have limitations:

• If the producer crashes and restarts, it gets a new PID. It has no memory of what it sent before.
• Idempotency only works within a single partition. If you need atomic writes across multiple partitions, you need transactions.
• Idempotency does not help on the consumer side. Even if each message exists exactly once in the topic, a consumer that crashes after processing but before committing its offset will reprocess the message after restart.

This is why idempotent producers are necessary but not sufficient for end-to-end exactly-once.

Transactional Producers: Atomic Writes Across Partitions

Transactions extend idempotency to cover multi-partition writes. A transactional producer can write to multiple topics and partitions as a single atomic operation — either all messages are committed and visible to consumers, or none of them are.

Why Transactions Matter: A Real-World Scenario

Consider a Kafka Streams application that reads from an input topic, processes messages, and writes results to an output topic. For each input message, it needs to do two things atomically:

1. Write the processed result to the output topic.
2. Commit the consumer offset for the input message (which is itself a write to the __consumer_offsets topic).

Without transactions, these are two separate writes. If the application writes the result but crashes before committing the offset, it will reprocess the input message after restart and write the result again. The output topic now has a duplicate.

With transactions, both writes happen atomically. Either the result is written and the offset is committed, or neither happens. On restart, the application picks up from the last committed offset and reprocesses cleanly.

How Transactions Work Internally

Kafka transactions use a Transaction Coordinator, which is a broker that manages the state of each transaction. The flow works like this:

1. The producer calls init_transactions(), which registers a Transactional ID with the coordinator. Unlike the PID, the Transactional ID survives restarts — you configure it explicitly.
2. The producer calls begin_transaction() to start a new transaction.
3. The producer sends messages to one or more partitions. These messages are written to the log but marked as “uncommitted.”
4. The producer calls commit_transaction(). The coordinator writes a “commit” marker to all involved partitions. Consumers configured with isolation.level=read_committed will only see messages up to the commit marker.
5. If anything goes wrong, the producer calls abort_transaction(), and the coordinator writes an “abort” marker. Consumers will skip all messages from the aborted transaction.

from confluent_kafka import Producer

transactional_producer = Producer({
    "bootstrap.servers": "kafka-1:9092,kafka-2:9092,kafka-3:9092",
    "enable.idempotence": True,
    "transactional.id": "order-processor-txn-1",  # Survives restarts
    "acks": "all",
    "retries": 2147483647,
})

# Initialize transactions (call once at startup)
transactional_producer.init_transactions()

try:
    # Start a transaction
    transactional_producer.begin_transaction()

    # Write to multiple topics/partitions atomically
    transactional_producer.produce("order-results", key=b"order-42",
                                   value=b'{"status": "processed"}')
    transactional_producer.produce("order-audit", key=b"order-42",
                                   value=b'{"action": "fulfilled"}')
    transactional_producer.produce("notifications", key=b"user-7",
                                   value=b'{"msg": "Your order shipped!"}')

    # Commit -- all three messages become visible atomically
    transactional_producer.commit_transaction()
except Exception as e:
    # Abort -- none of the three messages will be visible
    transactional_producer.abort_transaction()
    raise

The Transactional ID: Fencing Zombie Producers

The Transactional ID serves another critical purpose: zombie fencing. Consider this scenario. A producer with Transactional ID “txn-1” starts a transaction, writes some messages, and then freezes (perhaps due to a long garbage collection pause). The system assumes it has died and starts a new producer instance with the same Transactional ID “txn-1”. The new instance calls init_transactions(), which bumps the epoch associated with that Transactional ID. When the zombie producer wakes up and tries to commit its transaction, the broker rejects it because its epoch is stale. This prevents the zombie from accidentally committing partial or out-of-date data.

Think of it like a relay race. When a new runner takes the baton, the previous runner is no longer allowed to cross the finish line, even if they are still running. The baton (the current epoch) determines who is the legitimate runner.

Exactly-Once in Kafka Streams

Kafka Streams provides the cleanest path to exactly-once processing because it controls both the reading and writing sides. The read-process-write pattern wraps input consumption, state updates, and output production into a single transaction.

Enabling exactly-once in Kafka Streams is remarkably simple — it is a single configuration setting:

Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "order-processor");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-1:9092");
// This single setting enables exactly-once for the entire pipeline
props.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG,
          StreamsConfig.EXACTLY_ONCE_V2);

StreamsBuilder builder = new StreamsBuilder();
KStream<String, String> orders = builder.stream("raw-orders");

orders
    .filter((key, value) -> value.contains("premium"))
    .mapValues(value -> enrichOrder(value))
    .to("enriched-orders");

KafkaStreams streams = new KafkaStreams(builder.build(), props);
streams.start();

Under the hood, EXACTLY_ONCE_V2 (introduced in Kafka 2.5) does several things:

• Each stream task gets its own transactional producer.
• For every batch of input messages processed, Kafka Streams opens a transaction, writes all output messages, sends the consumer offset commits as part of the transaction, and commits the transaction.
• If the task crashes mid-transaction, the transaction is aborted. On restart, the task reads from the last committed offset and reprocesses cleanly with no duplicates in the output.

The V2 version is more efficient than the original EXACTLY_ONCE (now called EXACTLY_ONCE_BETA) because it uses a single transaction coordinator for all tasks on a given thread, rather than one per task. This significantly reduces the overhead of transactional processing.

Consumer-Side Exactly-Once: The Other Half of the Equation

Kafka’s transactional guarantees ensure that messages are written exactly once to topics. But what about the consumer that reads those messages and writes to an external system like a database? Kafka cannot control what happens outside its own ecosystem.

The solution is idempotent consumption — designing your consumer so that processing the same message multiple times produces the same result. There are several strategies:

Strategy 1: Idempotent Writes with Natural Keys

If your external system supports upserts (insert-or-update), use the message key as the primary key. Processing the same message twice simply overwrites the same row with the same data.

import psycopg2
from confluent_kafka import Consumer

consumer = Consumer({
    "bootstrap.servers": "kafka-1:9092",
    "group.id": "db-writer",
    "enable.auto.commit": False,
    "isolation.level": "read_committed",  # Only see committed transactions
})
consumer.subscribe(["enriched-orders"])

conn = psycopg2.connect("dbname=orders")

try:
    while True:
        msg = consumer.poll(1.0)
        if msg is None or msg.error():
            continue

        order = json.loads(msg.value())

        # UPSERT: processing the same message twice is harmless
        with conn.cursor() as cur:
            cur.execute("""
                INSERT INTO orders (order_id, status, amount, updated_at)
                VALUES (%s, %s, %s, NOW())
                ON CONFLICT (order_id)
                DO UPDATE SET status = EXCLUDED.status,
                             amount = EXCLUDED.amount,
                             updated_at = NOW()
            """, (order["order_id"], order["status"], order["amount"]))
        conn.commit()
        consumer.commit(message=msg)
except KeyboardInterrupt:
    pass
finally:
    consumer.close()
    conn.close()

Strategy 2: Transactional Outbox with Offset Tracking

For systems that do not support upserts, store the Kafka offset alongside your data in the same database transaction. On restart, check the stored offset to know where to resume.

# Store offset in the same transaction as the business data
with conn.cursor() as cur:
    cur.execute("BEGIN")
    cur.execute("""
        INSERT INTO processed_events (event_id, payload, kafka_offset, kafka_partition)
        VALUES (%s, %s, %s, %s)
    """, (event_id, payload, msg.offset(), msg.partition()))
    cur.execute("""
        INSERT INTO consumer_offsets (topic, partition_id, committed_offset)
        VALUES (%s, %s, %s)
        ON CONFLICT (topic, partition_id)
        DO UPDATE SET committed_offset = EXCLUDED.committed_offset
    """, (msg.topic(), msg.partition(), msg.offset()))
    cur.execute("COMMIT")

This way, either both the event and the offset are stored, or neither is. On restart, you query the consumer_offsets table and seek to the stored offset, guaranteeing no duplicates.

Performance Implications: The Cost of Exactly-Once

Exactly-once is not free. Every guarantee comes with a performance cost, and understanding these costs helps you decide when the guarantee is worth it.

Idempotent producers add minimal overhead. The broker must store and check sequence numbers per PID per partition, but this is an in-memory operation that adds microseconds to each write. In practice, the throughput impact is less than 3%. There is almost no reason not to enable idempotency.

Transactional producers add more significant overhead. Each transaction involves extra round trips to the transaction coordinator — begin, commit markers written to each involved partition, and end markers. Transactions also increase end-to-end latency because consumers with read_committed isolation cannot see messages until the transaction commits. If your transactions are small (covering a single message), the per-message overhead is high. If your transactions batch hundreds of messages, the overhead is amortized and becomes negligible.

Guarantee	Throughput Impact	Latency Impact	Use Case
At-most-once	None (baseline)	Lowest	Metrics, logs, telemetry
At-least-once	Minimal (~1%)	Low	Most applications
Exactly-once (idempotent)	Minimal (~3%)	Low	Any producer workload
Exactly-once (transactional)	Moderate (10-20%)	Higher	Financial, inventory, compliance

When You Actually Need Exactly-Once

Here is the uncomfortable truth: most applications do not need exactly-once semantics. At-least-once with idempotent consumers is simpler, faster, and sufficient for the vast majority of use cases.

You should use exactly-once when:

• Financial transactions: Charging a credit card, transferring money, adjusting account balances. Duplicates cost real money.
• Inventory management: Decrementing stock counts. A duplicate decrement means overselling.
• Compliance and audit logs: Regulatory requirements may demand that each event is recorded exactly once.
• Kafka Streams pipelines: When your entire pipeline is within Kafka, the EXACTLY_ONCE_V2 setting is cheap enough that there is little reason not to use it.

You probably do not need exactly-once when:

• Analytics and metrics: A duplicate pageview or CPU reading is statistically insignificant.
• Log aggregation: Duplicate log entries are annoying but not harmful.
• Notifications: Sending a user two identical emails is bad UX, but the business cost is low. Rate limiting or deduplication at the notification service is simpler than exactly-once Kafka processing.
• Search indexing: Re-indexing the same document is an idempotent operation by nature.

The general rule: if your downstream system can naturally handle duplicates (through upserts, idempotent operations, or deduplication logic), prefer at-least-once delivery with idempotent consumers. Reserve full transactional exactly-once for the cases where duplicates or losses cause real business harm.

Next Steps

Exactly-once semantics give you the strongest delivery guarantee Kafka offers, but they are one piece of a larger puzzle. Understanding when to apply them — and when simpler guarantees suffice — is what separates pragmatic engineers from those who over-engineer.

• Kafka Performance Tuning — understand the throughput impact of exactly-once settings
• Event-Driven Architecture — see how exactly-once fits into larger architectural patterns