Event-Driven Architecture with Kafka

Why Event-Driven Architecture Exists

Traditional software architectures are built around requests. A user clicks a button, which calls an API, which queries a database, which returns a response. Everything is synchronous, sequential, and tightly coupled. The order service calls the payment service, which calls the inventory service, which calls the shipping service. Each service waits for the next one to finish before proceeding.

This works fine at small scale. But as your system grows, the coupling becomes a bottleneck. Adding a new feature — say, a loyalty points service — means modifying the order service to call it. If the shipping service is slow, the entire chain slows down. If the payment service is down, orders cannot be placed at all, even though the user’s intent has been clearly expressed.

Event-driven architecture (EDA) flips this model. Instead of services calling each other directly, they communicate by producing and consuming events. When an order is placed, the order service publishes an “OrderPlaced” event. The payment service, the inventory service, the shipping service, and the loyalty points service all independently consume that event and react to it. The order service does not know or care who is listening. Adding a new service means subscribing to existing events — no changes to the producer.

Think of it like a newspaper. The journalist writes a story and publishes it. Readers choose which stories to read. The journalist does not personally deliver the paper to each reader or even know who the readers are. New readers can start subscribing without the journalist changing anything about their process.

Events vs Commands vs Queries

Before building an event-driven system, you need to understand three fundamental message types. Confusing them is one of the most common architectural mistakes.

Events: Things That Happened

An event is an immutable fact about something that already happened. “OrderPlaced,” “PaymentReceived,” “InventoryReserved.” Events are stated in past tense because they describe completed actions. The producer does not care who receives the event or what they do with it. Events are typically consumed by multiple services.

Events carry enough context for any consumer to react without calling back to the producer. A well-designed “OrderPlaced” event includes the order ID, the customer ID, the list of items, the total amount, and a timestamp. A poorly designed event just says “order 42 was placed” and forces every consumer to call the order service for details, which defeats the purpose of decoupling.

Commands: Things You Want to Happen

A command is a request directed at a specific service to perform an action. “ChargePayment,” “ReserveInventory,” “SendNotification.” Commands are stated in imperative form and are typically consumed by a single service. The sender expects a result — success or failure.

Commands create coupling because the sender must know which service handles the command. They are not inherently bad, but overusing them in an event-driven system turns Kafka into a glorified RPC mechanism, which misses the point of EDA.

Queries: Things You Want to Know

A query is a request for information. “GetOrderStatus,” “ListUserOrders.” Queries are synchronous by nature because the caller needs the result to proceed. Kafka is generally not the right tool for queries. Use REST APIs, GraphQL, or gRPC for synchronous lookups.

# Event: something that happened (past tense, no expectation of response)
order_placed_event = {
    "type": "OrderPlaced",
    "order_id": "ORD-2024-001",
    "customer_id": "CUST-789",
    "items": [
        {"product_id": "PROD-42", "quantity": 2, "price": 29.99},
        {"product_id": "PROD-17", "quantity": 1, "price": 49.99}
    ],
    "total": 109.97,
    "timestamp": "2026-07-12T10:30:00Z"
}

# Command: something you want to happen (imperative, expects a result)
charge_payment_command = {
    "type": "ChargePayment",
    "order_id": "ORD-2024-001",
    "customer_id": "CUST-789",
    "amount": 109.97,
    "payment_method_id": "PM-456"
}

Event Sourcing: State as a Sequence of Events

Most applications store the current state of things. An order in the database has a status field: “shipped.” But how did it get there? When was it created? When was it paid? Traditional databases overwrite the previous state with each update, losing history.

Event sourcing takes a radically different approach. Instead of storing current state, you store every event that led to the current state. The state becomes a derived view that you can reconstruct at any point by replaying events from the beginning.

The Bank Account Analogy

A bank account is the perfect analogy. Your bank does not just store your current balance. It stores every transaction: deposits, withdrawals, transfers, fees. Your balance is derived by summing all transactions. If there is a dispute, they can look at the full history. If they discover an error, they do not delete the wrong transaction — they add a correcting transaction. The history is immutable.

Kafka is a natural fit for event sourcing because it is already an immutable, append-only log with configurable retention. A Kafka topic is an event store.

import json
from confluent_kafka import Producer, Consumer
from datetime import datetime

producer = Producer({"bootstrap.servers": "kafka-1:9092"})

# Record every state change as an event
def create_order(order_id, customer_id, items):
    event = {
        "type": "OrderCreated",
        "order_id": order_id,
        "customer_id": customer_id,
        "items": items,
        "timestamp": datetime.utcnow().isoformat()
    }
    producer.produce("order-events", key=order_id.encode(),
                     value=json.dumps(event).encode())
    producer.flush()

def add_payment(order_id, payment_id, amount):
    event = {
        "type": "PaymentAdded",
        "order_id": order_id,
        "payment_id": payment_id,
        "amount": amount,
        "timestamp": datetime.utcnow().isoformat()
    }
    producer.produce("order-events", key=order_id.encode(),
                     value=json.dumps(event).encode())
    producer.flush()

def ship_order(order_id, tracking_number):
    event = {
        "type": "OrderShipped",
        "order_id": order_id,
        "tracking_number": tracking_number,
        "timestamp": datetime.utcnow().isoformat()
    }
    producer.produce("order-events", key=order_id.encode(),
                     value=json.dumps(event).encode())
    producer.flush()

# Rebuild current state from event history
def rebuild_order(events):
    """Apply each event in sequence to build current state."""
    state = {}
    for event in events:
        event_type = event["type"]
        if event_type == "OrderCreated":
            state = {
                "order_id": event["order_id"],
                "customer_id": event["customer_id"],
                "items": event["items"],
                "status": "created",
                "created_at": event["timestamp"]
            }
        elif event_type == "PaymentAdded":
            state["status"] = "paid"
            state["payment_id"] = event["payment_id"]
        elif event_type == "OrderShipped":
            state["status"] = "shipped"
            state["tracking_number"] = event["tracking_number"]
    return state

The Snapshot Optimization

Replaying thousands of events to rebuild state is expensive. The solution is snapshots — periodically save the current state so you only need to replay events since the last snapshot. This is similar to how video games use save files. You do not replay the entire game from the beginning; you load the last save and continue from there.

In practice, you store snapshots in a compacted Kafka topic or a database. When rebuilding state, load the latest snapshot and replay only the events that came after it.

CQRS: Separating Reads from Writes

CQRS (Command Query Responsibility Segregation) is the natural companion to event sourcing. The idea is simple: the model you use to write data and the model you use to read data do not have to be the same.

Consider an e-commerce platform. When a customer places an order, you need to validate inventory, apply discounts, check fraud signals, and record the order. This write path is complex and transactional. But when a customer views their order history, you need a simple, fast query that returns denormalized data. When an analytics team queries sales trends, they need aggregated, indexed data in a completely different shape.

CQRS solves this by creating separate read models optimized for each query pattern. Kafka is the bridge between the write model and the read models. The write service publishes events to Kafka, and one or more read model builders consume those events and project them into purpose-built data stores.

from confluent_kafka import Consumer
import json
import redis
from elasticsearch import Elasticsearch

# Read model builder: consumes events, builds multiple read-optimized views
consumer = Consumer({
    "bootstrap.servers": "kafka-1:9092",
    "group.id": "read-model-builder",
    "auto.offset.reset": "earliest",
    "enable.auto.commit": False,
})
consumer.subscribe(["order-events"])

redis_client = redis.Redis(host="localhost", port=6379)
es_client = Elasticsearch("http://localhost:9200")

try:
    while True:
        msg = consumer.poll(1.0)
        if msg is None or msg.error():
            continue

        event = json.loads(msg.value())
        order_id = event["order_id"]

        # Read Model 1: Redis cache for fast order lookups
        if event["type"] == "OrderCreated":
            redis_client.hset(f"order:{order_id}", mapping={
                "status": "created",
                "customer_id": event["customer_id"],
                "item_count": len(event["items"]),
            })
        elif event["type"] == "OrderShipped":
            redis_client.hset(f"order:{order_id}", "status", "shipped")

        # Read Model 2: Elasticsearch for full-text search
        es_client.index(index="order-events", body={
            "order_id": order_id,
            "event_type": event["type"],
            "timestamp": event["timestamp"],
            "data": event
        })

        consumer.commit(message=msg)
except KeyboardInterrupt:
    pass
finally:
    consumer.close()

The beauty of CQRS with Kafka is that adding a new read model requires zero changes to the write side. Need a new analytics dashboard? Create a new consumer group that reads from the same topic and projects data into a ClickHouse table. Need a recommendation engine? Create another consumer that builds a graph database from order events. The event stream is the single source of truth, and you can derive as many views as you need.

The Saga Pattern: Distributed Transactions Without Two-Phase Commit

In a microservices world, a single business operation often spans multiple services. Placing an order requires charging the customer (payment service), reserving inventory (inventory service), and scheduling delivery (shipping service). If any step fails, the preceding steps must be rolled back.

Traditional databases solve this with ACID transactions — all operations succeed or all are rolled back. But distributed transactions across microservices using two-phase commit (2PC) are fragile, slow, and create tight coupling between services. The coordinator must wait for all participants, and if the coordinator fails during the commit phase, the system can be left in an inconsistent state.

The saga pattern replaces a single distributed transaction with a sequence of local transactions, each with a compensating action that undoes its effect. Kafka coordinates the flow by passing messages between services.

Choreography vs Orchestration

There are two ways to implement sagas:

Choreography is decentralized. Each service publishes events after completing its step, and the next service reacts to those events. There is no central coordinator. This is simpler for short sagas (2-3 steps) but becomes hard to follow as the saga grows. Debugging a 7-step choreographed saga is like tracing a chain of dominos across a room.

Orchestration uses a central saga orchestrator that tells each service what to do and handles compensation on failure. The orchestrator is a single service that owns the saga logic, making it easier to understand, modify, and debug.

import json
from confluent_kafka import Producer, Consumer

producer = Producer({"bootstrap.servers": "kafka-1:9092"})

class OrderSagaOrchestrator:
    """Coordinates an order saga across payment, inventory, and shipping."""

    STEPS = [
        {"service": "payment", "action": "charge", "compensate": "refund"},
        {"service": "inventory", "action": "reserve", "compensate": "release"},
        {"service": "shipping", "action": "schedule", "compensate": "cancel"},
    ]

    def execute(self, order):
        completed_steps = []

        for step in self.STEPS:
            # Send command to the service
            command = {
                "saga_id": order["order_id"],
                "action": step["action"],
                "order": order
            }
            topic = f"{step['service']}.commands"
            producer.produce(topic, key=order["order_id"].encode(),
                           value=json.dumps(command).encode())
            producer.flush()

            # Wait for the service to respond
            reply = self._wait_for_reply(order["order_id"], step["service"])

            if reply["status"] == "success":
                completed_steps.append(step)
            else:
                # Step failed -- compensate all completed steps in reverse
                self._compensate(order, completed_steps)
                return {"status": "failed", "reason": reply.get("reason"),
                        "failed_at": step["service"]}

        return {"status": "completed", "order_id": order["order_id"]}

    def _compensate(self, order, completed_steps):
        """Undo all completed steps in reverse order."""
        for step in reversed(completed_steps):
            compensate_command = {
                "saga_id": order["order_id"],
                "action": step["compensate"],
                "order": order
            }
            topic = f"{step['service']}.commands"
            producer.produce(topic, key=order["order_id"].encode(),
                           value=json.dumps(compensate_command).encode())
        producer.flush()

    def _wait_for_reply(self, order_id, service):
        """Listen for a reply from the service."""
        reply_consumer = Consumer({
            "bootstrap.servers": "kafka-1:9092",
            "group.id": f"saga-{order_id}-{service}",
            "auto.offset.reset": "latest",
        })
        reply_consumer.subscribe([f"{service}.replies"])
        try:
            while True:
                msg = reply_consumer.poll(5.0)
                if msg and not msg.error():
                    reply = json.loads(msg.value())
                    if reply.get("saga_id") == order_id:
                        return reply
        finally:
            reply_consumer.close()

Real-World Example: Order Processing Pipeline

Here is how a complete order processing pipeline works with the saga pattern and Kafka:

User places order
    |
    v
Order Service --> publishes "OrderCreated" to Kafka
    |
    v
Saga Orchestrator reads "OrderCreated"
    |
    |--> sends "ChargePayment" to payment.commands
    |    Payment Service processes, publishes to payment.replies
    |
    |--> sends "ReserveInventory" to inventory.commands
    |    Inventory Service processes, publishes to inventory.replies
    |
    |--> sends "ScheduleShipment" to shipping.commands
    |    Shipping Service processes, publishes to shipping.replies
    |
    v
All succeeded? --> publishes "OrderCompleted"
Any failed?   --> compensates completed steps, publishes "OrderFailed"

Each service owns its own database and its own local transaction. The saga orchestrator only coordinates the sequence through Kafka. If the payment succeeds but inventory reservation fails because the item is out of stock, the orchestrator sends a “RefundPayment” compensating command. The customer is refunded, and the order is marked as failed with a clear reason.

When EDA Is Overkill and When It Shines

Event-driven architecture is powerful, but it is not the right choice for every system. Applying EDA to a simple CRUD application is like using a sledgehammer to hang a picture frame. Here is a honest assessment of when it helps and when it hurts.

EDA Shines When

• Multiple services need the same data. If five services need to know about new orders, events let them all consume independently without the order service knowing about any of them.
• You need an audit trail. Event sourcing gives you a complete, immutable history of every state change, which is invaluable for compliance, debugging, and analytics.
• Services have different availability requirements. If the notification service goes down, events accumulate in Kafka. When it comes back, it catches up. No data is lost, and the order service was never affected.
• You need to scale components independently. The write path can run on small, fast machines while read model builders run on larger instances with more memory. Each scales independently.
• Your domain is inherently event-driven. IoT sensor data, financial transactions, user activity streams — these are naturally sequences of events.

EDA Is Overkill When

• You have a monolith with 2-3 services. The overhead of Kafka, event schemas, consumer groups, and eventual consistency is not justified.
• Strong consistency is critical everywhere. EDA is inherently eventually consistent. If every read must reflect the latest write immediately, the CQRS read model lag (even if it is just milliseconds) is unacceptable.
• The team is small and unfamiliar with EDA. Debugging event-driven systems requires different skills than debugging request-response systems. Event ordering, consumer lag, partition rebalancing, and eventual consistency bugs are genuinely hard to reason about.
• You are building a simple CRUD API. A REST API backed by PostgreSQL is simpler, faster to build, and easier to maintain for straightforward create-read-update-delete operations.

The best approach is to start simple. Build your system with direct service-to-service calls. When you hit a specific pain point — tight coupling, scaling bottleneck, lost audit history — introduce EDA for that specific boundary. Do not architect your entire system as event-driven on day one because a blog post told you to.

Next Steps

Event-driven architecture is a set of patterns, not a technology. Kafka is one implementation, but the patterns apply regardless of the messaging system. Understanding the “why” behind each pattern helps you apply them judiciously rather than dogmatically.

• Exactly-Once Semantics — ensure your event-driven pipeline processes each event exactly once
• Monitoring and Operations — monitor the health of your event-driven system in production