Real-Time Data Streaming: Architectures and Patterns

Real-time data streaming is not just “batch but faster.” It is a fundamentally different way of thinking about data: instead of asking “what happened yesterday?”, you ask “what is happening right now?” This shift changes your architecture, your tooling, and your trade-offs.

Streaming architectures

Lambda architecture

Lambda runs batch and streaming in parallel and merges results at query time.

                    ┌─── Batch Layer (Spark) ──── Batch View ───┐
                    │                                            │
Raw Events ─────────┤                                            ├─── Query Layer
                    │                                            │
                    └─── Speed Layer (Flink) ──── Real-time View ┘

The batch layer reprocesses all historical data periodically to produce accurate, complete views. The speed layer handles only recent events for low-latency results. A serving layer merges both.

When to use Lambda: You need sub-second dashboards but also run complex ML training on historical data. Financial services and ad-tech commonly use this pattern.

The catch: You maintain two codepaths for the same business logic. When your definition of “active user” changes, you update it in both Spark and Flink.

Kappa architecture

Kappa eliminates the batch layer entirely. Everything flows through the stream processor. Historical reprocessing works by replaying the event log from the beginning.

Event Log (Kafka, infinite retention)
        │
        ▼
Stream Processor (Flink / Spark Structured Streaming)
        │
        ▼
Serving Layer (Druid / ClickHouse / Postgres)

When to use Kappa: Your transformations are simple enough to express in a single streaming job, and you can afford to replay the log for backfills.

The catch: Replaying petabytes through a stream processor is slower and more expensive than a batch engine optimized for it.

Choosing between them

Factor	Lambda	Kappa
Codebase	Two (batch + stream)	One (stream only)
Reprocessing speed	Fast (batch engine)	Slower (log replay)
Operational overhead	Higher (two systems)	Lower (one system)
Best for	Complex analytics + real-time	Event-driven microservices

Change Data Capture (CDC)

CDC captures row-level changes from a database and publishes them as events. Instead of polling a table every hour, you stream every INSERT, UPDATE, and DELETE as it happens.

Log-based CDC with Debezium

Debezium reads the database’s transaction log (WAL in Postgres, binlog in MySQL) and pushes change events to Kafka.

{
  "op": "u",
  "before": { "id": 42, "status": "pending", "amount": 100.00 },
  "after":  { "id": 42, "status": "shipped", "amount": 100.00 },
  "source": {
    "connector": "postgresql",
    "db": "orders_db",
    "table": "orders",
    "ts_ms": 1723190400000
  }
}

Setting up Debezium

Register a connector with Kafka Connect:

{
  "name": "orders-connector",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname": "db.internal",
    "database.port": "5432",
    "database.user": "replicator",
    "database.password": "${REPLICATOR_PASSWORD}",
    "database.dbname": "orders_db",
    "table.include.list": "public.orders,public.customers",
    "topic.prefix": "cdc",
    "plugin.name": "pgoutput",
    "slot.name": "debezium_slot",
    "publication.name": "dbz_publication"
  }
}

This produces Kafka topics like cdc.public.orders with one event per row change.

CDC patterns in practice

Streaming to a data warehouse:

from pyspark.sql import SparkSession
from pyspark.sql.functions import col, from_json, schema_of_json

spark = SparkSession.builder.appName("CDC-to-Warehouse").getOrCreate()

cdc_stream = (
    spark.readStream
    .format("kafka")
    .option("kafka.bootstrap.servers", "broker:9092")
    .option("subscribe", "cdc.public.orders")
    .option("startingOffsets", "earliest")
    .load()
)

# Parse the Debezium envelope
orders = (
    cdc_stream
    .selectExpr("CAST(value AS STRING) as json_str")
    .select(from_json(col("json_str"), cdc_schema).alias("data"))
    .select(
        col("data.after.id").alias("order_id"),
        col("data.after.status"),
        col("data.after.amount"),
        col("data.op").alias("operation")
    )
)

# Upsert into Iceberg table
query = (
    orders.writeStream
    .format("iceberg")
    .outputMode("append")
    .option("checkpointLocation", "/checkpoints/orders_cdc")
    .toTable("warehouse.orders_snapshot")
)

Common CDC use cases:

• Replicate operational databases to a data warehouse in near-real-time
• Invalidate caches when underlying data changes
• Trigger downstream workflows on specific data changes
• Keep search indexes (Elasticsearch) in sync with the source of truth

Event sourcing

Event sourcing stores every state change as an immutable event, rather than overwriting rows. The current state is derived by replaying all events.

Traditional CRUD vs event sourcing

CRUD approach — you overwrite the row:

UPDATE orders SET status = 'shipped', updated_at = NOW()
WHERE id = 42;
-- Previous state is lost

Event sourcing approach — you append an event:

events = [
    {"event": "OrderPlaced",   "order_id": 42, "amount": 100, "ts": "2026-08-09T10:00:00Z"},
    {"event": "PaymentReceived","order_id": 42, "amount": 100, "ts": "2026-08-09T10:05:00Z"},
    {"event": "OrderShipped",  "order_id": 42, "carrier": "FedEx", "ts": "2026-08-09T14:30:00Z"},
]

# Current state = replay all events for order 42
def get_order_state(order_id: int) -> dict:
    state = {}
    for event in get_events(order_id):
        if event["event"] == "OrderPlaced":
            state = {"id": order_id, "amount": event["amount"], "status": "placed"}
        elif event["event"] == "PaymentReceived":
            state["status"] = "paid"
        elif event["event"] == "OrderShipped":
            state["status"] = "shipped"
            state["carrier"] = event["carrier"]
    return state

When event sourcing makes sense

• Audit trails — financial systems, healthcare, compliance.
• Temporal queries — “what was the state of this order at 3 PM?”
• Debugging — replay events to reproduce bugs.
• Event-driven architectures — events are the natural interface between services.

When it does not

• Simple CRUD apps with no audit requirements.
• High-frequency updates where event log growth becomes a storage problem.
• Teams without experience managing event stores.

Streaming vs batch: the decision framework

Do not default to streaming. Ask these questions:

1. What latency does the business actually need?

If the answer is “daily report by 9 AM,” batch is simpler, cheaper, and correct. If the answer is “detect fraud within 5 seconds,” you need streaming.

2. Is the data source naturally event-driven?

Clickstreams, IoT sensors, and transaction logs are naturally events. A CSV file uploaded to S3 once a day is naturally a batch.

3. Can the team operate a streaming system?

Streaming infrastructure (Kafka, Flink, schema registry, monitoring) is significantly more complex to operate than a batch pipeline. Be honest about your team’s capabilities.

4. What is the cost tolerance?

Streaming compute runs 24/7. A Flink cluster processing 10K events/second costs materially more than a Spark job that runs for 20 minutes daily.

The hybrid reality

Most production data platforms combine both:

                    ┌── Kafka ── Flink ── Real-time dashboards
                    │
Event Sources ──────┤
                    │
                    └── Kafka ── S3 (landing) ── Spark/dbt ── Warehouse

Real-time events flow through Kafka to Flink for immediate use cases (alerts, fraud, live metrics). The same events land in object storage for batch processing, ML training, and historical analysis. This is not Lambda architecture — it is two independent pipelines serving different use cases from the same event stream.

Key takeaways

• Lambda gives you both accuracy and freshness but at the cost of dual maintenance.
• Kappa simplifies to a single codebase but trades off reprocessing speed.
• CDC turns database changes into a real-time event stream without modifying application code.
• Event sourcing stores every change as an immutable event, enabling audit trails and temporal queries.
• Default to batch unless you have a clear, business-justified need for sub-minute latency.

Next steps

• Batch vs Stream Processing — the foundational comparison.
• Apache Spark Fundamentals — the batch and micro-batch engine.
• Data Pipeline Testing — how to test streaming pipelines.