Kafka Streams: Real-Time Stream Processing Without a Cluster

Why Stream Processing Matters

Imagine two ways of getting the news. You could wait for tomorrow’s newspaper to arrive on your doorstep — that is batch processing. You wrote down everything that happened during the day, bundled it up, and delivered it hours later. Or you could watch live television, where you see events as they unfold in real time — that is stream processing.

In software systems, the difference has real consequences. A batch system that processes yesterday’s fraud detection data tells you about fraud that already happened. A streaming system flags the suspicious transaction while it is still happening, giving you a chance to block it. A batch system that aggregates website metrics gives you yesterday’s dashboard. A streaming system shows you what is happening on your website right now.

Apache Kafka is already a streaming platform for transporting data. But transporting data is only half the problem. You also need to transform, aggregate, filter, join, and enrich that data as it flows through. That is where Kafka Streams comes in.

What is Kafka Streams?

Kafka Streams is a Java library (not a separate system) for building stream processing applications that read from Kafka, transform data, and write results back to Kafka. This distinction is important: unlike heavyweight frameworks like Apache Flink or Spark Streaming, Kafka Streams does not require you to set up and maintain a separate processing cluster.

Your Kafka Streams application is just a regular Java application. You package it as a JAR, deploy it however you deploy any Java app (Docker, Kubernetes, a plain server), and it handles everything internally. To scale it, you simply run more instances. Kafka Streams uses the same consumer group mechanism you learned about in the consumer article to distribute partitions across instances. If one instance crashes, the others pick up its partitions automatically.

This “just a library” approach has several practical advantages:

• No ops overhead — No ZooKeeper-like coordination service, no cluster manager, no separate monitoring infrastructure.
• Same deployment model as your other apps — It deploys through your existing CI/CD pipeline.
• Elastic scaling — Start one instance during development, run fifty in production. Add or remove instances at any time.
• Built-in fault tolerance — Local state is backed up to Kafka changelog topics and automatically restored on failure.

To add Kafka Streams to your Java project, include a single dependency:

<!-- Maven -->
<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-streams</artifactId>
    <version>3.7.0</version>
</dependency>

KStream vs KTable: The Event Log vs Current State

Kafka Streams gives you two fundamental ways to think about data, and understanding the difference is essential for building correct stream processing applications. The analogy that makes this click is a bank account.

KStream: Every Transaction Ever

A KStream represents an unbounded sequence of independent events, where each event stands on its own. Think of it as your bank’s transaction history: “deposited $100,” “withdrew $30,” “deposited $500,” “withdrew $200.” Every transaction is a fact that happened at a point in time. You never “update” a previous transaction — you only add new ones to the log. In database terms, a KStream is a sequence of INSERT operations.

When would you use a KStream? Whenever each record represents an event that happened: a user click, a sensor reading, a log entry, a purchase. The full history matters, and no record replaces a previous one.

KTable: Your Current Balance

A KTable represents the current state for each key. Think of it as your bank balance. You do not care about every transaction that led to your balance of $370 — you just want to know the current number. When a new record arrives with a key that already exists, the new record replaces the old one. In database terms, a KTable is a sequence of UPSERT operations: insert if new, update if existing.

When would you use a KTable? Whenever you care about the latest value for a key: a user’s current profile, the latest price of a stock, the most recent configuration for a device. The current state matters, not the history.

GlobalKTable: A Lookup Table Available Everywhere

A GlobalKTable is a special variant of KTable where every instance of your application gets a complete copy of all data. Normal KTables are partitioned — each application instance only sees the data for the partitions assigned to it. A GlobalKTable replicates everything to every instance.

This is useful for small reference data that needs to be available for joins regardless of partitioning: country codes, currency exchange rates, product category mappings. The table must be small enough to fit in memory on each instance.

The Word Count Example: Step by Step

The word count example is to stream processing what “Hello World” is to programming. It takes lines of text as input and continuously counts how many times each word appears. Let us walk through every step in detail.

Before looking at the code, understand the data flow. Text lines arrive in a Kafka topic called text-input. Our application reads each line, splits it into individual words, groups those words by their identity (the word itself), and counts occurrences. The running word counts are written to an output topic called word-counts-output.

The important thing to realize is that this is a continuously updating computation. When a new line arrives, the counts are updated immediately. If the word “kafka” has been seen 47 times and a new line contains “kafka” twice, the count updates to 49 and that new count is published to the output topic.

import org.apache.kafka.streams.*;
import org.apache.kafka.streams.kstream.*;
import org.apache.kafka.common.serialization.Serdes;
import java.util.Arrays;
import java.util.Properties;

public class WordCountApp {
    public static void main(String[] args) {
        // --- Configuration ---
        Properties props = new Properties();
        // A unique identifier for this application. Kafka uses this as
        // the consumer group ID and as a prefix for internal topics.
        props.put(StreamsConfig.APPLICATION_ID_CONFIG, "wordcount-app");
        props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        // Default serializers/deserializers for keys and values.
        props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
            Serdes.String().getClass());
        props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
            Serdes.String().getClass());
        // Enable exactly-once processing (explained later).
        props.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG,
            StreamsConfig.EXACTLY_ONCE_V2);

        // --- Build the processing topology ---
        StreamsBuilder builder = new StreamsBuilder();

        // Step 1: Read lines of text from the input topic as a KStream.
        // Each record has a key (which we ignore) and a value (a line of text).
        KStream<String, String> textLines = builder.stream("text-input");

        // Step 2: Transform, group, and count.
        KTable<String, Long> wordCounts = textLines
            // flatMapValues: Split each line into words. One input line
            // produces multiple output records (one per word).
            .flatMapValues(line ->
                Arrays.asList(line.toLowerCase().split("\\W+")))
            // filter: Remove empty strings that result from splitting.
            .filter((key, word) -> !word.isEmpty())
            // groupBy: Re-key each record so the word itself becomes the key.
            // Records with the same key (same word) are routed to the
            // same partition for counting.
            .groupBy((key, word) -> word,
                Grouped.with(Serdes.String(), Serdes.String()))
            // count: Count records per key. This produces a KTable because
            // the count for each word is continuously updated (upsert).
            // The state is stored in a named state store for fault tolerance.
            .count(Materialized.as("word-counts-store"));

        // Step 3: Write the KTable back to an output topic.
        // toStream() converts the KTable to a KStream so it can be written.
        wordCounts.toStream().to("word-counts-output",
            Produced.with(Serdes.String(), Serdes.Long()));

        // --- Start the application ---
        KafkaStreams streams = new KafkaStreams(builder.build(), props);
        Runtime.getRuntime().addShutdownHook(new Thread(streams::close));
        streams.start();
    }
}

Let us trace what happens when the line “hello kafka hello” arrives. The flatMapValues step splits it into three records: “hello”, “kafka”, “hello”. The filter step passes all three through (none are empty). The groupBy step re-keys them so “hello” records are grouped together and “kafka” records are grouped separately. The count step increments the count for “hello” to 2 and “kafka” to 1. These updated counts are written to the output topic.

When the next line “kafka streams kafka” arrives, the counts update to: “hello” = 2, “kafka” = 3, “streams” = 1. The count operation maintains a running total in a state store, updating it incrementally with each new record rather than recomputing from scratch.

Windowed Aggregations: Counting Over Time Periods

Aggregations like count() produce a single running total across all time. But what if you need to count events in specific time periods — like “how many orders per hour?” or “how many login attempts per 5-minute window?”

Think of it like a store manager counting customers. Counting total customers ever is useful for annual reports, but counting customers per hour tells you when to schedule more staff. Windowed aggregations give you that time-based view.

Tumbling Windows: Fixed, Non-Overlapping Buckets

Tumbling windows divide time into fixed-size, non-overlapping periods. Each event belongs to exactly one window. Think of it as dividing a day into 24 one-hour buckets. A purchase at 2:15 PM goes into the 2:00-3:00 PM bucket, and that bucket has hard boundaries.

This is the most common windowing strategy and is suitable for periodic reporting: hourly sales totals, daily active users, five-minute error counts.

// Count events per user in 5-minute tumbling windows.
// Each event falls into exactly one 5-minute bucket.
KTable<Windowed<String>, Long> eventsPerWindow = events
    .groupByKey()
    .windowedBy(TimeWindows.ofSizeWithNoGrace(Duration.ofMinutes(5)))
    .count(Materialized.as("events-per-5min"));

Hopping Windows: Overlapping Buckets

Hopping windows are like tumbling windows, but they overlap. A 10-minute window that “hops” every 2 minutes means a new window starts every 2 minutes, and each window covers the last 10 minutes. A single event might fall into 5 different windows.

This is useful for smoothing out spikes. Instead of seeing a dramatic jump at a window boundary, hopping windows give you a rolling view. Think of it like a moving average in finance.

Session Windows: Activity-Based Grouping

Session windows are different from the others because they are not fixed-size. A session window groups events that occur near each other in time, with a configurable inactivity gap. If no events arrive for a user within the gap period, the session closes.

This maps naturally to user sessions on a website. A user clicks around for a while (events cluster together), then goes to lunch (a 30-minute gap of inactivity), then comes back. Session windows would capture the morning activity as one session and the afternoon activity as another.

// Group user clicks into sessions. A session ends after 30 minutes
// of inactivity. Each user has independent sessions.
KTable<Windowed<String>, Long> userSessions = clickstream
    .groupByKey()
    .windowedBy(SessionWindows.ofInactivityGapWithNoGrace(
        Duration.ofMinutes(30)))
    .count(Materialized.as("user-sessions"));

Stream-Table Joins: Enriching Events with Context

One of the most powerful patterns in Kafka Streams is joining a stream of events with a table of reference data. This is how you enrich raw events with context.

Consider an e-commerce system. Your orders stream contains bare-bones order data: customer ID, product ID, amount. Your customer-profiles table has rich customer information: name, loyalty tier, region. You want to produce enriched orders that combine both.

In traditional systems, you would query a database for each order. With Kafka Streams, the customer profile table is maintained locally (loaded from a Kafka topic), so the join is a fast, in-memory lookup with no external database call. When a customer updates their profile, the KTable updates automatically, and all subsequent orders for that customer are enriched with the latest information.

// Stream of orders, keyed by customer_id.
KStream<String, Order> orders = builder.stream("orders");

// Table of customer profiles, keyed by customer_id.
// This is continuously updated as profiles change.
KTable<String, CustomerProfile> customers =
    builder.table("customer-profiles");

// Join each order with the customer's current profile.
// The join key is the record key (customer_id) in both the
// stream and the table.
KStream<String, EnrichedOrder> enrichedOrders = orders.join(
    customers,
    (order, customer) -> new EnrichedOrder(
        order.getOrderId(),
        order.getAmount(),
        customer.getName(),
        customer.getTier(),
        customer.getRegion()
    )
);

enrichedOrders.to("enriched-orders");

The key requirement is that both the stream and the table must be keyed by the same field (customer_id in this case) and co-partitioned — meaning they use the same number of partitions and the same partitioning strategy. If they are not co-partitioned, you need a GlobalKTable (which replicates all data to every instance) instead.

Exactly-Once Semantics: Solving the Double-Charge Problem

Here is a scenario that illustrates why exactly-once processing matters. Your stream processing application reads a “charge customer $50” event, processes it (calls the payment gateway), and writes a “payment confirmed” event to the output topic. But at the exact moment between processing and writing the output, your application crashes.

When the application restarts, it re-reads the “charge customer $50” event (because the offset was never committed). It processes it again, charging the customer a second time. Now you have a double charge. The customer is unhappy, your support team is overwhelmed, and your finance department has a reconciliation nightmare.

Exactly-once semantics (EOS) prevent this by making the entire cycle — reading input, updating state, writing output, committing offsets — a single atomic transaction. Either all of it happens, or none of it does. If the application crashes in the middle, the transaction is rolled back, and on restart, it re-processes the event exactly once.

Enabling EOS in Kafka Streams is remarkably simple — a single configuration line:

props.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG,
    StreamsConfig.EXACTLY_ONCE_V2);

Under the hood, this enables transactional producers, atomic offset commits, and producer fencing (preventing “zombie” instances that might still be running after a crash from writing duplicate data). The performance overhead is minimal — typically a single-digit percentage increase in latency — and it is recommended for all production deployments where correctness matters.

It is worth noting that exactly-once applies within the Kafka ecosystem: reading from Kafka, processing, and writing back to Kafka. If your processing involves an external system (like calling a payment API), you still need to handle idempotency on that external call yourself.

State Stores: Where Kafka Streams Keeps Its Memory

Operations like count(), aggregate(), and joins require the application to remember things. The running count of each word, the latest customer profile, the total revenue per region — all of this is state. Kafka Streams manages this state automatically using local state stores backed by RocksDB, an embedded key-value database.

The clever part is the backup strategy. Every state store has a corresponding changelog topic in Kafka. Every state update is also written to this changelog. If an application instance crashes and restarts (possibly on a different machine), it replays the changelog topic to rebuild the exact state it had before the crash. This makes Kafka Streams applications fully fault-tolerant without requiring an external database.

You can also query state stores directly from your application for real-time lookups. This enables a powerful pattern: build a stream processing pipeline that continuously computes results, and expose those results through a REST API for dashboards or other services.

// After starting the streams application, query the word count
// state store directly -- no need to read from an output topic.
ReadOnlyKeyValueStore<String, Long> wordCountStore =
    streams.store(
        StoreQueryParameters.fromNameAndType(
            "word-counts-store",
            QueryableStoreTypes.keyValueStore()
        )
    );

// Fast point lookup: "How many times has 'kafka' appeared?"
Long count = wordCountStore.get("kafka");

This interactive query capability turns your Kafka Streams application into both a processing engine and a queryable data store, which can eliminate the need for a separate database in many architectures.

Summary

Kafka Streams brings stream processing directly into your application, eliminating the need for a separate processing cluster. The key concepts to take away are:

• KStream vs KTable — Use KStream for event logs where every record matters. Use KTable when you only care about the latest value per key.
• Windowed aggregations — Tumbling windows for periodic reports, hopping windows for smoothed rolling metrics, session windows for user activity sessions.
• Stream-table joins — Enrich events with reference data using fast, local lookups instead of database queries.
• Exactly-once semantics — A single configuration line prevents duplicate processing, double-charges, and data inconsistencies.
• State stores — Automatically managed, fault-tolerant local state with optional direct querying for real-time dashboards.

Next Steps

• Learn how to move data in and out of Kafka with Kafka Connect
• Understand the consumer side in Consumers and Consumer Groups