Kafka Real-World Use Cases: How Top Companies Use It

Why Real-World Examples Matter More Than Documentation

Kafka’s official documentation tells you what each configuration does. Blog posts explain how to set up a cluster. But understanding how Kafka actually works in production — at scale, under pressure, with real business constraints — requires studying the companies that push it to its limits.

These are not synthetic benchmarks or toy examples. These are systems handling millions of events per second, where downtime costs millions of dollars per minute, and where the engineering teams have spent years learning what works and what does not. Their lessons, mistakes, and architectural decisions are invaluable for anyone building Kafka-based systems.

Netflix: 8 Million Events Per Second for Recommendations

Netflix serves over 280 million subscribers across 190 countries, and nearly everything a user does generates an event. Play a movie, pause it, scroll through the catalog, rate a show, search for an actor — each interaction is captured, processed, and fed into the recommendation engine that powers 80% of what users watch.

The Architecture

Netflix’s Kafka deployment processes over 8 million events per second at peak. The architecture is built around a concept they call the data highway — a central Kafka cluster that acts as the universal transport layer for all event data.

When a user presses play on their TV, the Netflix client sends an event to the nearest edge server. The edge server batches these events and forwards them to a regional Kafka cluster. From there, events flow through multiple processing stages:

1. Real-time personalization: Kafka Streams applications process viewing events in real-time to update user profiles and recommendation models. When you finish watching a thriller, the recommendation model adjusts immediately — you will see more thrillers in your feed within seconds.

2. A/B test analytics: Every UI change, algorithm tweak, and content placement is an A/B test at Netflix. Events are consumed by analytics services that compute metrics for each test variant in near-real-time, allowing product teams to see experiment results within minutes rather than waiting for nightly batch jobs.

3. Content delivery optimization: Viewing start events trigger CDN pre-positioning. If many users in a region start watching a new release, Kafka-driven automation pushes more copies of that content to nearby CDN nodes.

User Device --> Edge Server --> Regional Kafka Cluster
                                        |
                                        |--> Real-time ML pipeline (Flink)
                                        |--> A/B test metrics (Spark Streaming)
                                        |--> CDN optimization service
                                        |--> Data warehouse (S3 + Iceberg)
                                        |--> Content analytics dashboards

Key Engineering Decisions

Netflix runs Kafka on AWS using a custom deployment called Keystone. Several architectural decisions stand out:

Multi-cluster with routing. Rather than running one massive Kafka cluster, Netflix runs multiple clusters organized by data type and criticality. Critical business events (billing, authentication) run on dedicated clusters with higher replication factors and stricter durability settings. High-volume but lower-criticality events (UI interactions, playback metrics) run on clusters optimized for throughput with relaxed durability.

Tiered storage. Netflix was an early adopter of Kafka’s tiered storage capability. Recent data (hours to days) lives on local broker disks for fast access. Older data is automatically moved to S3, allowing infinite retention without provisioning massive disk volumes. A Flink job that needs to reprocess last month’s data reads it from S3 seamlessly.

Consumer isolation. A misbehaving consumer that falls far behind and starts reading cold data (data not in the page cache) can impact broker performance for all consumers. Netflix isolates such consumers by routing their fetch requests to follower replicas rather than the leader, preventing them from polluting the leader’s page cache.

Uber: Real-Time Trip Pricing and ETAs

Uber processes over 1 trillion messages per day across its Kafka infrastructure. Kafka powers some of Uber’s most latency-sensitive systems: dynamic pricing (surge pricing), real-time ETAs, fraud detection, and driver-rider matching.

Dynamic Pricing Pipeline

When you open the Uber app and request a ride, the price you see is not a simple formula. It is the output of a real-time pipeline that considers current demand, available drivers, traffic conditions, weather, events in the area, and historical patterns — all computed within milliseconds.

The pipeline works like this. Every driver’s app continuously publishes GPS location updates to a Kafka topic. Every rider’s price request is also published. A Flink stream processing application consumes both streams, maintains a real-time supply-demand index for each geographic area, and publishes pricing multipliers to a pricing topic. The ride-request service consumes the pricing topic to apply the correct multiplier when calculating the fare.

The critical requirement is latency. If the pricing pipeline falls behind by even 30 seconds, the prices shown to users will not reflect current conditions. During a major event (a concert letting out, a sports game ending), demand can spike from baseline to 10x in under a minute. The pricing pipeline must react in seconds, not minutes.

# Simplified version of a real-time supply-demand tracker
# In production, this runs as a Flink stateful streaming application

from collections import defaultdict
import json
import time
from confluent_kafka import Consumer, Producer

consumer = Consumer({
    "bootstrap.servers": "kafka-1:9092",
    "group.id": "supply-demand-tracker",
    "auto.offset.reset": "latest",
    "enable.auto.commit": True,
    "fetch.min.bytes": 1,           # Low latency: fetch immediately
    "fetch.wait.max.ms": 10,        # Don't wait for batches
})
consumer.subscribe(["driver.locations", "ride.requests"])

producer = Producer({"bootstrap.servers": "kafka-1:9092"})

# Track supply and demand per geographic hex cell
supply = defaultdict(int)   # hex_id -> available drivers
demand = defaultdict(int)   # hex_id -> ride requests in last 5 min

def compute_surge(hex_id):
    """Compute surge multiplier based on supply/demand ratio."""
    s = max(supply[hex_id], 1)
    d = max(demand[hex_id], 1)
    ratio = d / s
    if ratio < 1.0: return 1.0      # No surge
    if ratio < 2.0: return 1.2      # Light surge
    if ratio < 3.0: return 1.5      # Moderate surge
    if ratio < 5.0: return 2.0      # Heavy surge
    return 2.5                       # Maximum surge

try:
    while True:
        msg = consumer.poll(0.01)  # 10ms poll timeout for low latency
        if msg is None or msg.error():
            continue

        event = json.loads(msg.value())
        hex_id = event.get("hex_id")

        if msg.topic() == "driver.locations":
            if event.get("status") == "available":
                supply[hex_id] += 1
            else:
                supply[hex_id] = max(0, supply[hex_id] - 1)
        elif msg.topic() == "ride.requests":
            demand[hex_id] += 1

        # Publish updated surge multiplier
        surge = compute_surge(hex_id)
        producer.produce("pricing.surge", key=hex_id.encode(),
                        value=json.dumps({
                            "hex_id": hex_id,
                            "surge_multiplier": surge,
                            "supply": supply[hex_id],
                            "demand": demand[hex_id],
                            "computed_at": time.time()
                        }).encode())
        producer.poll(0)
except KeyboardInterrupt:
    pass
finally:
    consumer.close()

Uber’s Kafka Lessons

Uber’s engineering blog has documented several hard-won lessons:

Dead letter queues are essential. In a system processing trillions of messages, even a 0.001% failure rate means millions of failed messages per day. Without DLQs, these failures would block the pipeline or be silently dropped. Uber routes failed messages to DLQ topics with full context (error reason, retry count, original timestamp) so they can be investigated and reprocessed.

Consumer group management at scale is hard. With thousands of consumer groups, rebalancing becomes a frequent event. Uber developed custom partition assignment strategies that minimize partition movement during rebalances, reducing the “stop-the-world” effect.

LinkedIn: The Birthplace of Kafka — 7 Trillion Messages Per Day

Kafka was born at LinkedIn in 2010 to solve a specific problem: how to move data between their growing number of systems without building point-to-point connections between every pair. Today, LinkedIn’s Kafka deployment processes over 7 trillion messages per day across hundreds of clusters, making it one of the largest Kafka deployments in the world.

What LinkedIn Uses Kafka For

LinkedIn’s Kafka handles an extraordinary variety of workloads:

Activity tracking: Every click, page view, search query, and profile view is an event flowing through Kafka. This is the original use case that Kafka was built for. The data powers the “Who viewed your profile” feature, content recommendations, and advertising targeting.

Operational metrics: LinkedIn uses Kafka to aggregate metrics from thousands of servers. System metrics (CPU, memory, disk, network) and application metrics (request latency, error rates, queue depths) all flow through Kafka to monitoring systems. This replaced a fragile system of log files and batch processing with a real-time pipeline.

Change Data Capture: LinkedIn uses Databus (their own CDC system) and Kafka to replicate database changes across datacenters. When a user updates their profile in one datacenter, the change flows through Kafka to all other datacenters within seconds.

Newsfeed generation: When you post an article on LinkedIn, it needs to appear in the feeds of your connections, followers, and potentially millions of other users based on algorithmic relevance. Kafka delivers the post event to the feed generation service, which computes who should see it and in what order.

LinkedIn’s Architectural Principles

Topic-per-use-case, not topic-per-entity. Early on, LinkedIn experimented with creating a topic for every entity type (users, companies, jobs, posts). This led to topic explosion and management overhead. They settled on broader topics organized by use case (activity-tracking, metrics, change-capture) with the entity type encoded in the message.

Rack-aware replication. LinkedIn runs Kafka across multiple racks in each datacenter. Partition replicas are spread across racks so that a single rack failure (power supply, top-of-rack switch) does not cause data loss. This is configured through the broker.rack property and Kafka’s rack-aware partition assignment.

Dedicated clusters by criticality. LinkedIn separates workloads into tiers. Tier-0 clusters handle revenue-critical data (ad delivery, premium subscriptions) with the strictest SLAs. Tier-1 handles user-facing features. Tier-2 handles analytics and batch workloads. Each tier has different replication factors, retention policies, and monitoring thresholds.

Fraud Detection: Real-Time Transaction Scoring

Financial institutions use Kafka as the backbone of real-time fraud detection systems. The challenge is evaluating every transaction — potentially millions per hour — within milliseconds, before the transaction is approved.

The Pipeline Architecture

A typical fraud detection pipeline looks like this:

Transaction     Feature         ML Model       Decision
Event   --->   Enrichment  --> Scoring   --->  Engine   ---> Approve/Deny
(Kafka)        (Flink)        (Kafka)         (Kafka)       (Response)
                  |                              |
                  v                              v
            User Profile DB              Alert Dashboard
            (Redis/Cassandra)            (Elasticsearch)

When a customer swipes their credit card, the point-of-sale system publishes a transaction event to Kafka. A Flink stream processing application enriches the transaction with context: the customer’s recent transaction history, their typical spending patterns, the merchant’s fraud rate, the geographic distance from the last transaction. This enriched event is published to another Kafka topic.

An ML scoring service consumes enriched transactions, runs them through a trained fraud detection model, and publishes a risk score. A decision engine consumes the scored transactions and either approves the transaction (score below threshold), denies it (score above threshold), or flags it for manual review (score in the gray zone).

# Simplified fraud scoring pipeline
import json
from confluent_kafka import Consumer, Producer

consumer = Consumer({
    "bootstrap.servers": "kafka-1:9092",
    "group.id": "fraud-scorer",
    "enable.auto.commit": False,
    "fetch.min.bytes": 1,
    "fetch.wait.max.ms": 5,        # Ultra-low latency
})
consumer.subscribe(["transactions.enriched"])
producer = Producer({"bootstrap.servers": "kafka-1:9092"})

def score_transaction(enriched_txn):
    """Run fraud detection model on enriched transaction."""
    features = {
        "amount": enriched_txn["amount"],
        "distance_from_last_txn_km": enriched_txn["geo_distance"],
        "time_since_last_txn_sec": enriched_txn["time_gap"],
        "merchant_fraud_rate": enriched_txn["merchant_risk"],
        "is_foreign": enriched_txn["is_foreign_txn"],
        "amount_vs_avg_ratio": (
            enriched_txn["amount"] / max(enriched_txn["avg_amount"], 1)
        ),
    }
    # In production, this calls a trained model (XGBoost, neural net, etc.)
    score = model.predict_proba(features)
    return score

try:
    while True:
        msg = consumer.poll(0.005)  # 5ms poll for real-time scoring
        if msg is None or msg.error():
            continue

        enriched_txn = json.loads(msg.value())
        risk_score = score_transaction(enriched_txn)

        decision = "approve" if risk_score < 0.3 else \
                   "review" if risk_score < 0.7 else "deny"

        result = {
            "transaction_id": enriched_txn["transaction_id"],
            "risk_score": risk_score,
            "decision": decision,
            "scored_at": time.time()
        }

        producer.produce("transactions.scored",
                        key=enriched_txn["transaction_id"].encode(),
                        value=json.dumps(result).encode())
        producer.poll(0)
        consumer.commit(message=msg)
except KeyboardInterrupt:
    pass
finally:
    consumer.close()

The critical metric for fraud detection is end-to-end latency — the time from card swipe to approve/deny decision. Most card networks require a response within 2 seconds. The Kafka pipeline must process the transaction, enrich it, score it, and make a decision within that window. This is why every consumer in the pipeline uses the lowest-latency settings: fetch.min.bytes=1, fetch.wait.max.ms=5, no batching.

IoT Data Ingestion: Millions of Sensor Events

IoT (Internet of Things) deployments generate enormous volumes of small messages from sensors, devices, and machines. A single manufacturing plant might have 10,000 sensors reporting temperature, pressure, vibration, and speed every second. A fleet of 50,000 vehicles reports GPS, engine diagnostics, and fuel levels every 5 seconds. Kafka is a natural fit for this workload because it excels at high-throughput ingestion of small messages.

The IoT Ingestion Pattern

IoT data typically flows through a gateway layer before reaching Kafka. Devices speak protocols like MQTT, CoAP, or HTTP. An IoT gateway (such as an MQTT broker or a custom HTTP endpoint) collects device messages and publishes them to Kafka topics.

Sensors/Devices --> MQTT Broker --> Kafka Connect (MQTT Source)
                                         |
                                         v
                                    Kafka Topics
                                    (sensor-data)
                                         |
                    +--------------------+--------------------+
                    |                    |                    |
                    v                    v                    v
              Real-time alerts     Time-series DB      Data Lake (S3)
              (Flink/Streams)      (InfluxDB/         (long-term
                                    TimescaleDB)       analysis)

The key design decisions for IoT ingestion:

Topic partitioning strategy. Partition by device ID or sensor ID so that all readings from a single device are ordered within a partition. This enables per-device anomaly detection (e.g., “has this sensor’s temperature exceeded its normal range?”).

Compaction for device state. Use a compacted topic to maintain the latest state of each device. When a monitoring dashboard asks “what is the current temperature of sensor X?”, it reads the latest value from the compacted topic rather than querying the full event stream.

Edge pre-processing. Not all sensor data needs to reach the central Kafka cluster. Edge processing filters out noise (readings within normal range), aggregates high-frequency data (average temperature every 10 seconds instead of every 100ms), and only sends anomalies and summaries to the central cluster. This dramatically reduces bandwidth and Kafka throughput requirements.

Log Aggregation at Scale

Before Kafka, log aggregation meant installing agents on every server that shipped log files to a central system like Elasticsearch or Splunk. These agents had to handle backpressure (what happens when the central system is slow), buffering (what happens during network outages), and routing (which logs go where). Each agent was another piece of software to configure, monitor, and update.

Kafka simplifies this by becoming the central log bus. Agents on each server publish log lines to Kafka topics. Downstream consumers route logs to their final destinations: Elasticsearch for search and dashboards, S3 for long-term archival, a real-time alerting service for error detection. If Elasticsearch falls behind, logs accumulate in Kafka rather than being lost or causing backpressure on the application servers.

The key advantage is decoupling ingestion from processing. Application servers produce logs at whatever rate they generate them. Kafka absorbs the volume. Downstream systems consume at their own pace. If you need to add a new log destination (a new SIEM tool, a compliance archive), you add a new consumer. No changes to the application servers or the log agents.

Log Aggregation Architecture

# Log producer: runs on each application server
import logging
import json
from confluent_kafka import Producer

class KafkaLogHandler(logging.Handler):
    """Custom logging handler that publishes log records to Kafka."""

    def __init__(self, bootstrap_servers, topic, service_name):
        super().__init__()
        self.producer = Producer({
            "bootstrap.servers": bootstrap_servers,
            "acks": "1",             # Balanced durability for logs
            "linger.ms": 100,        # Batch logs for efficiency
            "batch.size": 65536,
            "compression.type": "lz4",
        })
        self.topic = topic
        self.service_name = service_name

    def emit(self, record):
        log_entry = {
            "timestamp": record.created,
            "level": record.levelname,
            "service": self.service_name,
            "logger": record.name,
            "message": record.getMessage(),
            "hostname": record.hostname if hasattr(record, "hostname") else None,
            "trace_id": getattr(record, "trace_id", None),
        }
        self.producer.produce(
            self.topic,
            key=self.service_name.encode(),
            value=json.dumps(log_entry).encode()
        )
        self.producer.poll(0)

# Usage
logger = logging.getLogger("order-service")
logger.addHandler(KafkaLogHandler(
    bootstrap_servers="kafka-1:9092",
    topic="logs.application",
    service_name="order-service"
))

logger.info("Order created", extra={"trace_id": "abc-123"})
logger.error("Payment failed: timeout", extra={"trace_id": "abc-123"})

Lessons Learned: Common Mistakes Companies Make with Kafka

After studying dozens of Kafka deployments across industries, several patterns of mistakes emerge. These are not edge cases — they are the same mistakes made by teams of all sizes.

Mistake 1: Treating Kafka Like a Database

Kafka is a log, not a database. It is optimized for sequential writes and reads, not random lookups. Teams that use Kafka as their primary data store (expecting to query individual records by key) are fighting the tool. Use Kafka as the transport layer and materialize data into a database (Postgres, Cassandra, Redis) for queries.

Mistake 2: Too Many Partitions from Day One

It is tempting to create topics with 100 partitions “for future scale.” But each partition has overhead: open file handles, replication threads, and memory. A topic with 100 partitions and 10 messages per day wastes resources. Start with 6-12 partitions and increase when you have evidence that you need more parallelism. Remember: you can always add partitions, but you cannot reduce them.

Mistake 3: Ignoring Consumer Lag Until It Is a Crisis

Consumer lag is the single most important operational metric, yet many teams do not monitor it until users complain about stale data. By then, the consumer might be hours behind, and catching up could take just as long. Set up consumer lag monitoring and alerting from day one.

Mistake 4: Not Planning for Schema Evolution

The first version of your message format will not be the last. Teams that use raw JSON without a schema registry discover this painfully when a producer changes a field name and breaks every consumer. Use the Schema Registry with Avro or Protobuf from the start. The upfront investment is small compared to the cost of coordinating schema changes across 15 consumer teams.

Mistake 5: Running Kafka Without Monitoring

“It works on my laptop” is not a production readiness statement. Teams that deploy Kafka without Prometheus, Grafana, and alerting rules are operating blind. Under-replicated partitions, disk space exhaustion, and consumer group failures will happen. The question is whether you detect them in minutes or in hours.

Mistake 6: Using Kafka for Request-Reply

Kafka is built for asynchronous, one-directional data flow. Using it for synchronous request-reply patterns (publish a message, wait for a response on another topic) is possible but awkward. The latency is higher than HTTP or gRPC, and the complexity is significantly greater. Use the right tool for the job: Kafka for event streaming, gRPC for synchronous communication.

Next Steps

These case studies demonstrate that Kafka is not just a messaging system — it is an infrastructure primitive that enables entirely new architectural patterns. But the common thread across all successful deployments is disciplined operations: monitoring, schema management, capacity planning, and error handling. The technology works. The challenge is operating it well.

• Monitoring and Operations — set up the monitoring that every production Kafka deployment needs
• Building Data Pipelines — build the data pipelines these companies rely on
• KRaft and Cluster Management — manage clusters at the scale these companies operate