Message Queues: RabbitMQ, Kafka, and SQS Compared

Message queues sit between producers and consumers, buffering work so that services do not need to communicate synchronously. They handle backpressure, retry failed processing, and decouple services so that a slow consumer does not bring down a fast producer. But RabbitMQ, Kafka, and SQS solve this problem in fundamentally different ways.

Why message queues matter

Without a queue, service-to-service communication looks like this.

# Direct HTTP call - tightly coupled
async def process_order(order: Order):
    # If email service is down, order processing fails
    await http_client.post("http://email-service/send", json={
        "to": order.user_email,
        "template": "order_confirmation",
        "data": order.to_dict(),
    })

    # If analytics is slow, order processing is slow
    await http_client.post("http://analytics-service/events", json={
        "type": "order_placed",
        "data": order.to_dict(),
    })

With a queue, you publish a message and move on. Consumers process it when they can.

# With a message queue - decoupled
async def process_order(order: Order):
    await queue.publish("order.placed", order.to_dict())
    # Done. Email service and analytics pick up the message independently.

The key benefits are temporal decoupling (producer and consumer do not need to be running at the same time), load leveling (the queue absorbs spikes), and fan-out (multiple consumers can process the same event).

RabbitMQ: the traditional message broker

RabbitMQ implements the AMQP protocol and follows the traditional message broker model. Messages are routed through exchanges to queues based on routing keys and bindings.

Core concepts

• Exchange: Receives messages from producers and routes them to queues
• Queue: Stores messages until a consumer processes them
• Binding: Rules that connect exchanges to queues
• Consumer: Pulls messages from queues and acknowledges processing

Producer example

import pika
import json

def publish_order_event(order_data: dict):
    connection = pika.BlockingConnection(
        pika.ConnectionParameters(host="localhost")
    )
    channel = connection.channel()

    # Declare a topic exchange
    channel.exchange_declare(
        exchange="orders",
        exchange_type="topic",
        durable=True,
    )

    # Publish with routing key
    channel.basic_publish(
        exchange="orders",
        routing_key="order.placed",
        body=json.dumps(order_data),
        properties=pika.BasicProperties(
            delivery_mode=2,  # Persistent message
            content_type="application/json",
        ),
    )

    connection.close()

Consumer example

import pika
import json

def start_email_consumer():
    connection = pika.BlockingConnection(
        pika.ConnectionParameters(host="localhost")
    )
    channel = connection.channel()

    # Declare the queue
    channel.queue_declare(queue="email_notifications", durable=True)

    # Bind to the exchange with a routing pattern
    channel.queue_bind(
        exchange="orders",
        queue="email_notifications",
        routing_key="order.*",  # Matches order.placed, order.shipped, etc.
    )

    # Prefetch 1 message at a time
    channel.basic_qos(prefetch_count=1)

    def callback(ch, method, properties, body):
        order = json.loads(body)
        try:
            send_confirmation_email(order)
            ch.basic_ack(delivery_tag=method.delivery_tag)
        except Exception:
            # Reject and requeue
            ch.basic_nack(
                delivery_tag=method.delivery_tag,
                requeue=True,
            )

    channel.basic_consume(
        queue="email_notifications",
        on_message_callback=callback,
    )

    channel.start_consuming()

When to use RabbitMQ

RabbitMQ excels at task distribution. When you have work that needs to be processed exactly once and distributed across multiple workers, RabbitMQ’s acknowledgment model handles this well. It also supports complex routing patterns through exchange types (direct, topic, fanout, headers), which is useful when different consumers need different subsets of messages.

Apache Kafka: the distributed event log

Kafka is not a traditional message queue. It is a distributed, append-only commit log. Messages are written to partitioned topics and retained for a configurable duration. Consumers track their position (offset) in the log.

Core concepts

• Topic: A category of messages, split into partitions
• Partition: An ordered, immutable sequence of messages
• Offset: A consumer’s position within a partition
• Consumer group: A set of consumers that share the work of reading a topic

Producer example

from confluent_kafka import Producer
import json

def create_producer():
    return Producer({
        "bootstrap.servers": "localhost:9092",
        "acks": "all",  # Wait for all replicas
        "retries": 3,
        "linger.ms": 5,  # Batch messages for 5ms
    })

def publish_order_event(producer: Producer, order_data: dict):
    def delivery_callback(err, msg):
        if err:
            print(f"Delivery failed: {err}")
        else:
            print(f"Delivered to {msg.topic()} [{msg.partition()}] @ {msg.offset()}")

    producer.produce(
        topic="orders",
        key=str(order_data["user_id"]).encode(),  # Partition by user
        value=json.dumps(order_data).encode(),
        callback=delivery_callback,
    )

    producer.flush()  # Wait for delivery

# Usage
producer = create_producer()
publish_order_event(producer, {
    "order_id": "ord_123",
    "user_id": 42,
    "items": [{"sku": "WIDGET-A", "qty": 2}],
    "total": 49.98,
})

Consumer example

from confluent_kafka import Consumer, KafkaError
import json

def start_order_consumer():
    consumer = Consumer({
        "bootstrap.servers": "localhost:9092",
        "group.id": "email-notifications",
        "auto.offset.reset": "earliest",
        "enable.auto.commit": False,  # Manual commit for at-least-once
    })

    consumer.subscribe(["orders"])

    try:
        while True:
            msg = consumer.poll(timeout=1.0)

            if msg is None:
                continue

            if msg.error():
                if msg.error().code() == KafkaError._PARTITION_EOF:
                    continue
                raise Exception(msg.error())

            order = json.loads(msg.value().decode())

            try:
                process_order_notification(order)
                # Commit offset after successful processing
                consumer.commit(asynchronous=False)
            except Exception as e:
                print(f"Processing failed: {e}")
                # Don't commit - message will be reprocessed

    finally:
        consumer.close()

When to use Kafka

Kafka is designed for event streaming and high throughput. It shines when you need to process millions of events per second, when multiple independent consumers need to read the same data (each consumer group maintains its own offset), or when you need event replay (consumers can rewind to any point in history). It is also the foundation for stream processing with tools like Kafka Streams or Flink.

Amazon SQS: the managed queue

SQS is a fully managed queue service from AWS. You do not run any infrastructure. There are two flavors: Standard (at-least-once delivery, best-effort ordering) and FIFO (exactly-once processing, strict ordering).

Producer example

import boto3
import json

sqs = boto3.client("sqs", region_name="us-east-1")
QUEUE_URL = "https://sqs.us-east-1.amazonaws.com/123456789/order-events"

def publish_order_event(order_data: dict):
    response = sqs.send_message(
        QueueUrl=QUEUE_URL,
        MessageBody=json.dumps(order_data),
        MessageAttributes={
            "EventType": {
                "DataType": "String",
                "StringValue": "order.placed",
            }
        },
        # For FIFO queues only:
        # MessageGroupId=str(order_data["user_id"]),
        # MessageDeduplicationId=order_data["order_id"],
    )

    return response["MessageId"]

Consumer example

import boto3
import json

sqs = boto3.client("sqs", region_name="us-east-1")
QUEUE_URL = "https://sqs.us-east-1.amazonaws.com/123456789/order-events"

def poll_messages():
    while True:
        response = sqs.receive_message(
            QueueUrl=QUEUE_URL,
            MaxNumberOfMessages=10,
            WaitTimeSeconds=20,  # Long polling
            VisibilityTimeout=30,  # Hide message for 30s while processing
            MessageAttributeNames=["All"],
        )

        messages = response.get("Messages", [])

        for message in messages:
            order = json.loads(message["Body"])

            try:
                process_order(order)

                # Delete after successful processing
                sqs.delete_message(
                    QueueUrl=QUEUE_URL,
                    ReceiptHandle=message["ReceiptHandle"],
                )
            except Exception as e:
                print(f"Failed to process {message['MessageId']}: {e}")
                # Message becomes visible again after VisibilityTimeout

Dead letter queues

SQS has built-in dead letter queue (DLQ) support for messages that fail processing repeatedly.

# Configure DLQ via boto3
sqs.set_queue_attributes(
    QueueUrl=QUEUE_URL,
    Attributes={
        "RedrivePolicy": json.dumps({
            "deadLetterTargetArn": "arn:aws:sqs:us-east-1:123456789:order-events-dlq",
            "maxReceiveCount": "3",  # Move to DLQ after 3 failures
        })
    },
)

When to use SQS

SQS is the best choice when you are already on AWS and want zero operational overhead. You do not need to manage broker instances, handle replication, or worry about disk space. The FIFO variant gives you exactly-once semantics. The main limitation is that SQS does not support fan-out natively (you pair it with SNS for that) and it has lower throughput than Kafka.

Head-to-head comparison

Feature	RabbitMQ	Kafka	SQS
Model	Message broker	Event log	Managed queue
Ordering	Per-queue FIFO	Per-partition	Best-effort (Standard) or FIFO
Throughput	~50K msg/s	1M+ msg/s	~3K msg/s (Standard), ~300 msg/s (FIFO)
Retention	Until consumed	Configurable (days/weeks)	Up to 14 days
Replay	No	Yes (rewind offset)	No
Delivery	At-most-once or at-least-once	At-least-once	At-least-once (Standard), exactly-once (FIFO)
Routing	Complex (exchanges, bindings)	Topic-based	Simple (queue per consumer)
Operations	Self-managed or CloudAMQP	Self-managed or Confluent Cloud	Fully managed
Fan-out	Built-in (fanout exchange)	Built-in (consumer groups)	Requires SNS

Choosing the right queue

Choose RabbitMQ when you need complex routing logic, your throughput is moderate (under 50K messages per second), and you want a battle-tested broker with good tooling and a management UI.

Choose Kafka when you need high throughput, event replay, multiple independent consumers reading the same data, or you are building event-driven architectures where the event log is the source of truth.

Choose SQS when you are on AWS, want zero operational overhead, and your use case is simple task distribution or background job processing.

Common patterns across all queues

Regardless of which queue you choose, these patterns apply.

Idempotent consumers

Messages may be delivered more than once. Your consumers must handle duplicates.

def process_order(order: dict):
    # Check if already processed
    if redis_client.sismember("processed_orders", order["order_id"]):
        return  # Already handled

    # Process the order
    create_shipment(order)
    send_confirmation(order)

    # Mark as processed
    redis_client.sadd("processed_orders", order["order_id"])

Poison message handling

Some messages will never process successfully. Do not let them block your queue forever.

MAX_RETRIES = 3

def process_with_retry(message: dict):
    retry_count = message.get("_retry_count", 0)

    try:
        handle_message(message)
    except Exception as e:
        if retry_count >= MAX_RETRIES:
            # Send to dead letter queue
            publish_to_dlq(message, error=str(e))
        else:
            message["_retry_count"] = retry_count + 1
            republish_with_delay(message, delay_seconds=2 ** retry_count)

Wrapping Up

Message queues solve real problems around service decoupling, load leveling, and reliability. RabbitMQ gives you flexible routing with a traditional broker model. Kafka provides high-throughput event streaming with replay capability. SQS offers zero-ops simplicity within the AWS ecosystem. The choice depends on your throughput needs, whether you need event replay, how complex your routing is, and how much infrastructure you want to manage. Start with the simplest option that meets your requirements and migrate if your needs genuinely outgrow it.