Designing a Real-Time Chat System

A real-time chat system seems simple on the surface: send a message, the other person sees it. Underneath, it involves persistent connections, message ordering, delivery guarantees, presence tracking, and fan-out to potentially thousands of group members. This walkthrough covers how to design one that scales.

Requirements

Start with what the system needs to do.

Functional requirements:

• One-on-one messaging
• Group messaging (up to 500 members)
• Online/offline status (presence)
• Message history and search
• Read receipts
• Media attachments (images, files)

Non-functional requirements:

• Low latency (messages delivered within 200ms for online users)
• High availability (99.99% uptime)
• Message ordering guaranteed per conversation
• At-least-once delivery
• Support 50 million daily active users

High-level architecture

┌──────────┐     ┌───────────────┐     ┌──────────────┐
│  Client   │────▶│  API Gateway  │────▶│  Auth Service │
│ (Mobile/  │     │  / Load       │     └──────────────┘
│  Web)     │     │  Balancer     │
└──────────┘     └───────┬───────┘
      │                  │
      │ WebSocket        │ HTTP
      ▼                  ▼
┌──────────────┐  ┌──────────────┐
│  WebSocket   │  │  Chat        │
│  Gateway     │  │  Service     │
│  Cluster     │  │              │
└──────┬───────┘  └──────┬───────┘
       │                 │
       ▼                 ▼
┌──────────────┐  ┌──────────────┐
│  Message     │  │  Message     │
│  Queue       │  │  Database    │
│  (Kafka)     │  │              │
└──────────────┘  └──────────────┘

The key architectural decision is separating the WebSocket connections from the business logic. WebSocket gateways handle the persistent connections. Chat services handle the message processing logic.

WebSocket connection management

Each client maintains a single WebSocket connection to a gateway server. The gateway tracks which users are connected to which server.

# Simplified WebSocket gateway logic
import asyncio
import websockets
import json

# user_id -> websocket connection
connections: dict[str, websockets.WebSocketServerProtocol] = {}

async def handler(websocket, path):
    user_id = await authenticate(websocket)
    connections[user_id] = websocket
    
    # Register this connection in Redis
    await redis.hset("user_connections", user_id, server_id)
    await redis.publish("presence", json.dumps({
        "user_id": user_id,
        "status": "online"
    }))
    
    try:
        async for raw_message in websocket:
            message = json.loads(raw_message)
            await process_message(user_id, message)
    finally:
        del connections[user_id]
        await redis.hdel("user_connections", user_id)
        await redis.publish("presence", json.dumps({
            "user_id": user_id,
            "status": "offline"
        }))

When user A sends a message to user B, the flow is:

1. User A’s client sends the message over WebSocket to Gateway Server 1
2. Gateway Server 1 publishes the message to a message queue (Kafka)
3. The Chat Service consumes the message, persists it to the database
4. The Chat Service looks up where user B is connected (Redis)
5. If user B is on Gateway Server 2, the message is routed there
6. Gateway Server 2 pushes the message to user B’s WebSocket

Connection registry with Redis

class ConnectionRegistry:
    def __init__(self, redis_client, server_id):
        self.redis = redis_client
        self.server_id = server_id
    
    async def register(self, user_id: str):
        await self.redis.hset(
            "ws:connections",
            user_id,
            json.dumps({
                "server_id": self.server_id,
                "connected_at": time.time(),
            })
        )
    
    async def unregister(self, user_id: str):
        await self.redis.hdel("ws:connections", user_id)
    
    async def find_server(self, user_id: str) -> str | None:
        data = await self.redis.hget("ws:connections", user_id)
        if data:
            return json.loads(data)["server_id"]
        return None

Message storage

Messages need a storage solution optimized for two access patterns:

1. Fetch recent messages for a conversation (pagination)
2. Store messages durably with ordering guarantees

Schema design

-- Conversations table
CREATE TABLE conversations (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    type VARCHAR(10) NOT NULL,  -- 'direct' or 'group'
    created_at TIMESTAMP DEFAULT NOW(),
    updated_at TIMESTAMP DEFAULT NOW()
);

-- Conversation members
CREATE TABLE conversation_members (
    conversation_id UUID REFERENCES conversations(id),
    user_id UUID NOT NULL,
    joined_at TIMESTAMP DEFAULT NOW(),
    last_read_message_id UUID,
    PRIMARY KEY (conversation_id, user_id)
);

-- Messages table
CREATE TABLE messages (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    conversation_id UUID NOT NULL REFERENCES conversations(id),
    sender_id UUID NOT NULL,
    content TEXT,
    message_type VARCHAR(20) DEFAULT 'text',
    created_at TIMESTAMP DEFAULT NOW(),
    sequence_number BIGSERIAL
);

-- Index for fetching recent messages
CREATE INDEX idx_messages_conversation_seq
ON messages (conversation_id, sequence_number DESC);

Message retrieval

async def get_messages(
    conversation_id: str,
    before_sequence: int | None = None,
    limit: int = 50
) -> list[dict]:
    query = """
        SELECT id, sender_id, content, message_type, 
               created_at, sequence_number
        FROM messages
        WHERE conversation_id = $1
    """
    params = [conversation_id]
    
    if before_sequence:
        query += " AND sequence_number < $2"
        params.append(before_sequence)
    
    query += " ORDER BY sequence_number DESC LIMIT $" + str(len(params) + 1)
    params.append(limit)
    
    rows = await db.fetch(query, *params)
    return [dict(row) for row in rows]

Presence system

Presence tells users who is online. It needs to handle users going online/offline, users on multiple devices, and network disconnections without a clean disconnect.

class PresenceService:
    def __init__(self, redis_client):
        self.redis = redis_client
        self.heartbeat_interval = 30  # seconds
        self.timeout = 90  # seconds
    
    async def set_online(self, user_id: str):
        await self.redis.zadd(
            "presence:online",
            {user_id: time.time()}
        )
    
    async def heartbeat(self, user_id: str):
        await self.redis.zadd(
            "presence:online",
            {user_id: time.time()}
        )
    
    async def set_offline(self, user_id: str):
        await self.redis.zrem("presence:online", user_id)
    
    async def is_online(self, user_id: str) -> bool:
        score = await self.redis.zscore("presence:online", user_id)
        if score is None:
            return False
        return (time.time() - score) < self.timeout
    
    async def get_online_users(self, user_ids: list[str]) -> list[str]:
        cutoff = time.time() - self.timeout
        online = []
        for uid in user_ids:
            score = await self.redis.zscore("presence:online", uid)
            if score and score > cutoff:
                online.append(uid)
        return online
    
    async def cleanup_stale(self):
        """Remove users who missed heartbeats."""
        cutoff = time.time() - self.timeout
        removed = await self.redis.zremrangebyscore(
            "presence:online", 0, cutoff
        )
        return removed

Group chat and fan-out

When a message is sent to a group with 200 members, you need to deliver it to all of them. This is the fan-out problem.

Fan-out on write: When a message arrives, immediately push it to every member’s message queue. Fast reads, expensive writes.

Fan-out on read: Store the message once. When a user opens the chat, pull messages for all their conversations. Cheap writes, slower reads.

Most chat systems use fan-out on write for small groups and fan-out on read for very large groups (channels with thousands of members).

async def send_group_message(
    conversation_id: str,
    sender_id: str,
    content: str
):
    # 1. Persist the message
    message = await save_message(conversation_id, sender_id, content)
    
    # 2. Get group members
    members = await get_conversation_members(conversation_id)
    
    # 3. Fan-out to online members
    for member_id in members:
        if member_id == sender_id:
            continue
        
        server_id = await connection_registry.find_server(member_id)
        if server_id:
            # User is online, push via their WebSocket gateway
            await message_queue.publish(
                topic=f"gateway.{server_id}",
                message={
                    "type": "new_message",
                    "recipient_id": member_id,
                    "message": message,
                }
            )
        else:
            # User is offline, queue for push notification
            await push_notification_queue.enqueue({
                "user_id": member_id,
                "title": f"New message in {group_name}",
                "body": content[:100],
            })

Read receipts

Read receipts track which messages a user has seen in a conversation.

async def mark_read(
    user_id: str,
    conversation_id: str,
    last_read_message_id: str
):
    # Update the member's last read position
    await db.execute("""
        UPDATE conversation_members
        SET last_read_message_id = $1
        WHERE conversation_id = $2 AND user_id = $3
    """, last_read_message_id, conversation_id, user_id)
    
    # Notify other members
    members = await get_conversation_members(conversation_id)
    for member_id in members:
        if member_id != user_id:
            await push_to_user(member_id, {
                "type": "read_receipt",
                "conversation_id": conversation_id,
                "reader_id": user_id,
                "last_read_message_id": last_read_message_id,
            })

async def get_unread_count(user_id: str, conversation_id: str) -> int:
    row = await db.fetchone("""
        SELECT COUNT(*) as unread
        FROM messages m
        JOIN conversation_members cm
          ON cm.conversation_id = m.conversation_id
         AND cm.user_id = $1
        WHERE m.conversation_id = $2
          AND m.sequence_number > (
            SELECT sequence_number FROM messages
            WHERE id = cm.last_read_message_id
          )
    """, user_id, conversation_id)
    return row["unread"]

Scaling considerations

WebSocket gateway scaling: Each server handles 50K-100K concurrent connections. Add servers behind a load balancer. Use sticky sessions (by user ID hash) so reconnections hit the same server when possible.

Message database scaling: Partition messages by conversation_id. Recent conversations (hot data) can live in a faster store; older messages can be archived to cheaper storage.

Message ordering: Use a sequence number per conversation, generated by a single writer (the database, via a serial column) to avoid ordering conflicts.

Offline message sync: When a user comes online, they request messages since their last known sequence number for each conversation. This cursor-based sync avoids duplicates.

Media handling: Do not send files through the WebSocket. Upload media via HTTP to an object store (S3), get a URL, and send the URL as the message content.

The core of a chat system is managing persistent connections and routing messages efficiently. Start with the basics, get the message flow right, then add features like read receipts and presence on top. Every major chat service (WhatsApp, Slack, Discord) follows these same fundamental patterns.