Database Replication: Leaders, Followers, and Consistency

Imagine a library with only one copy of every book. If that library burns down, every book is gone. If too many people want the same book, they wait in a long line. Database replication solves both problems by making copies of your data and distributing them across multiple machines.

Why Replication Exists

Replication serves three distinct purposes, and understanding which one you care about shapes every design decision that follows.

High availability. If one database server dies, another copy can take over immediately. Without replication, a hardware failure means downtime until you restore from a backup — which could take hours.

Read scalability. A single database can handle a limited number of concurrent queries. By routing read traffic to multiple replicas, you multiply your read capacity without touching the primary server.

Disaster recovery. By placing replicas in geographically separate data centers, you protect against regional failures like power outages, natural disasters, or network partitions that take out an entire facility.

Most production systems want all three. The question is which replication architecture gives you the right balance of consistency, performance, and operational simplicity.

Single-Leader Replication

Single-leader replication is the most common pattern and the default in PostgreSQL, MySQL, and MongoDB. One node is designated the leader (also called primary or master). All writes go to the leader, and the leader streams changes to one or more followers (also called replicas or secondaries).

┌──────────────┐
│    Client     │
│  (writes)     │
└──────┬───────┘
       │
       ▼
┌──────────────┐     replication     ┌──────────────┐
│    Leader     │───────────────────▶│  Follower 1   │
│  (read/write) │───────────────────▶│  (read-only)  │
└──────────────┘                     └──────────────┘
       │
       │         replication         ┌──────────────┐
       └────────────────────────────▶│  Follower 2   │
                                     │  (read-only)  │
                                     └──────────────┘

The write path is simple: the client sends a write to the leader, the leader applies it to its local storage, then sends the change to all followers via a replication log (sometimes called a write-ahead log or binlog).

The read path offers a choice: read from the leader for guaranteed freshness, or read from a follower for better performance and load distribution, accepting that the data might be slightly stale.

Replication Lag: The Invisible Problem

Replication lag is the time between when a write is applied on the leader and when it becomes visible on a follower. In a healthy system, this might be a few milliseconds. Under load, it can stretch to seconds or even minutes.

This creates subtle user-experience bugs. Consider this scenario:

1. A user updates their profile picture.
2. The write goes to the leader and succeeds.
3. The user refreshes the page, and the read hits a follower.
4. The follower has not received the update yet.
5. The user sees their old profile picture and thinks the update failed.

This is called a read-your-writes consistency violation. Solutions include:

# Strategy 1: Read from leader for recently-written data
def get_user_profile(user_id, last_write_timestamp):
    if time.now() - last_write_timestamp < REPLICATION_LAG_THRESHOLD:
        return leader_db.query("SELECT * FROM users WHERE id = %s", user_id)
    else:
        return replica_db.query("SELECT * FROM users WHERE id = %s", user_id)

# Strategy 2: Track replication position
def get_user_profile(user_id, min_lsn):
    replica = pick_replica_at_least_at(min_lsn)
    return replica.query("SELECT * FROM users WHERE id = %s", user_id)

Strategy 1 routes reads to the leader for a short window after writes. Simple but increases leader load.

Strategy 2 tracks the log sequence number (LSN) of the last write and only reads from replicas that have caught up to that position. More sophisticated but requires infrastructure support.

Failover: When the Leader Dies

If the leader becomes unavailable, a follower must be promoted. This process is called failover, and it is surprisingly tricky to get right.

1. Detection. How do you know the leader is dead versus just slow? Most systems use heartbeat timeouts — if the leader does not respond within a threshold, it is declared dead.

2. Election. Which follower becomes the new leader? Typically the one with the most up-to-date replication position.

3. Reconfiguration. All clients and remaining followers must start pointing to the new leader. DNS changes, connection pool updates, and routing rules all need to switch.

The danger is split-brain: the old leader comes back online and both nodes accept writes, causing data divergence. Production systems use fencing mechanisms — revoking the old leader’s ability to accept writes — to prevent this.

Multi-Leader Replication

In multi-leader replication, multiple nodes accept writes simultaneously. Each leader replicates its changes to all other leaders. This is common in multi-datacenter deployments where you want writes to be fast in every region.

┌──────────────┐          ┌──────────────┐
│  Leader (US)  │◀────────▶│ Leader (EU)   │
│  read/write   │          │  read/write   │
└──────┬───────┘          └──────┬───────┘
       │                         │
  ┌────┴────┐               ┌────┴────┐
  │Followers│               │Followers│
  └─────────┘               └─────────┘

The main advantage: users in each region write to a local leader with low latency instead of sending every write across the ocean to a single leader. For a global application, this can mean the difference between 20ms and 200ms write latency.

The Conflict Problem

The fundamental challenge of multi-leader replication is write conflicts. If two users edit the same record at the same time on different leaders, you have conflicting changes that must be resolved.

Consider a shared document where User A in New York changes the title to “Q3 Report” and User B in London changes the title to “Quarterly Summary” at the same time. Both writes succeed locally, but when they replicate to each other, there is a conflict.

Common resolution strategies:

Last-write-wins (LWW). Attach a timestamp to each write and keep the latest one. Simple but data loss is possible — one user’s change silently disappears. Cassandra and DynamoDB use this by default.

Merge the values. Automatically combine conflicting values. For some data types (counters, sets) this works well. For others (text fields) it produces nonsense.

Keep both and let the application decide. Store all conflicting versions and present them to the user or application logic for resolution. CouchDB takes this approach.

Custom conflict resolution. Write application-specific logic that knows how to merge domain objects intelligently. This is the most work but produces the best results.

# Example: custom conflict resolution for a shopping cart
def resolve_cart_conflict(version_a, version_b):
    # Union of items — if either version has an item, keep it
    merged_items = {}
    for item in version_a['items'] + version_b['items']:
        key = item['product_id']
        if key in merged_items:
            merged_items[key]['quantity'] = max(
                merged_items[key]['quantity'],
                item['quantity']
            )
        else:
            merged_items[key] = item
    return {'items': list(merged_items.values())}

Leaderless Replication

Leaderless replication takes a radically different approach: there is no leader at all. Any node can accept reads and writes. The client sends writes to multiple nodes simultaneously and reads from multiple nodes, using quorum rules to ensure consistency.

This is the Dynamo-style architecture used by Cassandra, Riak, and Amazon DynamoDB.

Quorum Reads and Writes

With n total replicas, you configure:

• w = number of nodes that must acknowledge a write
• r = number of nodes you read from

If w + r > n, you are guaranteed that at least one node in your read set has the latest write. This is called a quorum.

Example: n=3, w=2, r=2

Write "Alice" to key "user:1":
  Node A: ✓ (acknowledges)
  Node B: ✓ (acknowledges)  ← write succeeds (w=2 met)
  Node C: ✗ (temporarily down)

Read key "user:1":
  Node A: returns "Alice"
  Node B: returns "Alice"   ← read succeeds (r=2 met)
  Result: "Alice" (consistent)

The beauty of this model is tunable consistency. You choose the trade-off:

• Strong consistency: w=n, r=1 or w=1, r=n — but availability suffers because all nodes must be reachable for writes or reads.
• High availability: w=1, r=1 — but you might read stale data.
• Balanced: w=2, r=2 with n=3 — tolerates one node failure for both reads and writes.

Read Repair and Anti-Entropy

When a node comes back online after being down, its data is stale. Two mechanisms fix this:

Read repair. When a client reads from multiple nodes and detects a stale value, it writes the latest value back to the stale node. This is piggyback repair — it happens during normal reads.

Anti-entropy. A background process continuously compares data across nodes and copies missing data to nodes that are behind. This catches data that is never read (and thus never repaired by read repair).

Synchronous vs Asynchronous Replication

This is a fundamental trade-off that applies to all replication architectures.

Synchronous replication means the leader waits for the follower to confirm it has written the data before acknowledging the write to the client. You get a guarantee that the follower has the data, but every write is as slow as the slowest replica.

Asynchronous replication means the leader acknowledges the write to the client immediately after writing it locally. The replication happens in the background. Writes are fast, but if the leader dies before replicating, those writes are lost.

Synchronous:
Client ──write──▶ Leader ──replicate──▶ Follower
                  Leader ◀──ack────────── Follower
Client ◀──ack──── Leader
Total latency: write + network round trip + follower write

Asynchronous:
Client ──write──▶ Leader ──replicate──▶ Follower (background)
Client ◀──ack──── Leader
Total latency: write only

Most production systems use semi-synchronous replication: one follower is synchronous (guaranteeing at least one backup copy), and the rest are asynchronous (keeping latency reasonable). PostgreSQL’s synchronous_commit setting and MySQL’s semi-sync replication both support this model.

Real-World: How Slack Handles Database Replication

Slack’s architecture offers a practical case study. They use MySQL with single-leader replication and have shared their approach publicly:

• Each Slack workspace is sharded to a specific database shard (they call this “team-level sharding”).
• Each shard has a primary and multiple read replicas.
• They use ProxySQL to route reads to replicas and writes to the primary.
• For read-your-writes consistency, they track the GTID (Global Transaction ID) from the last write and ensure the replica has reached that position before reading.

The key insight from Slack’s architecture is that they keep the replication model simple (single-leader) and invest their complexity budget in sharding and routing. They avoid multi-leader replication entirely because the conflict resolution complexity is not worth it for their access patterns.

Choosing a Replication Strategy

Single-leader is the right default. It is simple, well-understood, and sufficient for most applications. Start here unless you have a specific reason not to.

Multi-leader makes sense when you have users in multiple geographic regions and write latency matters. Be prepared to invest heavily in conflict resolution.

Leaderless shines when you need extreme availability and can tolerate eventual consistency. It works well for use cases like shopping carts, session stores, and activity feeds where losing an update is annoying but not catastrophic.

Wrapping Up

Database replication is about making copies of data across multiple machines to improve availability, read performance, and durability. The choice between single-leader, multi-leader, and leaderless replication comes down to your tolerance for complexity, your consistency requirements, and your geographic distribution. Start simple with single-leader replication and read replicas, measure your replication lag, and evolve the architecture only when the simpler model genuinely cannot serve your needs.