Idempotency Patterns: Keys, Deduplication, and Exactly-Once Semantics

A user clicks “Pay Now,” the request times out, and they click again. Did they get charged once or twice? In a distributed system, network failures, retries, and at-least-once delivery guarantees mean the same operation can be submitted multiple times. Idempotency ensures that processing the same operation multiple times produces the same result as processing it once. Without it, duplicate charges, duplicate orders, and duplicate messages are inevitable.

What Makes an Operation Idempotent?

An operation is idempotent if applying it N times has the same effect as applying it once.

Idempotent:
  SET balance = 500          (applying twice: balance is still 500)
  DELETE user WHERE id = 42  (deleting twice: user is still gone)
  PUT /users/42 { name: "Alice" }  (applying twice: same result)

Not idempotent:
  INCREMENT balance BY 100   (applying twice: balance increases by 200)
  POST /orders { item: "book" }   (applying twice: two orders created)
  INSERT INTO logs (message) VALUES ('event')  (two rows created)

HTTP methods have expected idempotency semantics: GET, PUT, and DELETE should be idempotent by design. POST is not idempotent by default, which is why most idempotency patterns focus on POST endpoints.

The Idempotency Key Pattern

The client generates a unique key (typically a UUID) for each logical operation and includes it in the request. The server uses this key to detect and deduplicate retries.

First attempt:
  POST /payments
  Idempotency-Key: 550e8400-e29b-41d4-a716-446655440000
  Body: { amount: 99.99, currency: "USD" }

  Server: process payment, store result keyed by idempotency key
  Response: 201 Created { payment_id: "pay_123", status: "completed" }

Retry (network timeout on first attempt):
  POST /payments
  Idempotency-Key: 550e8400-e29b-41d4-a716-446655440000
  Body: { amount: 99.99, currency: "USD" }

  Server: key exists, return stored result
  Response: 201 Created { payment_id: "pay_123", status: "completed" }

The payment is processed exactly once. The retry receives the same response as the original request.

Server-side implementation

from fastapi import FastAPI, Header, HTTPException
from uuid import UUID
import json

app = FastAPI()

class IdempotencyStore:
    def __init__(self, redis_client):
        self.redis = redis_client
        self.ttl = 86400  # 24 hours

    def get(self, key: str):
        result = self.redis.get(f"idempotency:{key}")
        if result:
            return json.loads(result)
        return None

    def lock(self, key: str) -> bool:
        """Acquire a lock to prevent concurrent processing of same key."""
        return self.redis.set(
            f"idempotency:{key}:lock", "1",
            nx=True, ex=60  # 60-second lock timeout
        )

    def store(self, key: str, status_code: int, body: dict):
        self.redis.set(
            f"idempotency:{key}",
            json.dumps({"status_code": status_code, "body": body}),
            ex=self.ttl
        )
        self.redis.delete(f"idempotency:{key}:lock")

    def unlock(self, key: str):
        self.redis.delete(f"idempotency:{key}:lock")


store = IdempotencyStore(redis_client)

@app.post("/payments")
async def create_payment(
    request: PaymentRequest,
    idempotency_key: str = Header(alias="Idempotency-Key")
):
    # Check for existing result
    existing = store.get(idempotency_key)
    if existing:
        return JSONResponse(
            status_code=existing["status_code"],
            content=existing["body"]
        )

    # Acquire lock for this key
    if not store.lock(idempotency_key):
        raise HTTPException(409, "Request with this key is already in progress")

    try:
        # Process the payment
        result = process_payment(request)
        response_body = {"payment_id": result.id, "status": result.status}

        # Store result for future retries
        store.store(idempotency_key, 201, response_body)

        return JSONResponse(status_code=201, content=response_body)
    except Exception as e:
        store.unlock(idempotency_key)
        raise

Key design decisions

Key generation: the client must generate the key, not the server. The key must be the same across retries of the same logical operation. UUIDs work well. Some APIs use a hash of the request body, but this fails if different logical operations happen to have identical bodies.

TTL: idempotency keys should expire. Twenty-four hours is common. A key that lives forever wastes storage and prevents the user from ever intentionally repeating the same operation.

Request matching: when a key is reused, should the server verify that the request body matches the original? Stripe does this and returns a 422 error if the body differs, preventing misuse.

Database-Level Idempotency

For operations backed by a database, you can enforce idempotency at the data layer using unique constraints:

-- Idempotency via unique constraint on business key
CREATE TABLE payments (
    id              UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    idempotency_key UUID UNIQUE NOT NULL,
    amount          DECIMAL(10,2) NOT NULL,
    status          VARCHAR(20) NOT NULL,
    created_at      TIMESTAMPTZ DEFAULT NOW()
);

-- Insert with ON CONFLICT: second attempt is a no-op
INSERT INTO payments (idempotency_key, amount, status)
VALUES ('550e8400-e29b-41d4-a716-446655440000', 99.99, 'completed')
ON CONFLICT (idempotency_key) DO NOTHING
RETURNING *;

This is simpler than the Redis approach and transactionally safe. The trade-off is that you cannot return the original response body without a follow-up query.

Message Queue Deduplication

In event-driven systems, idempotency extends to message processing. A consumer might receive the same message multiple times due to at-least-once delivery guarantees.

Deduplication store pattern

class IdempotentMessageHandler:
    def __init__(self, db, handler):
        self.db = db
        self.handler = handler

    def handle(self, message):
        message_id = message["id"]

        # Check if already processed
        if self.db.exists("processed_messages", message_id):
            return  # skip duplicate

        # Process the message
        with self.db.transaction():
            self.handler(message)
            # Record processing in the same transaction
            self.db.insert("processed_messages", {
                "message_id": message_id,
                "processed_at": now()
            })

The critical detail: recording the message as processed must happen in the same transaction as the business logic. Otherwise, a crash between processing and recording causes the message to be reprocessed.

Kafka exactly-once

Kafka provides exactly-once semantics through a combination of idempotent producers and transactional consumers:

Producer side:
  enable.idempotence=true
  Kafka assigns a producer ID and sequence number
  Broker deduplicates based on (producer_id, sequence_number)

Consumer side:
  read_committed isolation
  Consumer commits offsets in the same Kafka transaction as output

This achieves exactly-once within the Kafka ecosystem but not end-to-end. If the consumer writes to an external database, idempotency must be enforced there as well.

Making Non-Idempotent Operations Idempotent

Increment operations

Instead of INCREMENT balance BY 100, track which increments have been applied:

-- Instead of: UPDATE accounts SET balance = balance + 100
-- Use: apply a named adjustment idempotently

INSERT INTO balance_adjustments (account_id, adjustment_id, amount)
VALUES (42, 'adj-550e8400', 100)
ON CONFLICT (adjustment_id) DO NOTHING;

-- Balance is computed from adjustments
SELECT SUM(amount) FROM balance_adjustments WHERE account_id = 42;

Side effects (emails, notifications)

If processing a message triggers a side effect like sending an email, the deduplication check must happen before the side effect. A common pattern: write an “outbox” record in the same transaction as the business logic, and a separate process reads the outbox and sends emails idempotently.

Transaction:
  1. Process order
  2. Insert into outbox: { type: "send_email", order_id: 123 }

Outbox processor (separate process):
  1. Read unprocessed outbox entries
  2. Send email
  3. Mark entry as processed
  4. If email fails, retry later (outbox entry remains)

Idempotency Window and Replay Attacks

An idempotency key that lives forever creates a different problem: replay attacks. An attacker who captures a valid request can replay it indefinitely. Mitigations:

• TTL: expire keys after 24 to 48 hours.
• Timestamp validation: reject requests whose timestamp is too old.
• Request fingerprinting: bind the idempotency key to the authenticated user so a captured key cannot be used by a different account.

Key Takeaways

Idempotency is not optional in distributed systems. Network retries, at-least-once delivery, and user double-clicks guarantee that your system will receive duplicate requests. Use idempotency keys for API endpoints, deduplication stores for message consumers, and unique constraints at the database level. Record processing and business logic in the same transaction to avoid partial-processing gaps. Design your system so that duplicates are harmless, and you eliminate an entire class of production bugs.