Reliable Webhook Delivery with Retries and Idempotency

Webhooks are HTTP callbacks triggered by events. They sound simple: when something happens, send an HTTP POST to a URL. But in production, webhook delivery is full of failure modes. The receiver is down, the network times out, the receiver processes the event twice, or the receiver cannot verify the sender. This guide covers the patterns that make webhook delivery reliable.

The webhook delivery pipeline

A reliable webhook system has these components:

Event Occurs
    |
    v
Write to outbox table (same transaction as business logic)
    |
    v
Worker reads from outbox
    |
    v
Sign payload + send HTTP POST
    |
    v
Success? Mark as delivered
Failure? Schedule retry with backoff
    |
    v
Max retries exceeded? Move to dead letter queue

The outbox pattern

Never send webhooks directly from your business logic. If the HTTP call fails, you cannot retry it because the transaction context is gone. Instead, write the webhook event to an outbox table in the same database transaction.

CREATE TABLE webhook_outbox (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    event_type VARCHAR(100) NOT NULL,
    payload JSONB NOT NULL,
    target_url TEXT NOT NULL,
    idempotency_key VARCHAR(255) NOT NULL UNIQUE,
    status VARCHAR(20) DEFAULT 'pending', -- pending, delivered, failed, dead
    attempts INT DEFAULT 0,
    max_attempts INT DEFAULT 5,
    next_retry_at TIMESTAMPTZ DEFAULT NOW(),
    created_at TIMESTAMPTZ DEFAULT NOW(),
    delivered_at TIMESTAMPTZ,
    last_error TEXT
);

CREATE INDEX idx_outbox_pending ON webhook_outbox(next_retry_at)
    WHERE status = 'pending';

Writing to the outbox

def create_order(db, order_data):
    """Create an order and queue a webhook in the same transaction."""
    with db.transaction():
        order = db.execute(
            "INSERT INTO orders (customer_id, total) VALUES (%s, %s) RETURNING *",
            (order_data['customer_id'], order_data['total'])
        )

        db.execute("""
            INSERT INTO webhook_outbox (event_type, payload, target_url, idempotency_key)
            VALUES (%s, %s, %s, %s)
        """, (
            'order.created',
            json.dumps({
                'event': 'order.created',
                'data': {'order_id': order['id'], 'total': str(order['total'])},
                'timestamp': datetime.utcnow().isoformat()
            }),
            get_webhook_url(order_data['customer_id']),
            f"order.created.{order['id']}"  # idempotency key
        ))

    return order

Both the order creation and the webhook record are in the same transaction. If either fails, both roll back.

Retry with exponential backoff

A worker process polls the outbox and attempts delivery:

import hmac
import hashlib
import requests
import time
from datetime import datetime, timedelta

def process_webhook_outbox(db):
    """Poll and deliver pending webhooks."""
    while True:
        events = db.execute("""
            SELECT * FROM webhook_outbox
            WHERE status = 'pending' AND next_retry_at <= NOW()
            ORDER BY next_retry_at
            LIMIT 50
            FOR UPDATE SKIP LOCKED
        """)

        for event in events:
            deliver_webhook(db, event)

        time.sleep(1)  # poll interval


def deliver_webhook(db, event):
    """Attempt to deliver a single webhook."""
    signature = sign_payload(event['payload'], WEBHOOK_SECRET)

    try:
        response = requests.post(
            event['target_url'],
            json=json.loads(event['payload']),
            headers={
                'Content-Type': 'application/json',
                'X-Webhook-Signature': signature,
                'X-Idempotency-Key': event['idempotency_key'],
                'X-Webhook-Event': event['event_type'],
            },
            timeout=10
        )

        if 200 <= response.status_code < 300:
            db.execute("""
                UPDATE webhook_outbox
                SET status = 'delivered', delivered_at = NOW()
                WHERE id = %s
            """, (event['id'],))
        else:
            handle_failure(db, event, f"HTTP {response.status_code}: {response.text[:500]}")

    except requests.exceptions.RequestException as e:
        handle_failure(db, event, str(e))


def handle_failure(db, event, error):
    """Schedule retry or move to dead letter."""
    attempts = event['attempts'] + 1

    if attempts >= event['max_attempts']:
        db.execute("""
            UPDATE webhook_outbox
            SET status = 'dead', attempts = %s, last_error = %s
            WHERE id = %s
        """, (attempts, error, event['id']))
        alert_dead_letter(event)
    else:
        # exponential backoff: 10s, 40s, 160s, 640s, ...
        delay_seconds = 10 * (4 ** (attempts - 1))
        next_retry = datetime.utcnow() + timedelta(seconds=delay_seconds)

        db.execute("""
            UPDATE webhook_outbox
            SET attempts = %s, next_retry_at = %s, last_error = %s
            WHERE id = %s
        """, (attempts, next_retry, error, event['id']))

The retry schedule with exponential backoff:

Attempt	Delay	Cumulative
1	10 seconds	10s
2	40 seconds	50s
3	160 seconds (~3 min)	~3.5 min
4	640 seconds (~11 min)	~14 min
5	Dead letter	—

Signing webhook payloads

Receivers need to verify that a webhook came from your system, not an attacker. Sign the payload using HMAC-SHA256:

Sender side

def sign_payload(payload: str, secret: str) -> str:
    """Generate HMAC-SHA256 signature for webhook payload."""
    return hmac.new(
        secret.encode('utf-8'),
        payload.encode('utf-8'),
        hashlib.sha256
    ).hexdigest()

Receiver side

from fastapi import FastAPI, Request, HTTPException
import hmac
import hashlib

app = FastAPI()

@app.post("/webhooks/orders")
async def receive_webhook(request: Request):
    body = await request.body()
    signature = request.headers.get('X-Webhook-Signature', '')

    expected = hmac.new(
        WEBHOOK_SECRET.encode('utf-8'),
        body,
        hashlib.sha256
    ).hexdigest()

    if not hmac.compare_digest(signature, expected):
        raise HTTPException(status_code=401, detail="Invalid signature")

    # process the webhook
    payload = await request.json()
    idempotency_key = request.headers.get('X-Idempotency-Key')

    # idempotency check
    if await already_processed(idempotency_key):
        return {"status": "already_processed"}

    await process_event(payload, idempotency_key)
    return {"status": "ok"}

Use hmac.compare_digest instead of == to prevent timing attacks.

Idempotency on the receiver side

Webhooks may be delivered more than once (network issues, retries, duplicate events). Receivers must handle duplicates gracefully.

async def already_processed(idempotency_key: str) -> bool:
    """Check if this webhook was already processed."""
    result = await db.execute(
        "SELECT 1 FROM processed_webhooks WHERE idempotency_key = %s",
        (idempotency_key,)
    )
    return result is not None


async def process_event(payload: dict, idempotency_key: str):
    """Process the webhook event idempotently."""
    async with db.transaction():
        # record that we processed this event
        await db.execute(
            "INSERT INTO processed_webhooks (idempotency_key, processed_at) VALUES (%s, NOW())",
            (idempotency_key,)
        )
        # unique constraint on idempotency_key prevents duplicates

        # actual business logic
        if payload['event'] == 'order.created':
            await handle_order_created(payload['data'])

Dead letter handling

When a webhook exhausts all retries, it moves to a dead letter state. Do not silently drop it.

1. Alert: Notify the engineering team via Slack or PagerDuty.
2. Dashboard: Provide an admin UI to view and manually retry dead letters.
3. Root cause: Group dead letters by error type to identify systemic issues.
4. Manual retry endpoint: Allow operators to retry a specific webhook.

@app.post("/admin/webhooks/{webhook_id}/retry")
async def retry_webhook(webhook_id: str):
    await db.execute("""
        UPDATE webhook_outbox
        SET status = 'pending', attempts = 0, next_retry_at = NOW()
        WHERE id = %s AND status = 'dead'
    """, (webhook_id,))
    return {"status": "requeued"}

Best practices for webhook consumers

If you are building the receiving side:

1. Return 200 quickly: Do your processing asynchronously. Return 200 to acknowledge receipt, then process in a background job.
2. Handle duplicates: Always check the idempotency key before processing.
3. Verify signatures: Never trust unverified webhooks.
4. Log everything: Log the raw payload, headers, and your processing result for debugging.
5. Set up alerting: Monitor for sudden drops in webhook volume, which may indicate the sender stopped sending or your endpoint is failing.

Summary

Reliable webhook delivery requires the outbox pattern for transactional safety, exponential backoff for retries, HMAC signatures for authentication, idempotency keys for deduplication, and dead letter queues for undeliverable events. Both the sender and receiver share responsibility: the sender ensures delivery, and the receiver ensures idempotent processing.