System Design: Design a Webhook Delivery System

Webhooks are the standard mechanism for notifying external systems about events. When a payment succeeds, a deployment completes, or a form is submitted, your system sends an HTTP POST to a URL the customer configured. It sounds simple, but reliable webhook delivery at scale involves retry logic, ordering guarantees, security, and graceful degradation when receivers are slow or down.

Services like Stripe, GitHub, and Twilio all operate webhook delivery systems that send billions of events. Designing one reveals important patterns for reliable event-driven communication.

Functional Requirements

• Allow customers to register webhook endpoints (URLs) for specific event types.
• Deliver event payloads to registered endpoints via HTTP POST.
• Retry failed deliveries with configurable backoff.
• Sign payloads so receivers can verify authenticity.
• Provide a delivery log showing attempt history, status codes, and response times.
• Support disabling endpoints that are consistently failing.

Non-Functional Requirements

• Delivery guarantee: at-least-once. Every event must be delivered at least once to every registered endpoint.
• Latency: deliver events within 5 seconds of occurrence under normal conditions.
• Throughput: handle millions of events per hour.
• Reliability: survive individual component failures without losing events.

High-Level Architecture

The system has five components:

1. Event ingestion — an internal API or message queue where application services publish events.
2. Event store — a durable log of all events.
3. Dispatcher — matches events to registered endpoints and creates delivery tasks.
4. Delivery workers — execute HTTP requests to customer endpoints.
5. Management API — lets customers register endpoints, view delivery logs, and configure retry policies.

[App Service] -> [Event Queue] -> [Dispatcher] -> [Delivery Queue] -> [Workers] -> [Customer Endpoint]
                                                                           |
                                                                   [Delivery Log DB]

Event Ingestion

Application services publish events to a message queue (Kafka or SQS). Each event includes:

{
  "event_id": "evt_abc123",
  "type": "payment.completed",
  "created_at": "2026-07-02T10:00:00Z",
  "data": {
    "payment_id": "pay_xyz",
    "amount": 9999,
    "currency": "usd"
  }
}

The event_id is globally unique and serves as the idempotency key throughout the pipeline. If the same event is published twice (due to a producer retry), the system deduplicates it.

Store every event in a durable event store (append-only table or Kafka topic with long retention). This serves as the source of truth and enables replaying events if needed.

Endpoint Registration

Customers register endpoints through the management API:

{
  "endpoint_id": "ep_001",
  "url": "https://customer.com/webhooks",
  "events": ["payment.completed", "payment.failed"],
  "signing_secret": "whsec_...",
  "active": true
}

Store endpoints in a database indexed by event type for fast lookup. When an event arrives, the dispatcher queries all active endpoints subscribed to that event type.

URL validation. Before activating an endpoint, send a test event and verify the endpoint responds with a 2xx status. Reject endpoints pointing to private IP ranges (10.x, 192.168.x) to prevent SSRF attacks.

Dispatcher

The dispatcher consumes events from the ingestion queue and creates delivery tasks:

1. Read an event from the queue.
2. Look up all active endpoints subscribed to the event type.
3. For each endpoint, create a delivery task and enqueue it on the delivery queue.
4. Acknowledge the event on the ingestion queue.

Each delivery task contains the event payload, the endpoint URL, the signing secret, and metadata (attempt number, next retry time).

If no endpoints are registered for an event type, the dispatcher acknowledges the event without creating any tasks. The event is still retained in the event store for auditability.

Delivery Workers

Workers consume delivery tasks from the delivery queue and execute HTTP POST requests.

Request construction:

POST /webhooks HTTP/1.1
Host: customer.com
Content-Type: application/json
X-Webhook-ID: evt_abc123
X-Webhook-Timestamp: 1751450400
X-Webhook-Signature: v1=sha256_hmac_of_payload
User-Agent: CodeloomWebhooks/1.0

{"event_id": "evt_abc123", "type": "payment.completed", ...}

Success criteria. A delivery is considered successful if the endpoint responds with a 2xx status code within 30 seconds. Any other status code (3xx, 4xx, 5xx) or a timeout is treated as a failure.

Recording results. After each attempt, write a delivery log entry:

delivery_attempts
  id              UUID
  event_id        VARCHAR
  endpoint_id     VARCHAR
  attempt_number  INT
  status_code     INT (NULL if timeout)
  response_body   TEXT (first 1KB, for debugging)
  response_time_ms INT
  created_at      TIMESTAMP

Retry Strategy

Failed deliveries must be retried. The retry strategy balances speed (deliver quickly) with courtesy (do not overwhelm a struggling endpoint).

Exponential backoff with jitter. After each failure, wait an increasing amount of time before retrying:

• Attempt 1: immediate
• Attempt 2: 1 minute
• Attempt 3: 5 minutes
• Attempt 4: 30 minutes
• Attempt 5: 2 hours
• Attempt 6: 8 hours
• Attempt 7: 24 hours

Add random jitter (up to 20% of the delay) to prevent thundering herd effects when many endpoints fail simultaneously and all retries align.

Maximum attempts. After a configurable number of attempts (typically 5-7 over 24-72 hours), stop retrying and move the event to a dead letter queue.

Implementation. Use a delay queue (SQS with delay, or a scheduled task). When a delivery fails, re-enqueue the task with a visibility delay equal to the next backoff interval.

Dead Letter Queue

Events that exhaust all retry attempts land in a dead letter queue (DLQ). The system:

1. Marks the delivery as failed in the delivery log.
2. Sends a notification to the customer (email or dashboard alert) informing them of persistent delivery failures.
3. If an endpoint accumulates too many consecutive failures (for example, 50 in a row), automatically disable the endpoint to stop wasting resources.

Customers can manually replay events from the DLQ through the management dashboard once they fix their endpoint.

Payload Signing

Receivers need to verify that webhook requests actually came from your system and were not tampered with. Use HMAC-SHA256 signing:

1. Concatenate the event ID, timestamp, and raw request body: message = event_id.timestamp.body.
2. Compute HMAC-SHA256(message, signing_secret).
3. Include the signature in the X-Webhook-Signature header.

The receiver computes the same HMAC using their copy of the signing secret and compares it to the header value. If they match, the payload is authentic.

Replay protection. Include a timestamp in the signed message and the X-Webhook-Timestamp header. Receivers should reject requests with timestamps older than 5 minutes. This prevents an attacker who captures a webhook request from replaying it later.

Ordering Guarantees

Webhooks are delivered at-least-once, but strict ordering across events is expensive and often unnecessary. Within a single event type and entity (for example, all events for payment pay_xyz), relative ordering matters. Across entities, it does not.

Best-effort ordering. Process events in the order they arrive in the ingestion queue. Since the queue preserves order per partition, partition by entity ID (payment ID, user ID) to maintain per-entity ordering.

Consumer-side ordering. Include a monotonically increasing sequence number or the event timestamp in the payload. Let the receiver use this to detect out-of-order delivery and reorder if needed.

Do not guarantee strict global ordering. It requires serialization that destroys throughput.

Monitoring and Observability

• Delivery success rate: percentage of events delivered on the first attempt. Target above 95%.
• End-to-end latency: time from event creation to successful delivery.
• Retry rate: percentage of events requiring retries. A rising retry rate signals widespread endpoint issues.
• DLQ depth: number of events in the dead letter queue. Should be actively monitored and kept near zero.
• Endpoint health: track success rate per endpoint. Surface unhealthy endpoints in the customer dashboard.

Scaling Considerations

• Delivery workers: scale horizontally. Each worker pulls tasks independently from the delivery queue. Add workers when queue depth grows.
• Slow endpoints: a slow customer endpoint (taking 25 seconds to respond) ties up a worker thread. Use asynchronous HTTP clients with connection timeouts to prevent slow endpoints from blocking the worker pool.
• Per-endpoint rate limiting: limit the concurrency of deliveries to a single endpoint (for example, 10 concurrent requests). This prevents your system from overwhelming a customer’s server during a burst of events.
• Multi-region: deploy workers in multiple regions. Route delivery tasks to workers geographically close to the customer endpoint for lower latency.

Summary

A webhook delivery system is an event-driven pipeline with at-least-once delivery guarantees. The critical design decisions are the retry strategy (exponential backoff with jitter and a dead letter queue), the security model (HMAC signing with replay protection), and how you handle slow or failing endpoints (automatic disabling, per-endpoint rate limiting, and async HTTP clients). Start with a simple queue-based architecture and add sophistication as the number of endpoints and events grows.