REST API Webhooks Design: Patterns, Retries & Security

Webhooks flip the API model around. Instead of clients polling your API for changes, you push events to their servers the moment something happens. This article covers how to design a production-grade webhook system — the delivery pipeline, retry logic, security, and the management API your users need.

Webhook fundamentals

A webhook is an HTTP POST request that your server sends to a URL registered by the consumer. The body contains a structured event payload:

POST https://customer.example.com/webhooks/orders
Content-Type: application/json
X-Webhook-Id: evt_a1b2c3d4e5
X-Webhook-Timestamp: 1720396800
X-Webhook-Signature: sha256=5d5b09f6dcb2d53a5fffc60c4ac0d55fabdf556069...

{
  "id": "evt_a1b2c3d4e5",
  "type": "order.completed",
  "created_at": "2026-07-08T10:00:00Z",
  "data": {
    "order_id": "ord_xyz789",
    "total": 129.99,
    "currency": "USD",
    "customer_id": "cust_abc123"
  }
}

Event payload design

Consistent event envelope

Every webhook should use the same envelope structure:

function createEvent(type, data) {
  return {
    id: `evt_${crypto.randomUUID().replace(/-/g, '').slice(0, 12)}`,
    type,                           // e.g., "order.completed"
    api_version: '2026-07-01',      // API version when event was created
    created_at: new Date().toISOString(),
    data,                           // the actual payload
  };
}

Event type naming conventions

Use a resource.action pattern:

order.created
order.updated
order.completed
order.cancelled
payment.succeeded
payment.failed
invoice.finalized
customer.subscription.renewed

Fat vs thin payloads

Fat payload — include the full resource in data:

{
  "type": "order.completed",
  "data": {
    "order_id": "ord_xyz789",
    "total": 129.99,
    "items": [...],
    "shipping_address": {...}
  }
}

Thin payload — include only the ID and let the consumer fetch details:

{
  "type": "order.completed",
  "data": {
    "order_id": "ord_xyz789"
  }
}

Recommendation: Use fat payloads. They reduce the number of API calls consumers need to make and work better when network conditions are poor. Include a resource URL for consumers who want the latest state:

{
  "type": "order.completed",
  "data": {
    "order_id": "ord_xyz789",
    "total": 129.99,
    "url": "https://api.example.com/v1/orders/ord_xyz789"
  }
}

Delivery pipeline

Architecture

A reliable webhook system separates event creation from delivery:

1. Event creation — your application code creates an event record in the database
2. Queue — a background worker picks up pending events
3. Delivery — the worker sends the HTTP request
4. Retry — failed deliveries are retried with backoff
5. Dead letter — permanently failed events are stored for inspection

// Step 1: Create the event (in your application code)
async function emitEvent(type, data) {
  const event = createEvent(type, data);

  await db.webhookEvents.create({
    id: event.id,
    type: event.type,
    payload: JSON.stringify(event),
    status: 'pending',
    created_at: event.created_at,
  });

  // Enqueue for delivery
  await queue.publish('webhook-deliveries', { event_id: event.id });
}

// Step 2-3: Worker processes the queue
async function processWebhookDelivery(message) {
  const event = await db.webhookEvents.findById(message.event_id);
  if (!event || event.status === 'delivered') return;

  // Find all subscriptions for this event type
  const subscriptions = await db.webhookSubscriptions.findByEventType(event.type);

  for (const sub of subscriptions) {
    await deliverToEndpoint(event, sub);
  }
}

async function deliverToEndpoint(event, subscription) {
  const payload = JSON.parse(event.payload);
  const timestamp = Math.floor(Date.now() / 1000);
  const signature = signPayload(subscription.secret, timestamp, event.payload);

  try {
    const response = await fetch(subscription.url, {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
        'X-Webhook-Id': event.id,
        'X-Webhook-Timestamp': String(timestamp),
        'X-Webhook-Signature': `sha256=${signature}`,
        'User-Agent': 'ExampleApp-Webhooks/1.0',
      },
      body: event.payload,
      signal: AbortSignal.timeout(30000), // 30s timeout
    });

    if (response.status >= 200 && response.status < 300) {
      await db.webhookDeliveries.create({
        event_id: event.id,
        subscription_id: subscription.id,
        status: 'delivered',
        response_status: response.status,
        delivered_at: new Date(),
      });
    } else {
      throw new Error(`HTTP ${response.status}`);
    }
  } catch (err) {
    await scheduleRetry(event, subscription, err.message);
  }
}

Retry strategy

Use exponential backoff with jitter. A typical schedule:

Attempt	Delay	Cumulative
1	Immediate	0
2	1 minute	1 min
3	5 minutes	6 min
4	30 minutes	36 min
5	2 hours	~2.5 hours
6	8 hours	~10.5 hours
7	24 hours	~34.5 hours

const RETRY_DELAYS_MS = [
  0,
  60_000,           // 1 min
  300_000,          // 5 min
  1_800_000,        // 30 min
  7_200_000,        // 2 hours
  28_800_000,       // 8 hours
  86_400_000,       // 24 hours
];

async function scheduleRetry(event, subscription, errorMessage) {
  const delivery = await db.webhookDeliveries.findOrCreate({
    event_id: event.id,
    subscription_id: subscription.id,
  });

  const attempt = (delivery.attempt_count || 0) + 1;

  if (attempt >= RETRY_DELAYS_MS.length) {
    // Move to dead letter
    await db.webhookDeliveries.update(delivery.id, {
      status: 'failed',
      attempt_count: attempt,
      last_error: errorMessage,
    });

    // Optionally disable the subscription after too many failures
    await checkAndDisableEndpoint(subscription);
    return;
  }

  const delay = RETRY_DELAYS_MS[attempt];
  const jitter = Math.random() * delay * 0.1; // 10% jitter

  await db.webhookDeliveries.update(delivery.id, {
    status: 'retrying',
    attempt_count: attempt,
    last_error: errorMessage,
    next_retry_at: new Date(Date.now() + delay + jitter),
  });

  await queue.publishDelayed('webhook-deliveries', {
    event_id: event.id,
    subscription_id: subscription.id,
  }, delay + jitter);
}

Signature verification

Webhook consumers need to verify that requests actually came from your server. Use HMAC-SHA256 with a shared secret.

Producer side (your server)

import crypto from 'crypto';

function signPayload(secret, timestamp, body) {
  const signedContent = `${timestamp}.${body}`;
  return crypto
    .createHmac('sha256', secret)
    .update(signedContent)
    .digest('hex');
}

Consumer side (your user’s server)

import crypto from 'crypto';

function verifyWebhookSignature(req, secret) {
  const timestamp = req.headers['x-webhook-timestamp'];
  const signature = req.headers['x-webhook-signature'];

  // Reject old timestamps (prevent replay attacks)
  const age = Math.abs(Date.now() / 1000 - parseInt(timestamp));
  if (age > 300) { // 5 minutes
    throw new Error('Webhook timestamp too old');
  }

  const expectedSig = crypto
    .createHmac('sha256', secret)
    .update(`${timestamp}.${req.rawBody}`)
    .digest('hex');

  const expected = `sha256=${expectedSig}`;

  if (!crypto.timingSafeEqual(
    Buffer.from(signature),
    Buffer.from(expected)
  )) {
    throw new Error('Invalid webhook signature');
  }
}

// Express middleware for consuming webhooks
app.post('/webhooks/orders',
  express.raw({ type: 'application/json' }),
  (req, res) => {
    try {
      req.rawBody = req.body.toString();
      verifyWebhookSignature(req, process.env.WEBHOOK_SECRET);

      const event = JSON.parse(req.rawBody);
      // Process the event...
      handleOrderEvent(event);

      res.status(200).json({ received: true });
    } catch (err) {
      res.status(401).json({ error: err.message });
    }
  }
);

Idempotent event handling

Webhooks can be delivered more than once (retries, network issues). Consumers must handle duplicates:

async function handleOrderEvent(event) {
  // Check if we already processed this event
  const existing = await db.processedEvents.findById(event.id);
  if (existing) {
    console.log(`Event ${event.id} already processed, skipping`);
    return;
  }

  // Process the event
  await processOrder(event.data);

  // Mark as processed
  await db.processedEvents.create({
    id: event.id,
    type: event.type,
    processed_at: new Date(),
  });
}

Webhook management API

Give your users a REST API to manage their webhook subscriptions:

// Create a webhook subscription
app.post('/api/webhooks', authenticate, async (req, res) => {
  const { url, events } = req.body;

  // Generate a signing secret
  const secret = crypto.randomBytes(32).toString('hex');

  const subscription = await db.webhookSubscriptions.create({
    user_id: req.user.id,
    url,
    events,      // e.g., ["order.completed", "order.cancelled"]
    secret,
    active: true,
  });

  // Return the secret only once — user must save it
  res.status(201).json({
    data: {
      id: subscription.id,
      url: subscription.url,
      events: subscription.events,
      secret,    // only returned on creation
      active: subscription.active,
    },
  });
});

// List subscriptions
app.get('/api/webhooks', authenticate, async (req, res) => {
  const subs = await db.webhookSubscriptions.findByUserId(req.user.id);
  res.json({
    data: subs.map(s => ({
      id: s.id,
      url: s.url,
      events: s.events,
      active: s.active,
      // Do NOT return the secret
    })),
  });
});

// View delivery history
app.get('/api/webhooks/:id/deliveries', authenticate, async (req, res) => {
  const deliveries = await db.webhookDeliveries.findBySubscription(req.params.id);
  res.json({
    data: deliveries.map(d => ({
      event_id: d.event_id,
      event_type: d.event_type,
      status: d.status,
      response_status: d.response_status,
      attempt_count: d.attempt_count,
      last_error: d.last_error,
      delivered_at: d.delivered_at,
    })),
  });
});

// Manually retry a failed delivery
app.post('/api/webhooks/:id/deliveries/:eventId/retry', authenticate, async (req, res) => {
  await queue.publish('webhook-deliveries', {
    event_id: req.params.eventId,
    subscription_id: req.params.id,
  });
  res.status(202).json({ status: 'queued' });
});

Testing webhooks during development

Developers need a way to receive webhooks on localhost. Provide guidance and tooling:

# ngrok exposes localhost to the internet
ngrok http 3000

# Then register the webhook with the ngrok URL:
# https://abc123.ngrok.io/webhooks/orders

Better yet, provide a webhook testing endpoint in your dashboard that sends a test event to any registered URL:

app.post('/api/webhooks/:id/test', authenticate, async (req, res) => {
  const sub = await db.webhookSubscriptions.findById(req.params.id);

  const testEvent = createEvent('test.ping', {
    message: 'This is a test webhook delivery.',
    timestamp: new Date().toISOString(),
  });

  await deliverToEndpoint(
    { id: testEvent.id, payload: JSON.stringify(testEvent) },
    sub
  );

  res.json({ status: 'sent', event_id: testEvent.id });
});

Summary

A well-designed webhook system needs five things: a consistent event envelope with typed events, a delivery pipeline that separates event creation from HTTP delivery, exponential backoff retries with a dead letter queue, HMAC signature verification for security, and a management API that lets users register endpoints and inspect delivery history. Build for at-least-once delivery and document that consumers must handle duplicates idempotently. The result is a real-time integration layer that is far more efficient than polling.