System Design: Design an API Gateway

An API gateway sits between clients and your backend services. It is the single entry point for all API traffic, responsible for routing, authentication, rate limiting, protocol translation, and response aggregation. Without one, every microservice must independently handle cross-cutting concerns, leading to duplicated logic and inconsistent behavior.

Products like Kong, Envoy, and AWS API Gateway all solve this problem. Understanding how to design one from scratch reveals the trade-offs behind these tools.

Functional Requirements

• Route incoming HTTP requests to the appropriate backend service based on path, host, and headers.
• Authenticate requests using API keys, JWTs, or OAuth tokens.
• Enforce rate limits per client, per route, or globally.
• Transform requests and responses (header injection, body mapping, protocol translation).
• Aggregate responses from multiple backend services into a single response.
• Provide a management API for configuring routes and policies dynamically.

Non-Functional Requirements

• Low latency: the gateway adds overhead to every request, so it must be fast. Target under 5ms of added latency.
• High availability: the gateway is a single point of failure. It must be deployed redundantly.
• High throughput: handle tens of thousands of requests per second per node.
• Extensibility: new cross-cutting concerns should be easy to add without modifying core logic.

High-Level Architecture

The gateway has three layers:

1. Edge layer — load balancers (L4) distributing traffic across gateway nodes.
2. Gateway nodes — stateless processes that execute the request pipeline.
3. Control plane — a management service that stores and distributes configuration (routes, policies, plugins).

A request flows through the gateway node as follows: ingress, plugin chain (authentication, rate limiting, transformation), routing to the backend, response plugin chain, and egress.

The Request Pipeline

The core abstraction is a middleware pipeline (also called a filter chain). Each request passes through an ordered list of plugins. Each plugin can inspect or modify the request, short-circuit the pipeline (for example, return a 401), or pass the request to the next plugin.

Client -> [Auth Plugin] -> [Rate Limit Plugin] -> [Transform Plugin] -> [Router] -> Backend
                                                                                      |
Client <- [Response Transform] <- [Logging Plugin] <----------------------------------+

Plugins run in two phases: the request phase (before the backend call) and the response phase (after the backend responds). This dual-phase design lets plugins add response headers, modify response bodies, or record metrics about the complete request lifecycle.

Routing

The router maps incoming requests to backend services. A route configuration looks like:

{
  "id": "user-service",
  "match": {
    "hosts": ["api.example.com"],
    "paths": ["/users/*"],
    "methods": ["GET", "POST"]
  },
  "target": {
    "service": "user-service",
    "port": 8080,
    "load_balancing": "round_robin"
  },
  "plugins": ["jwt-auth", "rate-limit-100rpm"],
  "strip_prefix": true,
  "timeout_ms": 5000
}

For efficient matching, compile routes into a trie (prefix tree). Path segments form the trie nodes, and wildcards or regex segments are handled at leaf nodes. Host-based routing adds a first-level lookup by hostname before entering the path trie.

When routes change, the control plane pushes updated configurations to all gateway nodes. Use a watch mechanism (etcd watches, Redis pub/sub, or long polling) so nodes pick up changes within seconds without restarts.

Authentication

The auth plugin validates credentials and attaches identity information to the request context. Common strategies:

API keys: look up the key in a fast store (Redis or an in-memory cache backed by a database). Return 401 if the key is missing or revoked.

JWT validation: verify the token’s signature using the issuer’s public key (fetched and cached from a JWKS endpoint). Check expiration, audience, and required claims. This is entirely stateless, making it ideal for high-throughput gateways.

OAuth token introspection: for opaque tokens, call the authorization server’s introspection endpoint. Cache the result for the token’s remaining lifetime to avoid per-request overhead.

After authentication, inject the caller’s identity (user ID, scopes, roles) as headers so downstream services can authorize without re-validating the token.

Rate Limiting

The gateway enforces rate limits to protect backend services. The challenge is doing this across multiple gateway nodes.

Token bucket per client: each client gets a bucket that refills at a fixed rate. Use Redis to store bucket state, sharing it across all gateway nodes. The EVALSHA command with a Lua script atomically checks and decrements the bucket.

Sliding window counters: for simpler implementations, count requests per client in fixed time windows using Redis INCR with EXPIRE. A sliding window log gives more accuracy at the cost of more memory.

Rate limit responses should include standard headers: X-RateLimit-Limit, X-RateLimit-Remaining, and X-RateLimit-Reset so clients can self-throttle.

Request and Response Transformation

The transformation plugin modifies requests before they reach the backend and responses before they reach the client. Common transformations include:

• Header injection: add internal headers (request ID, caller identity, trace context).
• Path rewriting: strip or replace path prefixes when the public API structure differs from the internal service structure.
• Body mapping: translate between API versions, rename fields, or filter sensitive fields from responses.
• Protocol translation: accept GraphQL queries at the edge and translate them into REST calls to backend services, or accept REST and forward as gRPC.

Keep transformations declarative when possible (a JSON mapping spec) rather than requiring custom code for each route.

Configuration and Control Plane

The control plane stores all gateway configuration: routes, plugin settings, SSL certificates, and consumer credentials. It exposes a management API for operators and CI/CD pipelines.

Store configuration in a database (PostgreSQL) with a version number. When configuration changes, increment the version and notify gateway nodes via a pub/sub channel. Each gateway node caches the current configuration in memory and applies it atomically when a new version arrives.

This decouples the control plane from the data plane. If the control plane goes down, gateway nodes continue operating with their last known configuration. They just cannot receive updates until the control plane recovers.

Health Checks and Circuit Breaking

The gateway should actively health-check backend services rather than discovering failures through client requests.

Active health checks: periodically send a lightweight request (HTTP GET to /health) to each backend instance. Remove unhealthy instances from the load balancing pool. Restore them after consecutive successful checks.

Passive health checks: track error rates from real traffic. If a backend starts returning 5xx errors above a threshold, temporarily remove it (circuit breaker pattern). This reacts faster than active checks for sudden failures.

Combine both approaches. Active checks catch backends that are completely down. Passive checks catch backends that are up but misbehaving.

Scaling the Gateway

Gateway nodes are stateless, so horizontal scaling is straightforward. Place them behind an L4 load balancer (hardware or cloud-managed) that distributes connections by IP hash or least connections.

Connection pooling: maintain persistent connections to backend services. Creating a new TCP connection per request adds latency and exhausts ephemeral ports under load. A connection pool per backend service, sized to match the backend’s capacity, solves this.

CPU optimization: TLS termination is CPU-intensive. Use hardware acceleration (AES-NI) and session resumption (TLS session tickets) to reduce handshake costs. For very high traffic, terminate TLS at dedicated proxy nodes before the gateway.

Memory optimization: buffer as little of the request and response body as possible. Stream large payloads through the gateway rather than loading them into memory.

Observability

The gateway is the ideal place to capture system-wide observability data.

• Access logs: log every request with method, path, status, latency, and client identity. Ship logs to a centralized system for analysis.
• Metrics: emit request count, error rate, and latency histograms per route and per backend. Use Prometheus-compatible counters and histograms.
• Distributed tracing: inject or propagate trace context headers (W3C Trace Context or B3). The gateway should create the root span for each request.

Summary

An API gateway is a stateless reverse proxy powered by a plugin pipeline. The critical design decisions are the middleware architecture (which determines extensibility), the routing engine (which determines matching performance), and the separation of control plane from data plane (which determines operational resilience). Start with routing, authentication, and rate limiting. Add transformation and aggregation as the number of backend services grows.