Health Checks and Readiness Probes Done Right

A health check endpoint tells infrastructure whether your service is working. Load balancers use it to route traffic away from broken instances. Orchestrators use it to restart hung processes. Monitoring tools use it to trigger alerts. Despite being simple in concept, health checks are frequently implemented in ways that cause more outages than they prevent.

The most common mistake is a health check that tests too much. When your /health endpoint queries the database, checks Redis, pings three external APIs, and verifies disk space, a temporary blip in any one of those systems makes your service report as unhealthy. The load balancer pulls it from rotation. Now you have an outage caused by your health check, not by your service.

Liveness vs readiness

Modern orchestrators like Kubernetes distinguish between two questions:

Liveness: “Is the process stuck?” If the liveness check fails, the orchestrator kills the container and starts a new one. This should detect deadlocks, infinite loops, and unresponsive event loops. It should not check external dependencies.

Readiness: “Can this instance handle traffic?” If the readiness check fails, the orchestrator stops sending traffic to this instance but does not kill it. The instance stays alive and may recover. This is where dependency checks belong.

Liveness:  "Can you respond at all?"     -> Kill if no
Readiness: "Should I send you traffic?"  -> Remove from pool if no

Implementing liveness

A liveness endpoint should do almost nothing. If the HTTP server can accept a connection and return a response, the process is alive:

app.get('/livez', (req, res) => {
  res.status(200).json({ status: 'alive' });
});

That is the entire implementation. No database check, no cache check, no dependency check. If this endpoint stops responding, the process is truly stuck and should be restarted.

Some teams add a basic event loop check for Node.js:

let lastTick = Date.now();

setInterval(() => {
  lastTick = Date.now();
}, 1000);

app.get('/livez', (req, res) => {
  const lag = Date.now() - lastTick;
  if (lag > 5000) {
    return res.status(503).json({
      status: 'stuck',
      eventLoopLagMs: lag,
    });
  }
  res.status(200).json({ status: 'alive' });
});

If the event loop is blocked for more than 5 seconds, something is seriously wrong and a restart is warranted.

Implementing readiness

The readiness endpoint checks whether the service can actually do useful work. This means checking critical dependencies:

app.get('/readyz', async (req, res) => {
  const checks = {};
  let healthy = true;

  // Database check
  try {
    await pool.query('SELECT 1');
    checks.database = { status: 'up' };
  } catch (err) {
    checks.database = { status: 'down', error: err.message };
    healthy = false;
  }

  // Redis check
  try {
    await redis.ping();
    checks.redis = { status: 'up' };
  } catch (err) {
    checks.redis = { status: 'down', error: err.message };
    healthy = false;
  }

  const statusCode = healthy ? 200 : 503;
  res.status(statusCode).json({
    status: healthy ? 'ready' : 'not_ready',
    checks,
    timestamp: new Date().toISOString(),
  });
});

When the database is down, the readiness check fails, the load balancer stops routing traffic to this instance, and users see a clean error from a healthy instance rather than a timeout from a broken one.

Startup probes

Some applications take a long time to start: loading ML models, warming caches, running migrations. Without a startup probe, Kubernetes might kill the container before it finishes initializing because the liveness probe fails.

A startup probe runs during initialization and replaces the liveness probe until it succeeds. Once it passes, Kubernetes switches to the regular liveness probe.

let initialized = false;

async function startup() {
  await runMigrations();
  await warmCache();
  await loadModel();
  initialized = true;
}

app.get('/startupz', (req, res) => {
  if (initialized) {
    return res.status(200).json({ status: 'started' });
  }
  res.status(503).json({ status: 'starting' });
});

Kubernetes probe configuration

apiVersion: v1
kind: Pod
spec:
  containers:
    - name: api
      image: myapp:latest
      ports:
        - containerPort: 3000
      startupProbe:
        httpGet:
          path: /startupz
          port: 3000
        failureThreshold: 30
        periodSeconds: 2
        # Allows up to 60 seconds for startup
      livenessProbe:
        httpGet:
          path: /livez
          port: 3000
        initialDelaySeconds: 0
        periodSeconds: 10
        failureThreshold: 3
        timeoutSeconds: 2
      readinessProbe:
        httpGet:
          path: /readyz
          port: 3000
        periodSeconds: 5
        failureThreshold: 2
        timeoutSeconds: 3

Key settings:

• periodSeconds: How often the probe runs. 5-10 seconds for liveness, 3-5 seconds for readiness.
• failureThreshold: How many consecutive failures before action is taken. Set liveness to 3 (allow transient hiccups). Set readiness to 1-2 (remove from pool quickly).
• timeoutSeconds: How long to wait for a response. A health check that takes 5 seconds is itself a problem. Keep this at 2-3 seconds.

Dependency check timeouts

Never let a dependency check block the health endpoint indefinitely. Set aggressive timeouts:

async function checkWithTimeout(fn, timeoutMs) {
  return Promise.race([
    fn(),
    new Promise((_, reject) =>
      setTimeout(() => reject(new Error('Health check timeout')), timeoutMs)
    ),
  ]);
}

// In the readiness handler
try {
  await checkWithTimeout(() => pool.query('SELECT 1'), 2000);
  checks.database = { status: 'up' };
} catch (err) {
  checks.database = { status: 'down', error: err.message };
  healthy = false;
}

If the database takes more than 2 seconds to respond to SELECT 1, something is wrong enough to stop sending traffic.

Degraded states

Not every dependency failure should mark the service as unready. If your service can function without the cache (falling back to the database), the cache being down is a degradation, not an outage:

app.get('/readyz', async (req, res) => {
  const checks = {};
  let healthy = true;
  let degraded = false;

  // Critical: must be up
  try {
    await checkWithTimeout(() => pool.query('SELECT 1'), 2000);
    checks.database = { status: 'up' };
  } catch (err) {
    checks.database = { status: 'down' };
    healthy = false;
  }

  // Non-critical: service works without it
  try {
    await checkWithTimeout(() => redis.ping(), 1000);
    checks.cache = { status: 'up' };
  } catch (err) {
    checks.cache = { status: 'degraded' };
    degraded = true;
  }

  const status = !healthy ? 'not_ready' : degraded ? 'degraded' : 'ready';
  const statusCode = healthy ? 200 : 503;

  res.status(statusCode).json({ status, checks });
});

A degraded status returns 200 (so traffic keeps flowing) but signals to monitoring that something needs attention.

Avoiding cascading failures

A common anti-pattern: Service A’s readiness check calls Service B’s health endpoint. Service B’s readiness check calls Service C. If Service C goes down, Services A and B both report as unhealthy, even though they could still serve many requests.

Rules for dependency checks in readiness probes:

1. Only check direct dependencies: database, cache, message broker.
2. Never check downstream services. If Service B is down, let requests to Service B fail individually rather than pulling Service A out of rotation entirely.
3. Use cached health status for expensive checks. Poll the dependency every 10 seconds in the background and serve the cached result in the health endpoint.

let dbHealthy = true;

// Background check every 10 seconds
setInterval(async () => {
  try {
    await checkWithTimeout(() => pool.query('SELECT 1'), 2000);
    dbHealthy = true;
  } catch {
    dbHealthy = false;
  }
}, 10000);

app.get('/readyz', (req, res) => {
  const status = dbHealthy ? 'ready' : 'not_ready';
  res.status(dbHealthy ? 200 : 503).json({ status });
});

This prevents the health check itself from adding load to an already struggling database.

Security considerations

Health check endpoints should not leak sensitive information. The detailed dependency status with error messages is useful for internal monitoring but should not be exposed publicly:

app.get('/readyz', async (req, res) => {
  const detailed = req.headers['x-health-detail'] === process.env.HEALTH_SECRET;
  const result = await runChecks();

  if (detailed) {
    return res.status(result.healthy ? 200 : 503).json(result);
  }

  res.status(result.healthy ? 200 : 503).json({
    status: result.healthy ? 'ready' : 'not_ready',
  });
});

Alternatively, bind detailed health endpoints to an internal port that is not exposed through the load balancer.

Health checks look trivial but their design directly affects your service’s availability. Keep liveness simple, make readiness reflect real capability, set aggressive timeouts, and never let a health check create the outage it was supposed to prevent.