Graceful Shutdown: Finishing Work Before Exiting

When a deployment replaces your running service, the orchestrator sends a termination signal. If your service exits immediately, in-flight HTTP requests get broken connections, database transactions are left uncommitted, and background jobs lose their progress. Graceful shutdown is the practice of finishing current work before exiting.

Every production service needs this. Kubernetes sends SIGTERM and waits 30 seconds before SIGKILL. Docker Compose does the same with a 10-second default. AWS ECS, systemd, and Heroku all follow the same pattern. Your service must listen for the signal and use that window to wind down cleanly.

The shutdown sequence

A well-behaved service follows this sequence when it receives SIGTERM:

1. Stop accepting new work. Close the listening socket so the load balancer stops sending requests.
2. Finish in-flight work. Let active HTTP requests complete, let running background jobs finish their current item.
3. Close resources. Flush logs, close database connections, disconnect from message brokers.
4. Exit with code 0.

If any step takes too long, a hard timeout forces exit to prevent the process from hanging indefinitely.

Signal handling in Node.js

import { createServer } from 'http';

const server = createServer(app);
let isShuttingDown = false;

server.listen(3000, () => {
  console.log('Server listening on port 3000');
});

async function shutdown(signal) {
  if (isShuttingDown) return;
  isShuttingDown = true;
  console.log(`${signal} received. Starting graceful shutdown.`);

  // 1. Stop accepting new connections
  server.close(async () => {
    console.log('HTTP server closed. No new connections.');

    try {
      // 2. Finish background work
      await backgroundWorker.drain();

      // 3. Close resources
      await database.end();
      await redis.quit();

      console.log('Shutdown complete.');
      process.exit(0);
    } catch (err) {
      console.error('Error during shutdown:', err);
      process.exit(1);
    }
  });

  // 4. Hard timeout: force exit if graceful shutdown takes too long
  setTimeout(() => {
    console.error('Graceful shutdown timed out. Forcing exit.');
    process.exit(1);
  }, 25000); // Must be less than orchestrator's kill timeout
}

process.on('SIGTERM', () => shutdown('SIGTERM'));
process.on('SIGINT', () => shutdown('SIGINT'));

The server.close() call stops accepting new TCP connections but lets existing connections finish. The callback fires when all connections have ended or the keep-alive timeout expires.

Handling in-flight requests

server.close() waits for keep-alive connections to end naturally. If clients hold connections open, this can take minutes. Add a middleware that rejects new requests during shutdown:

app.use((req, res, next) => {
  if (isShuttingDown) {
    res.set('Connection', 'close');
    res.status(503).json({ error: 'Service is shutting down' });
    return;
  }
  next();
});

Setting Connection: close tells the client to stop reusing the connection. The server can then close it after the response.

For tracking in-flight requests precisely:

let activeRequests = 0;

app.use((req, res, next) => {
  activeRequests++;
  res.on('finish', () => { activeRequests--; });
  next();
});

async function waitForRequests(timeoutMs) {
  const deadline = Date.now() + timeoutMs;
  while (activeRequests > 0 && Date.now() < deadline) {
    await new Promise(resolve => setTimeout(resolve, 100));
  }
}

Draining background workers

Background job processors (Bull, BullMQ, Celery, Sidekiq) need their own shutdown handling. The pattern is the same: stop pulling new jobs, finish the current one, then exit.

import { Worker } from 'bullmq';

const worker = new Worker('emails', async (job) => {
  await sendEmail(job.data);
}, { connection: redis });

// During shutdown
async function drainWorker() {
  await worker.close(); // Finishes current job, stops pulling new ones
}

If a job takes too long, the hard timeout will kill the process and the job will be retried by the queue (assuming at-least-once delivery semantics). Design jobs to be idempotent so retries are safe.

Python (Flask/Gunicorn)

Gunicorn handles graceful shutdown at the master process level. When it receives SIGTERM, it sends SIGTERM to workers, waits for graceful_timeout seconds, then sends SIGKILL.

# gunicorn.conf.py
graceful_timeout = 25
timeout = 30

For custom cleanup in a Flask app:

import signal
import sys

def shutdown_handler(signum, frame):
    print("Shutting down gracefully...")
    db.session.remove()
    db.engine.dispose()
    sys.exit(0)

signal.signal(signal.SIGTERM, shutdown_handler)

Go

Go’s http.Server has built-in graceful shutdown support:

srv := &http.Server{Addr: ":8080", Handler: mux}

go func() {
    if err := srv.ListenAndServe(); err != http.ErrServerClosed {
        log.Fatalf("Server error: %v", err)
    }
}()

quit := make(chan os.Signal, 1)
signal.Notify(quit, syscall.SIGTERM, syscall.SIGINT)
<-quit

ctx, cancel := context.WithTimeout(context.Background(), 25*time.Second)
defer cancel()

if err := srv.Shutdown(ctx); err != nil {
    log.Printf("Forced shutdown: %v", err)
}

db.Close()
log.Println("Server exited cleanly")

srv.Shutdown(ctx) stops accepting new connections and waits for active ones to complete, up to the context deadline.

Health check coordination

In Kubernetes, the sequence is: Pod receives SIGTERM, then (concurrently) the Service removes the Pod from its endpoints. There is a race: the load balancer might still send traffic for a few seconds after SIGTERM arrives.

The fix is a pre-stop delay or a readiness probe that goes unhealthy during shutdown:

app.get('/healthz', (req, res) => {
  if (isShuttingDown) {
    return res.status(503).send('shutting down');
  }
  res.status(200).send('ok');
});

In the Kubernetes pod spec:

lifecycle:
  preStop:
    exec:
      command: ["sleep", "5"]
readinessProbe:
  httpGet:
    path: /healthz
    port: 3000
  periodSeconds: 5

The 5-second preStop sleep gives the load balancer time to remove the pod from rotation before the application starts rejecting requests.

Timeout strategy

You need two timeouts:

1. Application shutdown timeout: How long your code waits for in-flight work. Set this to 20-25 seconds.
2. Orchestrator kill timeout: How long the orchestrator waits before sending SIGKILL. Kubernetes default is 30 seconds (terminationGracePeriodSeconds).

The application timeout must be shorter than the orchestrator timeout. If your app needs 25 seconds, set the orchestrator to 35 seconds. This gives your code time to clean up before the process is forcefully terminated.

# Kubernetes pod spec
spec:
  terminationGracePeriodSeconds: 35

Common mistakes

Not handling signals at all. The default behavior for SIGTERM in most runtimes is immediate termination. Without a handler, your service drops all in-flight work.

Calling process.exit() too early. Exiting before async cleanup completes (database flushes, log flushes) loses data. Always await cleanup before exiting.

Infinite shutdown. A shutdown handler that waits forever for a stuck database connection prevents the process from ever exiting. Always set a hard timeout.

Ignoring SIGINT. Developers press Ctrl+C during local development. Handle SIGINT the same way as SIGTERM for a consistent experience.

Graceful shutdown is not optional for production services. It is the difference between zero-downtime deployments and a stream of 502 errors every time you push code. Handle the signal, drain the work, close the resources, and exit cleanly.