Incident Management: Runbooks, On-Call, and Postmortems

Why Incident Management Matters

Every production system will eventually fail. The difference between a minor blip and a catastrophic outage often comes down to how well your team responds. Incident management is the structured process of detecting, responding to, and learning from production incidents.

Without a clear process, incidents devolve into chaos: multiple people making conflicting changes, unclear communication with stakeholders, and the same problems recurring because nobody captured what went wrong. A good incident management process reduces downtime, decreases stress for on-call engineers, and turns failures into opportunities to improve your systems.

Defining Severity Levels

Clear severity definitions ensure everyone agrees on the urgency of an incident. Here is a practical four-level framework:

Severity	Description	Response Time	Examples
SEV1	Complete service outage affecting all users	Immediate (< 5 min)	Database down, payment processing failed, data breach
SEV2	Major feature degraded, significant user impact	15 minutes	Search broken, API latency > 10x normal, partial data loss
SEV3	Minor feature affected, limited user impact	1 hour	Admin panel errors, delayed email notifications, cosmetic bugs
SEV4	No user impact, internal tooling issues	Next business day	CI/CD pipeline broken, staging environment down

Document these levels where everyone can find them. Link to them in your alerting tools so whoever gets paged knows immediately whether to wake up additional people.

Setting Up On-Call Rotations

On-call rotations distribute the responsibility of responding to incidents across your team. A sustainable rotation prevents burnout and ensures coverage.

Rotation Structure

A typical setup uses PagerDuty, Opsgenie, or Grafana OnCall:

# Example PagerDuty-style rotation config
rotation:
  name: "Backend Team Primary"
  type: weekly
  participants:
    - alice@company.com
    - bob@company.com
    - carol@company.com
    - dave@company.com
  handoff_time: "09:00"
  handoff_day: "Monday"
  timezone: "America/New_York"

escalation_policy:
  name: "Backend Escalation"
  rules:
    - targets:
        - type: rotation
          name: "Backend Team Primary"
      escalation_delay_minutes: 10
    - targets:
        - type: user
          name: "Engineering Manager"
      escalation_delay_minutes: 15
    - targets:
        - type: user
          name: "VP Engineering"
      escalation_delay_minutes: 20

On-Call Best Practices

One-week rotations work well for most teams. Shorter rotations create too much context switching; longer ones lead to burnout.

Follow-the-sun rotations for distributed teams ensure nobody gets paged at 3 AM. If your team spans US and European time zones, split the 24-hour coverage between them.

Compensate on-call engineers with time off, extra pay, or both. Being on-call is real work, and treating it as an unpaid obligation will drive engineers away.

Limit alert volume. If your on-call engineer gets paged more than twice per shift on average, your alerts need tuning. High alert volume leads to alert fatigue, where engineers start ignoring pages.

Shadow rotations let new team members learn the on-call process by pairing with an experienced responder before taking primary on-call themselves.

Building Effective Runbooks

A runbook is a documented procedure for diagnosing and resolving a specific type of incident. When an engineer gets paged at 2 AM, they need clear, actionable steps rather than having to reason through the problem from scratch.

Runbook Template

# runbook: high-api-latency.md
---
title: "High API Latency"
severity: SEV2
alert_name: "api_latency_p99_high"
last_updated: 2026-07-01
owner: backend-team
---

## Symptoms
- API p99 latency exceeds 2 seconds
- Users report slow page loads
- Alert: api_latency_p99_high fires in Grafana

## Impact
- User-facing API responses are slow
- Downstream services may time out
- Revenue impact if checkout flow is affected

## Quick Diagnosis

### Step 1: Check current latency
```bash
curl -w "Total: %{time_total}s\n" -o /dev/null -s https://api.example.com/health

Step 2: Check resource utilization

# CPU and memory on API pods
kubectl top pods -n production -l app=api

# Database connection pool
kubectl exec -n production deploy/api -- curl -s localhost:9090/metrics | grep db_pool

Step 3: Check recent deployments

kubectl rollout history deployment/api -n production

Common Causes and Fixes

Database slow queries

1. Check slow query log: Grafana dashboard “Database Performance”
2. If a specific query is slow, check for missing indexes
3. Temporary fix: increase database read replicas

Memory pressure

1. Check pod memory usage against limits
2. If pods are near limits, restart them: kubectl rollout restart deployment/api -n production
3. Long-term: investigate memory leaks or increase limits

Traffic spike

1. Check request rate on the “API Traffic” dashboard
2. Verify autoscaler is working: kubectl get hpa -n production
3. Manually scale if needed: kubectl scale deployment/api --replicas=10 -n production

Escalation

• If not resolved in 30 minutes, page the database team
• If customer-facing impact lasts > 1 hour, notify the incident commander

### Runbook Principles

**Be specific.** "Check the logs" is not helpful. "Run `kubectl logs -n production -l app=api --tail=100 | grep ERROR`" is helpful.

**Include the expected output.** Tell the reader what normal looks like so they can identify what is abnormal.

**Keep them updated.** Runbooks rot quickly. Assign owners and review them quarterly. After every incident, update the relevant runbook with what you learned.

**Link from alerts.** Every alert should include a link to the relevant runbook. When PagerDuty fires, the engineer should be one click away from the diagnosis steps.

## The Incident Response Process

When an incident occurs, follow a structured response process:

### 1. Detect and Acknowledge

The on-call engineer acknowledges the page, confirming they are working on it. This stops the escalation timer.

### 2. Assess Severity

Based on the alert and initial investigation, assign a severity level. This determines who else needs to be involved and how communication should flow.

### 3. Assemble the Response Team

For SEV1 and SEV2 incidents, open a dedicated incident channel:

Slack channel: #inc-2026-07-07-api-outage

Assign roles:
- **Incident Commander (IC)**: Coordinates the response, makes decisions about escalation and communication. Does not debug.
- **Technical Lead**: Leads the debugging effort.
- **Communications Lead**: Updates stakeholders, writes status page updates.
- **Scribe**: Documents the timeline of actions and decisions in the incident channel.

### 4. Communicate

Post a status update template at the start:

Incident: API Latency Degradation Severity: SEV2 Impact: API response times elevated, affecting ~30% of requests Status: Investigating IC: Alice Next update: 15 minutes

Update stakeholders at regular intervals, even if the update is "still investigating." Silence creates anxiety.

### 5. Mitigate

Focus on restoring service first, root cause analysis second. Common mitigation actions:

- Roll back a recent deployment
- Scale up capacity
- Redirect traffic to a healthy region
- Enable a feature flag to disable a problematic feature
- Restart affected services

### 6. Resolve and Close

Once the service is restored, confirm with monitoring that metrics have returned to normal. Close the incident channel with a summary and schedule a postmortem if the incident was SEV1 or SEV2.

## Running Blameless Postmortems

A postmortem (also called a retrospective or incident review) is a structured meeting to learn from an incident. The most important principle is that postmortems must be **blameless**. You are looking for systemic causes, not individuals to punish.

### Postmortem Document Template

```yaml
# Postmortem: API Outage 2026-07-07
---
date: 2026-07-07
duration: 47 minutes
severity: SEV2
authors: [Alice, Bob]
reviewers: [Carol, Engineering Manager]
status: Complete
---

## Summary
On July 7th from 14:23 to 15:10 UTC, the API experienced elevated
latency affecting approximately 30% of requests. The root cause was
a database migration that added a full table scan query without an
appropriate index.

## Impact
- 30% of API requests experienced latency > 5 seconds
- 12 customer support tickets filed
- Estimated revenue impact: $2,400

## Timeline (all times UTC)
- 14:15 - Deploy v2.14.0 containing migration #487
- 14:23 - Latency alert fires, Alice acknowledges
- 14:28 - Alice identifies database CPU at 95%
- 14:35 - Bob joins, identifies slow query from migration #487
- 14:42 - Decision to roll back v2.14.0
- 14:48 - Rollback deployed, latency begins recovering
- 15:10 - All metrics returned to normal, incident closed

## Root Cause
Migration #487 added a new query that performed a sequential scan on
the orders table (42M rows) without an index on the filter column.
Under production load, this saturated the database CPU.

## What Went Well
- Alert fired within 8 minutes of degradation
- Runbook for high API latency guided initial diagnosis
- Rollback was executed smoothly

## What Went Poorly
- Migration was not tested against production-sized data
- No query performance review in the deployment checklist
- Took 20 minutes to identify the specific query

## Action Items
| Action | Owner | Priority | Due Date |
|--------|-------|----------|----------|
| Add query performance testing to CI pipeline | Bob | P1 | 2026-07-14 |
| Require EXPLAIN ANALYZE for new queries in PR review | Carol | P2 | 2026-07-21 |
| Add database query latency to deployment canary checks | Alice | P1 | 2026-07-14 |
| Update high-latency runbook with database query diagnosis steps | Alice | P3 | 2026-07-28 |

Running the Meeting

Schedule within 48 hours of the incident while details are fresh. Invite everyone involved plus relevant stakeholders.

Walk through the timeline chronologically. Ask “what happened next?” rather than “why did you do that?”

Focus on systems, not people. Instead of “Bob missed the slow query,” ask “What process could have caught this slow query before production?”

Generate concrete action items with owners, priorities, and due dates. Track them in your project management tool and review completion in subsequent team meetings.

Share the postmortem broadly. Other teams learn from your incidents, and transparency builds a culture of learning rather than hiding failures.

Measuring Incident Management

Track metrics to improve your process over time:

• MTTD (Mean Time to Detect): How long between a problem starting and an alert firing.
• MTTA (Mean Time to Acknowledge): How long between an alert and someone responding.
• MTTR (Mean Time to Resolve): How long between detection and full resolution.
• Incident frequency: How many incidents occur per week or month, broken down by severity.
• Action item completion rate: What percentage of postmortem action items are completed on time.

Review these metrics monthly. If MTTD is high, your monitoring needs improvement. If MTTR is high, your runbooks or on-call training need work. If incident frequency is climbing, you have a systemic reliability problem.

Wrapping Up

Effective incident management combines clear severity definitions, sustainable on-call practices, actionable runbooks, structured response procedures, and blameless postmortems. None of these components are difficult individually, but the discipline to maintain them consistently is what separates reliable organizations from chaotic ones. Start by documenting your severity levels and building runbooks for your three most common alerts. From there, formalize your postmortem process and begin tracking incident metrics. The goal is not to prevent all incidents but to respond to them quickly, learn from them systematically, and build increasingly resilient systems over time.