Incident Management and Writing Runbooks

Why Incident Management Matters

Without a defined process, incidents devolve into chaos. Five engineers are in a Slack thread, each investigating a different theory, nobody is communicating with customers, and the person who actually knows the system is asleep because nobody thought to page them.

A structured incident management process does not prevent outages. It reduces the time to resolve them and prevents the organizational dysfunction that makes bad situations worse.

Incident Severity Levels

Define severity levels before you need them. During an incident is the wrong time to debate whether something is a SEV1 or SEV2.

Severity	Criteria	Response
SEV1	Complete service outage affecting all users, data loss, security breach	Page immediately, all hands, status page updated every 15 min
SEV2	Partial outage, degraded experience for significant user segment	Page on-call, dedicated incident channel, status page updated every 30 min
SEV3	Minor degradation, workaround available, limited user impact	On-call investigates during business hours, no page overnight
SEV4	Cosmetic issues, minor bugs, no user-facing impact	Track in issue tracker, fix in normal sprint

The exact definitions depend on your business. For a payments company, any transaction failure is SEV1. For a social media app, a slow-loading feed might be SEV3. Define what matters to your users and your business.

Incident Roles

Assign explicit roles during an incident. Without roles, everyone tries to do everything and nothing gets done efficiently.

Incident Commander (IC): Owns the incident. Makes decisions about escalation, communication, and prioritization. Does not debug. Coordinates the people who debug. The IC’s job is to ask “what is the current theory?” and “who is working on what?” every five minutes.

Technical Lead: The person actively investigating and implementing fixes. Communicates findings to the IC. Can request additional responders.

Communications Lead: Updates the status page, communicates with customer support, and posts updates to internal channels. Shields the technical lead from interruptions.

Scribe: Documents the timeline in real time. Records what was tried, what worked, what did not. This becomes the foundation for the postmortem.

For small teams, one person may fill multiple roles. The IC and communications lead can be the same person. But the IC and technical lead should be different people whenever possible, because debugging requires focused attention that coordination prevents.

The Incident Lifecycle

1. Detection

An alert fires, a customer reports an issue, or an engineer notices something wrong. The first person to detect the problem declares an incident and assigns a severity.

2. Triage

The on-call engineer assesses the situation. What is the impact? How many users are affected? Is there a workaround? They assign a severity level and create an incident channel.

/incident declare
Title: Checkout page returning 500 errors
Severity: SEV2
Channel: #inc-2024-0712-checkout-500s

3. Response

The IC assembles the right people. The technical lead starts investigating. The communications lead posts an initial status page update.

Communication cadence matters. In the incident channel, post updates every 10 to 15 minutes even if the update is “still investigating, no new findings.” Silence creates anxiety and causes people to interrupt the technical lead asking for status.

4. Mitigation

The goal is to stop the bleeding, not to find the root cause. Rolling back a deployment, failing over to a secondary region, or toggling a feature flag are all valid mitigations. Fix it properly later. Right now, restore service.

5. Resolution

The incident is resolved when user impact has ended and the service is stable. The IC confirms resolution, updates the status page, and schedules a postmortem.

Writing Runbooks

A runbook is a step-by-step document that tells the on-call engineer what to do when a specific alert fires. The target audience is someone who is half-awake, stressed, and may not be deeply familiar with the system.

Runbook Template

# Runbook: HighErrorRate on Checkout Service

## Alert
- Name: HighErrorRate
- Severity: SEV2 when error rate > 5%, SEV1 when > 20%
- Dashboard: https://grafana.internal/d/checkout-service

## Impact
Users cannot complete purchases. Revenue is directly affected.

## Quick Diagnosis

1. Check the error rate dashboard. Is it all endpoints or specific ones?
   - Dashboard: https://grafana.internal/d/checkout-errors-by-endpoint

2. Check recent deployments (last 2 hours):
   ```bash
   kubectl -n production rollout history deployment/checkout-service

3. Check downstream dependencies:
   • Payment provider status: https://status.stripe.com
   • Database: https://grafana.internal/d/rds-checkout
   • Redis cache: https://grafana.internal/d/redis-checkout

Common Causes and Fixes

Recent deployment caused the issue

# Roll back to previous version
kubectl -n production rollout undo deployment/checkout-service

# Verify error rate is dropping
# Wait 2-3 minutes, check dashboard

Database connection pool exhaustion

# Check active connections
kubectl -n production exec -it deployment/checkout-service -- \
  curl localhost:8080/debug/db-pool

# If connections are maxed, restart pods one at a time
kubectl -n production rollout restart deployment/checkout-service

Payment provider outage

• Check https://status.stripe.com
• If Stripe is down, enable the payment queue feature flag:

  curl -X POST https://feature-flags.internal/api/flags/payment-queue \
    -H "Authorization: Bearer $TOKEN" \
    -d '{"enabled": true}'

• This queues payments and processes them when Stripe recovers.
• Notify #payments-team in Slack.

Escalation

• If not resolved in 30 minutes: page the checkout team lead
• If data inconsistency suspected: page the database team
• Escalation contacts: https://wiki.internal/on-call-contacts

### Runbook Principles

**Be specific.** "Check the logs" is useless. "Run `kubectl logs -n production deployment/checkout-service --tail=100 | grep ERROR`" is useful.

**Include links.** Every dashboard, status page, and wiki reference should be a clickable link. Do not make someone search for the Grafana dashboard while the site is down.

**Order by likelihood.** Put the most common cause first. If 70 percent of checkout errors are caused by bad deployments, the rollback instructions should be step one.

**Include verification steps.** After each fix, tell the reader how to confirm it worked. "Wait 2-3 minutes, check the error rate dashboard. It should be below 1 percent."

**Keep it current.** A runbook that references a service that was renamed six months ago is worse than no runbook because it wastes time. Review runbooks quarterly. Add a "last reviewed" date at the top.

## Blameless Postmortems

Every SEV1 and SEV2 incident should have a postmortem within five business days. The postmortem is not about finding who screwed up. It is about finding what in your systems, processes, and tooling allowed the failure to happen and how to prevent recurrence.

### Postmortem Template

```markdown
# Postmortem: Checkout 500 Errors - July 12, 2026

## Summary
Checkout service returned 500 errors for 23 minutes affecting
approximately 1,200 users. Estimated revenue impact: $18,000.

## Timeline (all times UTC)
- 14:32 - Deployment of checkout-service v2.4.1 begins
- 14:35 - Deployment completes, new pods pass readiness checks
- 14:38 - Error rate alert fires (HighErrorRate > 5%)
- 14:40 - On-call engineer acknowledges, creates incident channel
- 14:42 - IC assigned, begins triage
- 14:45 - Root cause identified: new code path hits unindexed query
- 14:47 - Decision to roll back
- 14:48 - Rollback initiated
- 14:51 - Rollback complete, error rate returning to normal
- 14:55 - Error rate below 1%, incident resolved

## Root Cause
Version 2.4.1 introduced a new product recommendation query that
performed a full table scan on the products table (3.2M rows).
Under load, this exhausted the database connection pool within
3 minutes of deployment.

## Contributing Factors
- The query was not tested with production-scale data
- Load testing in staging uses a 10K row dataset
- Database connection pool exhaustion was not covered in readiness checks
- The change was reviewed but the reviewer did not check query plans

## Action Items
| Action | Owner | Due Date | Status |
|---|---|---|---|
| Add index on products(category, score) | @alice | July 15 | Done |
| Add staging dataset with 1M+ rows | @bob | July 22 | In Progress |
| Add DB connection pool metric to readiness check | @charlie | July 19 | Open |
| Add query plan review to PR checklist | @alice | July 15 | Done |
| Create alert for connection pool saturation > 80% | @dave | July 17 | Open |

## Lessons Learned
- Readiness checks that only test HTTP 200 are insufficient.
  They should verify that critical dependencies are healthy.
- Staging environments with unrealistic data volumes hide
  performance problems that only appear in production.

What Makes a Good Postmortem

Concrete action items with owners and due dates. “We should improve testing” is not an action item. “Add a staging dataset with 1M+ product rows, owned by Bob, due July 22” is an action item.

Follow-up tracking. Action items that are never completed are worse than no action items because they create a false sense of progress. Track them in your issue tracker and review completion rates monthly.

No blame. The person who deployed the bad code is not the root cause. The system that allowed untested code to reach production is the root cause. If a single human mistake can cause an outage, the system is fragile.

Share widely. Post the postmortem where everyone can read it. Other teams learn from your incidents. Hiding postmortems means every team makes the same mistakes independently.

Getting Started

If you have no incident process today, start with three things. First, define your severity levels and write them down where everyone can find them. Second, write one runbook for your most common alert. Third, run one postmortem for your next incident, even if it is a SEV3. The practice of writing it down and identifying action items is more valuable than the template you use.

Incident management is a skill. Like any skill, it improves with practice. Run game days where you simulate incidents and practice the process. The first time you use your incident process should not be during a real outage.