SRE Golden Signals Monitoring Guide

Google’s Site Reliability Engineering book introduced the concept of four golden signals: latency, traffic, errors, and saturation. These four metrics give you a surprisingly complete picture of any service’s health. Instead of drowning in hundreds of custom metrics, you focus on the signals that actually tell you whether users are having a good time or a bad one.

The Four Signals

Latency measures how long it takes to serve a request. The key distinction is between successful requests and failed requests. A fast error is still an error. Track latency for successful responses separately so that a flood of quick 500s does not drag down your perceived latency.

Traffic measures the demand on your system. For an HTTP service this is requests per second. For a streaming service it might be concurrent connections. For a batch pipeline it could be records processed per minute. Pick the unit that represents real user demand.

Errors measure the rate of requests that fail. This includes explicit failures like HTTP 5xx responses, implicit failures like a 200 response with wrong content, and policy failures like responses that exceed a latency SLO.

Saturation measures how full your service is. CPU utilization, memory pressure, queue depth, disk I/O, thread pool usage. Saturation signals tell you how close you are to the cliff before you fall off it.

Instrumenting with Prometheus

Most services expose golden signal metrics through a combination of histograms, counters, and gauges. Here is a practical setup for an HTTP service.

Latency

Use a histogram to capture request duration distributions:

# prometheus.yml scrape config
scrape_configs:
  - job_name: api
    metrics_path: /metrics
    static_configs:
      - targets: ['api:8080']

Your application exposes a histogram like http_request_duration_seconds. Query the 95th percentile latency for successful requests:

histogram_quantile(0.95,
  sum(rate(http_request_duration_seconds_bucket{
    job="api", status!~"5.."}[5m]
  )) by (le)
)

Alert when p95 latency exceeds your SLO:

groups:
  - name: golden-signals
    rules:
      - alert: HighLatencyP95
        expr: |
          histogram_quantile(0.95,
            sum(rate(http_request_duration_seconds_bucket{
              job="api", status!~"5.."}[5m]
            )) by (le)
          ) > 0.5
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "P95 latency above 500ms for 5 minutes"

Traffic

Track requests per second using a counter:

sum(rate(http_requests_total{job="api"}[5m]))

Break it down by endpoint to see where demand concentrates:

sum(rate(http_requests_total{job="api"}[5m])) by (handler)

Traffic alerts are useful for detecting sudden drops, which often indicate an upstream problem or a DNS issue:

      - alert: TrafficDrop
        expr: |
          sum(rate(http_requests_total{job="api"}[5m]))
          < 0.5 * sum(rate(http_requests_total{job="api"}[5m] offset 1h))
        for: 10m
        labels:
          severity: critical
        annotations:
          summary: "Traffic dropped below 50% of one hour ago"

Errors

Calculate the error rate as a percentage of total traffic:

sum(rate(http_requests_total{job="api", status=~"5.."}[5m]))
/
sum(rate(http_requests_total{job="api"}[5m]))
* 100

Alert when errors exceed your error budget burn rate. A simple version:

      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{job="api", status=~"5.."}[5m]))
          /
          sum(rate(http_requests_total{job="api"}[5m]))
          > 0.01
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Error rate above 1% for 5 minutes"

Saturation

Saturation depends on the resource. For CPU:

1 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) by (instance)

For memory pressure on a container:

container_memory_usage_bytes{container="api"}
/
container_spec_memory_limit_bytes{container="api"}

For a connection pool or queue depth, expose a custom gauge from your application:

thread_pool_active_threads / thread_pool_max_threads

Alert before you hit the limit, not after:

      - alert: HighMemorySaturation
        expr: |
          container_memory_usage_bytes{container="api"}
          /
          container_spec_memory_limit_bytes{container="api"}
          > 0.85
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Container memory above 85% of limit"

Building a Dashboard

A single Grafana dashboard with four panels covers the golden signals for one service:

1. Latency panel — a heatmap or line chart showing p50, p95, and p99 over time
2. Traffic panel — a rate graph showing requests per second, optionally split by endpoint
3. Errors panel — a percentage graph with the error rate and a horizontal line at the SLO threshold
4. Saturation panel — a gauge or line chart for CPU, memory, and queue depth

Keep these four panels at the top of every service dashboard. Additional detail panels can live below, but the golden signals should be visible without scrolling.

Common Mistakes

Averaging latency. Averages hide tail latency. A service with a 50ms average might have a 2-second p99. Use percentiles.

Ignoring successful versus failed latency. If errors return in 2ms and successes take 200ms, mixing them gives a misleadingly low latency number. Filter by status code.

Alerting on raw CPU percentage. High CPU is not inherently bad. High CPU combined with rising latency is bad. Correlate saturation with user-facing signals before paging someone.

Treating all errors equally. A 404 is not the same as a 500. Separate client errors from server errors in your error rate calculation.

From Signals to Action

Golden signals are not just dashboard decorations. They feed into SLOs, error budgets, and on-call decisions. When latency breaches the SLO, that is a signal to investigate. When the error budget burns too fast, that is a signal to slow down feature releases and invest in reliability.

The four golden signals give you a shared vocabulary across teams. Instead of “the service feels slow,” you say “p95 latency crossed 400ms at 14:32 and the error rate is at 0.8 percent.” That precision turns vague complaints into actionable investigations.

Wrap-up

Latency, traffic, errors, and saturation. Four signals, four dashboard panels, four sets of alerts. They will not catch every problem, but they will catch the ones that matter to users. Start here, layer in distributed tracing and logs when you need to dig deeper, and resist the urge to monitor everything before you monitor the right things.