Prometheus Alerting Rules and AlertManager

How Alerting Works

Prometheus evaluates alerting rules at a regular interval (typically every 15 to 60 seconds). When a rule’s PromQL expression returns results, the alert transitions to pending. If it stays active for the configured for duration, it becomes firing and Prometheus sends it to AlertManager. AlertManager then handles deduplication, grouping, routing, and delivering notifications to Slack, PagerDuty, email, or webhooks.

The separation matters. Prometheus decides when something is wrong. AlertManager decides who to tell and how.

Writing Alerting Rules

Alerting rules live in YAML files loaded by Prometheus:

# rules/application.yml
groups:
  - name: application
    interval: 30s
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
          /
          sum(rate(http_requests_total[5m])) by (service)
          > 0.05
        for: 5m
        labels:
          severity: critical
          team: backend
        annotations:
          summary: "High error rate on {{ $labels.service }}"
          description: >
            Service {{ $labels.service }} has a 5xx error rate of
            {{ $value | humanizePercentage }} over the last 5 minutes.
          runbook_url: "https://wiki.internal/runbooks/high-error-rate"

      - alert: HighLatencyP99
        expr: |
          histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service))
          > 2.0
        for: 10m
        labels:
          severity: warning
          team: backend
        annotations:
          summary: "P99 latency above 2s on {{ $labels.service }}"
          description: >
            The 99th percentile response time for {{ $labels.service }}
            is {{ $value | humanizeDuration }}.

The for field is crucial. A five-minute for duration means the condition must be continuously true for five minutes before the alert fires. This prevents transient spikes from waking people up at 3 AM.

Essential Alert Categories

Here are the categories every production system should cover:

Infrastructure Alerts

groups:
  - name: infrastructure
    rules:
      - alert: HighCPUUsage
        expr: |
          100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
          > 85
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "CPU usage above 85% on {{ $labels.instance }}"

      - alert: DiskSpaceRunningLow
        expr: |
          (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100
          < 15
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Disk space below 15% on {{ $labels.instance }}"

      - alert: DiskSpaceCritical
        expr: |
          (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100
          < 5
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Disk space below 5% on {{ $labels.instance }}"

      - alert: MemoryPressure
        expr: |
          (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100
          > 90
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Memory usage above 90% on {{ $labels.instance }}"

Application Alerts

      - alert: ServiceDown
        expr: up{job="api-server"} == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "{{ $labels.instance }} is unreachable"

      - alert: QueueBacklogGrowing
        expr: |
          predict_linear(job_queue_length[1h], 3600) > 10000
        for: 30m
        labels:
          severity: warning
        annotations:
          summary: "Queue backlog predicted to exceed 10k in the next hour"

The predict_linear function is powerful. Instead of alerting when a queue is already too large, it alerts when the trend suggests it will be too large soon.

Configuring AlertManager

AlertManager uses a separate configuration file:

# alertmanager.yml
global:
  resolve_timeout: 5m
  slack_api_url: "https://hooks.slack.com/services/T00/B00/xxxx"

route:
  receiver: default-slack
  group_by: ["alertname", "service"]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

  routes:
    - match:
        severity: critical
      receiver: pagerduty-critical
      group_wait: 10s
      repeat_interval: 1h

    - match:
        severity: warning
        team: backend
      receiver: backend-slack

    - match:
        severity: warning
        team: platform
      receiver: platform-slack

receivers:
  - name: default-slack
    slack_configs:
      - channel: "#alerts-general"
        title: '{{ .GroupLabels.alertname }}'
        text: >-
          {{ range .Alerts }}
          *{{ .Annotations.summary }}*
          {{ .Annotations.description }}
          {{ end }}
        send_resolved: true

  - name: pagerduty-critical
    pagerduty_configs:
      - service_key: "your-pagerduty-integration-key"
        description: '{{ .CommonAnnotations.summary }}'
        severity: critical

  - name: backend-slack
    slack_configs:
      - channel: "#alerts-backend"
        send_resolved: true

  - name: platform-slack
    slack_configs:
      - channel: "#alerts-platform"
        send_resolved: true

Routing Logic

AlertManager routes are evaluated top-down. The first matching route wins. The top-level route acts as a catch-all default.

group_by controls how alerts are batched. Grouping by alertname and service means all firing alerts with the same name and service get bundled into one notification instead of flooding the channel.

group_wait is how long AlertManager waits after the first alert in a group arrives before sending the notification. This lets it batch alerts that fire at roughly the same time.

repeat_interval is how often AlertManager resends an already-firing alert. For critical pages, one hour is reasonable. For warnings, four hours prevents fatigue.

Inhibition Rules

Inhibition lets you suppress less important alerts when a more important one is already firing:

inhibit_rules:
  - source_match:
      severity: critical
    target_match:
      severity: warning
    equal: ["alertname", "instance"]

  - source_match:
      alertname: ServiceDown
    target_match:
      alertname: HighErrorRate
    equal: ["service"]

If ServiceDown is firing for the payments service, the HighErrorRate alert for the same service gets suppressed. Obviously the error rate is high when the service is completely down. You do not need two pages for the same incident.

Silences

Silences temporarily mute alerts during known maintenance windows. You can create them through the AlertManager UI or API:

# Create a silence for a planned deployment
amtool silence add \
  --alertmanager.url=http://alertmanager:9093 \
  --author="deploy-bot" \
  --comment="Planned deployment of payments service v2.4" \
  --duration=30m \
  alertname="HighErrorRate" service="payments"

Always set a duration. Silences without expiration lead to missed alerts days later when everyone has forgotten the silence exists.

Testing Rules

Before deploying, validate your rules with promtool:

# Check rule syntax
promtool check rules rules/application.yml

# Run unit tests
promtool test rules tests/application_test.yml

A test file looks like this:

# tests/application_test.yml
rule_files:
  - ../rules/application.yml

evaluation_interval: 1m

tests:
  - interval: 1m
    input_series:
      - series: 'http_requests_total{service="api", status="500"}'
        values: "0+10x20"
      - series: 'http_requests_total{service="api", status="200"}'
        values: "0+100x20"

    alert_rule_test:
      - eval_time: 10m
        alertname: HighErrorRate
        exp_alerts:
          - exp_labels:
              service: api
              severity: critical
              team: backend

This simulates 20 minutes of data where the error rate is roughly 9 percent, and asserts that HighErrorRate fires by the 10-minute mark.

Reducing Alert Fatigue

The biggest failure mode is too many alerts. When the on-call engineer gets forty notifications per shift, they start ignoring all of them.

Practical rules to follow: only alert on symptoms (high error rate, high latency) rather than causes (CPU usage, memory usage) unless the cause directly maps to customer impact. Use warning severity for things that can wait until business hours. Reserve critical for situations that need human intervention right now. Every alert should have a runbook_url annotation that links to a document explaining what to check and what to do.

If an alert fires and the correct response is always “do nothing and wait,” delete that alert. If the correct response is always the same set of steps, automate those steps and remove the alert.