SRE Practices: SLIs, SLOs, and Error Budgets Explained

What Is Site Reliability Engineering?

Site Reliability Engineering (SRE) is a discipline that applies software engineering principles to operations problems. Originated at Google, SRE provides a framework for running reliable production systems without sacrificing development velocity.

At its core, SRE answers a fundamental question: how reliable does a service need to be? The answer is almost never “100%.” Pursuing perfect reliability is infinitely expensive and slows down feature development to a crawl. SRE provides tools to quantify the right level of reliability and make informed trade-offs.

The three key concepts are:

• SLI (Service Level Indicator): A measurement of your service’s behavior.
• SLO (Service Level Objective): A target value for an SLI.
• Error Budget: The allowed amount of unreliability, derived from your SLO.

Service Level Indicators (SLIs)

An SLI is a carefully chosen metric that quantifies some aspect of the service experience. Good SLIs measure what users actually care about, not what is convenient to measure.

The Four Golden Signals

Google’s SRE book defines four golden signals that apply to most services:

Latency: How long it takes to serve a request. Measure this as a distribution, not just an average. The p50 tells you what most users experience; the p99 reveals the worst cases.

Traffic: How much demand is being placed on the system. For a web service, this is typically requests per second. For a data pipeline, it might be records processed per second.

Errors: The rate of requests that fail. This includes explicit errors (HTTP 500s) and implicit errors (HTTP 200 with wrong content, or responses that exceed a latency threshold).

Saturation: How full your service is. This measures resource utilization approaching limits: CPU, memory, disk, connection pools.

Choosing SLIs

For a typical web API, these SLIs work well:

slis:
  availability:
    description: "Proportion of successful HTTP requests"
    formula: "count(http_status < 500) / count(all requests)"
    measurement: "Prometheus counter http_requests_total with status label"

  latency:
    description: "Proportion of requests served within threshold"
    formula: "count(response_time < 300ms) / count(all requests)"
    measurement: "Prometheus histogram http_request_duration_seconds"

  correctness:
    description: "Proportion of requests returning correct results"
    formula: "count(valid_response) / count(all requests)"
    measurement: "Application-level validation metric"

Measuring SLIs with Prometheus

Here is how to calculate SLIs using Prometheus queries:

# Availability SLI: percentage of non-5xx responses
- record: sli:availability:ratio_rate5m
  expr: |
    sum(rate(http_requests_total{status!~"5.."}[5m]))
    /
    sum(rate(http_requests_total[5m]))

# Latency SLI: percentage of requests under 300ms
- record: sli:latency:ratio_rate5m
  expr: |
    sum(rate(http_request_duration_seconds_bucket{le="0.3"}[5m]))
    /
    sum(rate(http_request_duration_seconds_count[5m]))

These recording rules compute SLIs as ratios between 0 and 1. A value of 0.997 means 99.7% of requests met the criteria.

Service Level Objectives (SLOs)

An SLO is a target value for an SLI over a defined time window. It states how reliable your service should be.

Setting SLOs

Consider these factors when setting SLOs:

User expectations: What level of reliability do your users need? An internal tool can tolerate more downtime than a payment processor.

Current performance: Look at your historical SLI data. Your initial SLO should be achievable based on current performance, then tighten it over time.

Dependencies: Your service can only be as reliable as its least reliable hard dependency. If your database has 99.9% availability, your API cannot promise 99.99%.

Cost: Higher reliability requires more investment in redundancy, testing, and operational maturity. A 99.99% SLO costs significantly more than 99.9%.

Example SLO Document

service: checkout-api
owner: payments-team
review_date: 2026-10-01

slos:
  - name: availability
    sli: "Proportion of HTTP requests with status < 500"
    target: 99.9%
    window: 30 days (rolling)
    rationale: >
      Payment processing is critical to revenue. Users expect
      checkout to work virtually every time they attempt it.

  - name: latency
    sli: "Proportion of requests completing within 500ms"
    target: 99.0%
    window: 30 days (rolling)
    rationale: >
      Checkout flow abandonment increases significantly when
      response times exceed 500ms. We allow 1% of requests to
      be slower to accommodate periodic processing spikes.

What SLOs Mean in Practice

Here is what different SLO targets mean in terms of allowed downtime over 30 days:

SLO	Allowed Downtime (30 days)	Error Budget
99%	7 hours 12 minutes	1%
99.5%	3 hours 36 minutes	0.5%
99.9%	43 minutes	0.1%
99.95%	21 minutes	0.05%
99.99%	4.3 minutes	0.01%

Notice the dramatic difference between 99.9% and 99.99%. That extra nine means ten times less allowed downtime and typically requires a significant investment in infrastructure and operational practices.

Error Budgets

The error budget is the inverse of your SLO. If your SLO is 99.9% availability, your error budget is 0.1%. Over a 30-day window, you can afford 43 minutes of downtime.

Why Error Budgets Work

Error budgets transform the relationship between development and operations from adversarial to collaborative. Without error budgets:

• Operations wants to freeze changes to protect uptime.
• Development wants to ship features as fast as possible.
• They argue endlessly about when it is “safe” to deploy.

With error budgets:

• Both teams agree on the SLO.
• As long as the error budget has room, ship features freely.
• When the error budget is exhausted, focus shifts to reliability work.
• The decision is data-driven, not political.

Calculating Error Budget Consumption

Track how much of your error budget has been consumed:

# Prometheus recording rules for error budget tracking

# Total error budget for the window (e.g., 0.001 for 99.9% SLO)
- record: error_budget:total
  expr: 1 - 0.999

# Consumed error budget over the SLO window
- record: error_budget:consumed:ratio
  expr: |
    1 - (
      sum_over_time(sli:availability:ratio_rate5m[30d])
      /
      count_over_time(sli:availability:ratio_rate5m[30d])
    )
    /
    error_budget:total

# Remaining error budget as a percentage
- record: error_budget:remaining:percent
  expr: (1 - error_budget:consumed:ratio) * 100

Error Budget Policies

Document what happens at different error budget thresholds:

error_budget_policy:
  service: checkout-api

  thresholds:
    - remaining: "> 50%"
      actions:
        - "Normal development velocity"
        - "Standard deployment cadence"
        - "Routine reliability improvements"

    - remaining: "25% - 50%"
      actions:
        - "Increase deployment testing rigor"
        - "Prioritize one reliability improvement per sprint"
        - "Review recent incidents for patterns"

    - remaining: "< 25%"
      actions:
        - "Halt non-critical feature deployments"
        - "Dedicate 50% of engineering time to reliability"
        - "Require additional review for all changes"

    - remaining: "0% (budget exhausted)"
      actions:
        - "Freeze all deployments except reliability fixes"
        - "Conduct a thorough reliability review"
        - "Leadership escalation for resource allocation"
        - "Resume normal operations when budget regenerates"

SLAs vs SLOs

People often confuse SLOs and SLAs. They serve different purposes:

SLO (Service Level Objective) is an internal target. It is what your engineering team aims for. There are no contractual consequences for missing it, but it triggers error budget policies.

SLA (Service Level Agreement) is an external contract with customers. It includes financial penalties (credits, refunds) if you fail to meet it. SLAs should always be less strict than SLOs. If your SLO is 99.9%, your SLA might be 99.5%, giving you a buffer before contractual obligations kick in.

SLA (99.5%) <-- customer contract, penalties if violated
SLO (99.9%) <-- engineering target, error budget policies
Actual (99.95%) <-- current performance

Building SLO Dashboards

Create a Grafana dashboard that shows SLO compliance and error budget status at a glance:

# Grafana dashboard panels (conceptual)

panels:
  - title: "Availability SLI (30-day rolling)"
    type: stat
    query: "avg_over_time(sli:availability:ratio_rate5m[30d]) * 100"
    thresholds:
      - value: 99.9
        color: green
      - value: 99.5
        color: yellow
      - value: 0
        color: red

  - title: "Error Budget Remaining"
    type: gauge
    query: "error_budget:remaining:percent"
    max: 100
    thresholds:
      - value: 50
        color: green
      - value: 25
        color: yellow
      - value: 0
        color: red

  - title: "Error Budget Burn Rate"
    type: timeseries
    query: "error_budget:consumed:ratio"
    description: "Shows how fast error budget is being consumed"

  - title: "SLO Compliance History"
    type: timeseries
    query: "avg_over_time(sli:availability:ratio_rate5m[30d]) * 100"
    reference_line: 99.9

Alerting on Error Budget Burn Rate

Rather than alerting on raw error rates, alert on how fast you are consuming your error budget. This approach reduces alert noise while catching real problems:

# Fast burn: consuming error budget 14x faster than sustainable
# Alerts quickly for major incidents
- alert: ErrorBudgetFastBurn
  expr: |
    (1 - sli:availability:ratio_rate1h) > (14 * (1 - 0.999))
    and
    (1 - sli:availability:ratio_rate5m) > (14 * (1 - 0.999))
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: "Error budget burning 14x faster than sustainable"

# Slow burn: consuming error budget 3x faster than sustainable
# Catches slow degradation over hours
- alert: ErrorBudgetSlowBurn
  expr: |
    (1 - sli:availability:ratio_rate6h) > (3 * (1 - 0.999))
    and
    (1 - sli:availability:ratio_rate30m) > (3 * (1 - 0.999))
  for: 15m
  labels:
    severity: warning
  annotations:
    summary: "Error budget burning 3x faster than sustainable"

This multi-window, multi-burn-rate approach (from Google’s SRE Workbook) provides fast detection for major incidents and reliable detection for slow degradation, while keeping false positive rates low.

Getting Started with SRE Practices

Adopting SRE practices does not require reorganizing your entire company. Start incrementally:

1. Pick one critical service and define SLIs based on what users care about.
2. Set an initial SLO based on current performance (e.g., p50 of last 30 days).
3. Instrument the SLI in your monitoring system and build a dashboard.
4. Track the error budget for one quarter before enforcing policies.
5. Write an error budget policy and get buy-in from engineering leadership.
6. Iterate: tighten SLOs as reliability improves, expand to more services.

Wrapping Up

SRE practices give you a quantitative framework for reliability decisions. SLIs measure what matters to users, SLOs set clear targets, and error budgets create a shared language between product and engineering teams. The power of this approach is that it replaces subjective arguments about reliability with data-driven decisions. When the error budget is healthy, ship features aggressively. When it is depleted, invest in reliability. This balance is what lets organizations move fast without breaking things permanently.