Monitoring with Prometheus and Grafana

Prometheus scrapes metrics from your services. Grafana visualizes them. Together they form the most popular open-source monitoring stack.

Architecture

Prometheus pulls metrics from HTTP endpoints on a schedule. Your application exposes a /metrics endpoint. Prometheus scrapes it, stores the time series, and you query it with PromQL.

App (/metrics) ← Prometheus (scrape + store) → Grafana (visualize)
                                              → Alertmanager (notify)

Running with Docker Compose

services:
  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin

Prometheus configuration

prometheus.yml:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'my-app'
    static_configs:
      - targets: ['host.docker.internal:8080']

  - job_name: 'node-exporter'
    static_configs:
      - targets: ['node-exporter:9100']

Metric types

Counter

A value that only goes up (requests, errors, bytes processed).

from prometheus_client import Counter

REQUEST_COUNT = Counter(
    'http_requests_total',
    'Total HTTP requests',
    ['method', 'endpoint', 'status']
)

@app.route('/api/users')
def get_users():
    REQUEST_COUNT.labels(method='GET', endpoint='/api/users', status='200').inc()
    return users

Gauge

A value that can go up or down (temperature, active connections, queue size).

from prometheus_client import Gauge

ACTIVE_CONNECTIONS = Gauge(
    'active_connections',
    'Number of active connections'
)

ACTIVE_CONNECTIONS.inc()   # connection opened
ACTIVE_CONNECTIONS.dec()   # connection closed
ACTIVE_CONNECTIONS.set(42) # set directly

Histogram

Observe a distribution of values (request duration, response size).

from prometheus_client import Histogram

REQUEST_DURATION = Histogram(
    'http_request_duration_seconds',
    'Request duration in seconds',
    ['endpoint'],
    buckets=[0.01, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0]
)

@app.route('/api/data')
def get_data():
    with REQUEST_DURATION.labels(endpoint='/api/data').time():
        return process_data()

PromQL basics

Instant vectors

http_requests_total
http_requests_total{method="GET"}
http_requests_total{status=~"5.."}

Rate (per-second increase)

rate(http_requests_total[5m])

Aggregation

sum(rate(http_requests_total[5m])) by (endpoint)

Percentiles from histograms

histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

Useful queries

# Error rate
sum(rate(http_requests_total{status=~"5.."}[5m]))
/ sum(rate(http_requests_total[5m]))

# Request rate per endpoint
sum(rate(http_requests_total[5m])) by (endpoint)

# Memory usage in MB
process_resident_memory_bytes / 1024 / 1024

# CPU usage
rate(process_cpu_seconds_total[5m])

Alerting rules

alert_rules.yml:

groups:
  - name: app_alerts
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m]))
          / sum(rate(http_requests_total[5m])) > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High error rate (> 5%)"

      - alert: HighLatency
        expr: |
          histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) > 1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "P95 latency > 1 second"

Grafana dashboards

1. Add Prometheus as a data source: Configuration → Data Sources → Prometheus → URL: http://prometheus:9090
2. Create a new dashboard
3. Add panels with PromQL queries

Panel examples

Request rate: sum(rate(http_requests_total[5m])) by (endpoint)

Error rate percentage: 100 * sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))

P95 latency: histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

Node exporter for system metrics

Monitor CPU, memory, disk, and network:

# docker-compose.yml
node-exporter:
  image: prom/node-exporter:latest
  ports:
    - "9100:9100"

Useful queries:

# CPU usage percentage
100 - (avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Memory usage percentage
(1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100

# Disk usage percentage
100 - (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} * 100)

Summary

Prometheus collects metrics, PromQL queries them, Grafana visualizes them, and Alertmanager notifies you. Instrument your application with counters for throughput, histograms for latency, and gauges for current state. Start with request rate, error rate, and p95 latency — the three metrics that tell you if your service is healthy.