Building Grafana Dashboards for Monitoring

Dashboard Philosophy

A dashboard should answer a specific question. “Is the checkout flow healthy?” is a good question. “Show me everything” is not. The most common mistake is cramming thirty panels into one dashboard and ending up with a wall of graphs that nobody reads.

Good dashboards follow the USE method (Utilization, Saturation, Errors) for infrastructure or the RED method (Rate, Errors, Duration) for services. Pick one and build around it.

Data Sources

Before building panels, connect Grafana to your data. Go to Configuration, then Data Sources, then Add data source. For Prometheus, you typically just need the URL:

URL: http://prometheus:9090
Access: Server (default)

Grafana supports dozens of data sources: Prometheus, Loki (logs), Tempo (traces), InfluxDB, Elasticsearch, PostgreSQL, CloudWatch, and more. You can query multiple data sources in a single dashboard.

Panel Types and When to Use Them

Time series is the workhorse. Use it for any metric that changes over time: request rates, error rates, latency percentiles, CPU usage.

Stat panels show a single current value with optional sparkline. Good for “current error rate” or “requests per second right now.”

Gauge panels show a value relative to a threshold. Use them for percentage-based metrics like disk usage or memory usage where you care about how close you are to a limit.

Bar gauge works well for comparing values across instances: memory usage per pod, request count per endpoint.

Table panels are useful for listing the top N consumers, current alert states, or any data that is naturally tabular.

Heatmap panels visualize distribution over time. They are excellent for latency histograms where you want to see not just the average but the shape of the distribution shifting.

Logs panels (with Loki) show log lines correlated with the time range of your graphs. When you see a spike in errors, you can scroll down to the logs panel and see what happened.

Building a Service Dashboard

Here is a RED-method dashboard for an API service, query by query.

Request Rate

sum(rate(http_requests_total{service="api"}[5m])) by (method, status_code)

Use a time series panel. Set the legend to {{ method }} {{ status_code }}. This shows you the traffic shape and lets you spot sudden drops or spikes.

Error Rate

sum(rate(http_requests_total{service="api", status_code=~"5.."}[5m]))
/
sum(rate(http_requests_total{service="api"}[5m]))

Use a stat panel with thresholds. Set the unit to “Percent (0.0-1.0)”. Add thresholds: green below 0.01, yellow below 0.05, red above 0.05. This gives you a single number that turns red when errors spike.

Latency (Duration)

histogram_quantile(0.50, sum(rate(http_request_duration_seconds_bucket{service="api"}[5m])) by (le))

Create three queries in the same panel for p50, p95, and p99. Change 0.50 to 0.95 and 0.99 for the other two. Set the legend for each query to p50, p95, p99. Set the unit to seconds.

Saturation

# Active connections or in-flight requests
sum(http_in_flight_requests{service="api"}) by (instance)

This tells you how loaded each instance is. If in-flight requests approach your concurrency limit, you are about to start rejecting traffic.

Template Variables

Variables make dashboards reusable across services and environments. Define them in Dashboard Settings, then Variables:

Name: service
Type: Query
Data source: Prometheus
Query: label_values(http_requests_total, service)

Name: environment
Type: Query
Query: label_values(http_requests_total{service="$service"}, environment)

Now update your queries to use $service and $environment:

sum(rate(http_requests_total{service="$service", environment="$environment"}[5m])) by (method)

Dropdowns appear at the top of the dashboard. Select a different service and every panel updates. This turns one dashboard into a dashboard for every service.

Annotations

Annotations mark events on your graphs. Deploy markers are the most valuable: when you see a latency spike, you can immediately check if it correlates with a deployment.

You can add annotations from the Grafana API:

curl -X POST http://grafana:3000/api/annotations \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $GRAFANA_API_KEY" \
  -d '{
    "text": "Deployed api v2.4.1",
    "tags": ["deploy", "api"],
    "time": 1719878400000
  }'

Add this to your CI/CD pipeline so every deployment is automatically annotated.

Dashboard Layout Principles

Put the most important information at the top. A row of stat panels showing current error rate, p99 latency, and request rate gives an instant health check. Detailed time series graphs go below.

Group related panels into rows. Use collapsible rows for deep-dive sections that you only need during incidents (per-instance breakdowns, database query metrics, cache hit rates).

Use consistent time ranges. If your rate queries use [5m] windows, your dashboard’s default time range should be at least 30 minutes so you see meaningful trends.

Set sensible Y-axis limits. A latency graph with auto-scaling that jumps between 0-50ms and 0-5s depending on the time range makes it hard to develop intuition for what “normal” looks like.

Dashboard as Code

Clicking through the Grafana UI is fine for prototyping, but production dashboards should be version-controlled.

JSON Export

Every Grafana dashboard has a JSON model. Go to Dashboard Settings, then JSON Model, and copy the content. Store it in your Git repository:

monitoring/
  dashboards/
    api-service.json
    infrastructure.json
    database.json
  provisioning/
    dashboards.yml
    datasources.yml

Provisioning

Configure Grafana to load dashboards from disk on startup:

# provisioning/dashboards.yml
apiVersion: 1
providers:
  - name: "default"
    orgId: 1
    folder: "Production"
    type: file
    disableDeletion: false
    updateIntervalSeconds: 30
    options:
      path: /var/lib/grafana/dashboards
      foldersFromFilesStructure: true

# provisioning/datasources.yml
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: false

Mount your dashboard JSON files into /var/lib/grafana/dashboards via a ConfigMap (Kubernetes) or a Docker volume, and Grafana loads them automatically.

Grafonnet

For teams that manage many dashboards, Grafonnet is a Jsonnet library that generates Grafana dashboard JSON programmatically:

local grafana = import 'grafonnet/grafana.libsonnet';
local dashboard = grafana.dashboard;
local prometheus = grafana.prometheus;
local graphPanel = grafana.graphPanel;

dashboard.new('API Service', tags=['api', 'production'])
.addPanel(
  graphPanel.new('Request Rate', datasource='Prometheus')
  .addTarget(
    prometheus.target(
      'sum(rate(http_requests_total{service="$service"}[5m])) by (method)',
      legendFormat='{{ method }}'
    )
  ),
  gridPos={ h: 8, w: 12, x: 0, y: 0 }
)

This approach prevents dashboard sprawl. You define panel templates once and generate consistent dashboards for every service.

Common Mistakes

Too many panels. If you need to scroll, you have too many. Split into multiple dashboards linked via drill-down.

No variable filters. Hardcoding service names means duplicating dashboards for every service. Use variables.

Missing units. A graph showing “2.5” means nothing. Is that seconds, milliseconds, requests per second, or percent? Always set the unit.

Ignoring the default time range. If the default is “last 6 hours” but your rate windows are 1 minute, the dashboard loads slowly and the graphs are noisy. Match the default range to your use case.

Start with one RED dashboard for your most critical service. Get it right, templatize it with variables, export the JSON, and you have a pattern you can replicate across your entire stack.