System Design: Design a Metrics Collection Pipeline

Metrics are the vital signs of a distributed system. CPU usage, request latency, error rates, queue depths — these numbers tell you whether your system is healthy, degrading, or on fire. A metrics collection pipeline ingests these numbers from thousands of sources, stores them as time series, and makes them queryable for dashboards and alerts.

Products like Prometheus, Datadog, and InfluxDB each take a different approach to this problem. Designing one from scratch reveals the core trade-offs around collection models, storage engines, and query patterns.

Functional Requirements

• Collect numeric metrics from application instances, infrastructure, and cloud services.
• Store metrics as time series (metric name, tags, timestamp, value).
• Support queries by metric name, tag filters, and time range.
• Provide aggregation functions (sum, avg, max, min, percentiles) across time and tag dimensions.
• Support alerting rules that trigger when metrics cross thresholds.
• Downsample old metrics to reduce storage costs.

Non-Functional Requirements

• Write throughput: ingest millions of data points per second.
• Query latency: dashboard queries should return within 2 seconds.
• Retention: 15 days at full resolution, 1 year downsampled.
• Availability: metric ingestion must not block on query load.

Pull vs. Push Collection

The first design decision is how metrics get from the source to the pipeline.

Pull model (Prometheus style). A central collector scrapes an HTTP endpoint (/metrics) on each target at a fixed interval (every 15 seconds). The collector maintains a list of targets via service discovery (Kubernetes API, Consul, DNS).

Advantages: the collector controls the scrape rate, targets are stateless (no queuing logic), and a missing scrape immediately signals a target is down.

Disadvantages: does not work well for short-lived jobs (they may terminate before being scraped), and pull does not scale to millions of targets from a single collector without hierarchical federation.

Push model (Datadog, StatsD style). Each application pushes metrics to a collector endpoint at its own pace. A lightweight agent on each host batches and forwards metrics.

Advantages: works for short-lived processes, and the collection infrastructure scales more naturally because targets push to a pool of collectors behind a load balancer.

Disadvantages: requires backpressure handling on the collector side, and determining whether a target is down (as opposed to simply not producing metrics) is harder.

Hybrid approach. Use pull for long-lived services with stable endpoints and push for serverless functions, batch jobs, and ephemeral containers. Most production systems end up here.

Ingestion Pipeline

Regardless of collection model, metrics flow through an ingestion pipeline:

1. Reception: collectors receive data points (metric name, tags, timestamp, value).
2. Validation: reject malformed data points (missing timestamp, invalid metric name, cardinality bombs).
3. Pre-aggregation: optionally aggregate data points before storage. For example, if 50 instances report http_requests_total, pre-aggregate by summing them into a single time series per service rather than storing 50 separate series.
4. Routing: send data points to the correct storage shard based on the metric name hash.
5. Write: batch data points and write them to the time-series store.

Cardinality control is critical. A metric with high-cardinality tags (like user_id or request_id) creates millions of unique time series, overwhelming the storage engine. Enforce limits: reject metrics with more than N unique tag combinations, or drop high-cardinality tags at ingestion.

Time-Series Storage Engine

Time-series data has unique access patterns: writes are append-only (new data points always have increasing timestamps), reads are range-based (show me CPU usage for the last hour), and old data is accessed infrequently.

Storage structure. Group data points by time series (unique combination of metric name and tags). Within each series, store data points sorted by timestamp. Use a columnar layout: separate columns for timestamps and values, enabling better compression.

Compression. Time-series data compresses extremely well:

• Timestamps: use delta-of-delta encoding. If scrapes happen every 15 seconds, most deltas are identical, so the delta-of-delta is zero. Store these in 1-2 bits.
• Values: use XOR encoding (Gorilla compression). Consecutive values of the same metric are often similar, so their XOR has many leading and trailing zeros. Store only the significant bits.

With these techniques, each data point compresses to roughly 1.4 bytes on average (compared to 16 bytes raw).

Block-based layout. Organize data into time-based blocks (for example, 2-hour blocks). A block contains all data points for all series in that time range. During the current block, write to an in-memory buffer (write-ahead log for durability). When the block window closes, flush to disk as an immutable, compressed file.

Immutable blocks simplify the system: no in-place updates, no fragmentation, straightforward backup and replication.

Indexing

To query “show me http_request_duration for service=checkout in region=us-east,” the engine needs an index from labels to time series IDs.

Use an inverted index on labels. For each label pair (key=value), maintain a sorted list of series IDs (a posting list). To find series matching multiple labels, intersect the posting lists. This is the same technique search engines use.

Store the inverted index in memory for the hot time range. For older blocks, store the index alongside the block on disk and load it on demand.

Query Engine

The query language lets users explore metrics. A query like:

avg(rate(http_request_duration_seconds[5m])) by (service)

means: compute the per-second rate of http_request_duration_seconds over a 5-minute window, then average across instances grouped by the service label.

Query execution steps:

1. Series selection: use the inverted index to find all series matching the metric name and label filters.
2. Data fetch: read the raw data points for the selected series within the requested time range.
3. Range functions: apply functions like rate() (compute per-second increase), increase(), or delta() over sliding windows.
4. Aggregation: aggregate across series using sum, avg, max, min, or quantile, grouped by specified labels.
5. Return: format the result as a list of time series or a single scalar.

For dashboard queries, cache recent query results with short TTLs (30 seconds). Dashboards auto-refresh, and most queries are identical across refreshes.

Downsampling

Full-resolution data (15-second intervals) is expensive to store for long periods. Downsample older data to reduce costs:

• 0-15 days: full resolution (15-second intervals).
• 15-90 days: 5-minute resolution. For each 5-minute window, store the min, max, sum, and count. This allows reconstructing averages and detecting spikes.
• 90-365 days: 1-hour resolution.

Run downsampling as a background process that reads full-resolution blocks, computes aggregates, and writes downsampled blocks. Delete the original full-resolution blocks after the retention window.

When a query spans multiple resolution tiers, the query engine transparently stitches them together, using the highest available resolution for each time range.

Alerting

Alerting evaluates rules against live metrics and notifies humans when thresholds are breached.

An alert rule consists of:

• A metric query (for example, rate(http_errors_total[5m]) > 10).
• A duration (the condition must hold for 5 minutes before firing, to avoid transient spikes).
• A severity and notification channel.

Evaluation. A dedicated alerting component evaluates all rules on a fixed interval (every 30 seconds). It queries the storage engine and compares results against thresholds.

Alert states. An alert transitions through states: inactive -> pending (condition met but duration not yet elapsed) -> firing (notify). When the condition clears, the alert transitions to resolved and sends a recovery notification.

Deduplication and grouping. Group related alerts (all services in the same region failing) into a single notification. Deduplicate so that an ongoing incident does not generate repeated alerts.

Scaling

• Sharding writes: partition time series across storage nodes by hashing the series ID. Each node handles ingestion and queries for its partition.
• Replication: replicate each partition to at least two nodes for durability. Use asynchronous replication to avoid impacting write latency.
• Query federation: for cross-shard queries, a query coordinator fans out to all relevant shards, merges results, and returns them. Keep the fan-out bounded by using label-based routing.
• Dedicated roles: separate ingestion nodes from query nodes. Ingestion is write-heavy and sequential; querying is read-heavy and random. Different hardware profiles suit each.

Summary

A metrics pipeline collects numeric data points from distributed sources, stores them as compressed time series, and makes them queryable for dashboards and alerts. The key design decisions are pull vs. push collection, the time-series compression scheme (delta-of-delta and XOR encoding cut storage by 10x), how you index labels for fast series selection, and how you downsample for cost-effective long-term retention.