Observability: Logs, Metrics, and Traces Explained

Observability vs Monitoring

Monitoring and observability are related but distinct concepts. Monitoring tells you when something is wrong through predefined checks and thresholds. You decide in advance what to watch for, and you get alerted when those conditions occur.

Observability is the ability to understand the internal state of a system by examining its outputs. An observable system lets you ask arbitrary questions you did not anticipate in advance. When monitoring tells you “the error rate is high,” observability lets you drill into which users are affected, which endpoints are failing, what changed, and why.

Observability is built on three pillars: logs, metrics, and traces. Each pillar provides a different lens into system behavior, and the real power comes from correlating them together.

Pillar 1: Logs

Logs are timestamped records of discrete events. They are the most familiar form of observability data because every application produces them.

Structured vs Unstructured Logs

Unstructured logs are human-readable but hard to query:

[2026-07-07 14:23:01] ERROR: Failed to process payment for user 12345. Connection timeout to payment gateway.

Structured logs use a consistent format (typically JSON) that machines can parse and index:

{
  "timestamp": "2026-07-07T14:23:01.342Z",
  "level": "error",
  "service": "checkout-api",
  "message": "Failed to process payment",
  "user_id": "12345",
  "payment_id": "pay_abc123",
  "error": "connection_timeout",
  "gateway": "stripe",
  "duration_ms": 30000,
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736"
}

Structured logs are dramatically more useful. You can filter by any field, aggregate across services, and correlate with traces using the trace_id.

Logging Best Practices

Log at the right level. Use consistent severity levels:

log_levels:
  DEBUG: "Detailed diagnostic information for development"
  INFO: "Routine operational events (request processed, job completed)"
  WARN: "Unexpected but handled situations (retry succeeded, fallback used)"
  ERROR: "Failures that affect the current operation but not the service"
  FATAL: "Failures that crash or stop the service"

Include context. Every log entry should answer who, what, when, and where:

{
  "timestamp": "2026-07-07T14:23:01Z",
  "level": "info",
  "service": "order-api",
  "instance": "order-api-7b4d6f-x2k9p",
  "request_id": "req_789",
  "trace_id": "4bf92f3577b34da6",
  "user_id": "12345",
  "action": "order_created",
  "order_id": "ord_456",
  "total": 89.99,
  "items_count": 3
}

Do not log sensitive data. Never log passwords, API keys, credit card numbers, or personal health information. Use redaction or tokenization for any data that must be referenced.

Log Aggregation Stack

A common open-source stack for log aggregation:

# Minimal Promtail + Loki + Grafana setup
# docker-compose.yml

version: "3.8"
services:
  loki:
    image: grafana/loki:2.9.0
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml
    volumes:
      - loki_data:/loki

  promtail:
    image: grafana/promtail:2.9.0
    volumes:
      - /var/log:/var/log
      - ./promtail-config.yaml:/etc/promtail/config.yml
    command: -config.file=/etc/promtail/config.yml

  grafana:
    image: grafana/grafana:10.2.0
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin

volumes:
  loki_data:

Promtail config to ship logs to Loki:

# promtail-config.yaml
server:
  http_listen_port: 9080

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: application
    static_configs:
      - targets:
          - localhost
        labels:
          job: app-logs
          __path__: /var/log/myapp/*.log
    pipeline_stages:
      - json:
          expressions:
            level: level
            service: service
            trace_id: trace_id
      - labels:
          level:
          service:
          trace_id:

Pillar 2: Metrics

Metrics are numerical measurements collected at regular intervals. Unlike logs, which record individual events, metrics aggregate data over time, making them efficient for dashboards and alerting.

Metric Types

Counter: A value that only increases. Use it for things you count: requests served, errors encountered, bytes transferred.

# Prometheus metric
http_requests_total{method="GET", path="/api/orders", status="200"} 145892

Gauge: A value that goes up and down. Use it for current state: temperature, queue depth, active connections.

active_connections{service="api"} 342
memory_usage_bytes{instance="api-1"} 1073741824

Histogram: Records the distribution of values in configurable buckets. Use it for latency, request sizes, or anything where percentiles matter.

http_request_duration_seconds_bucket{le="0.1"} 24054
http_request_duration_seconds_bucket{le="0.25"} 33444
http_request_duration_seconds_bucket{le="0.5"} 34500
http_request_duration_seconds_bucket{le="1"} 34782
http_request_duration_seconds_bucket{le="+Inf"} 34900
http_request_duration_seconds_count 34900
http_request_duration_seconds_sum 6721.5

Summary: Similar to histogram but calculates percentiles on the client side. Useful when you need specific quantiles without pre-configured buckets.

Instrumenting an Application

Here is how to expose Prometheus metrics from a Python Flask application:

from prometheus_client import Counter, Histogram, generate_latest
from flask import Flask, request, Response
import time

app = Flask(__name__)

REQUEST_COUNT = Counter(
    'http_requests_total',
    'Total HTTP requests',
    ['method', 'path', 'status']
)

REQUEST_LATENCY = Histogram(
    'http_request_duration_seconds',
    'HTTP request latency',
    ['method', 'path'],
    buckets=[0.01, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0]
)

@app.before_request
def start_timer():
    request.start_time = time.time()

@app.after_request
def record_metrics(response):
    latency = time.time() - request.start_time
    REQUEST_COUNT.labels(
        method=request.method,
        path=request.path,
        status=response.status_code
    ).inc()
    REQUEST_LATENCY.labels(
        method=request.method,
        path=request.path
    ).observe(latency)
    return response

@app.route('/metrics')
def metrics():
    return Response(generate_latest(), mimetype='text/plain')

Prometheus Scrape Configuration

# prometheus.yml
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'api'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)

Key Metrics to Track

Follow the RED method for request-driven services:

• Rate: Requests per second
• Errors: Failed requests per second
• Duration: Distribution of request latencies

And the USE method for resources:

• Utilization: Percentage of resource capacity in use
• Saturation: Amount of queued work
• Errors: Count of error events

Pillar 3: Traces

Distributed traces follow a single request as it flows through multiple services. They show you the entire journey of a request, including which services were called, how long each took, and where failures occurred.

Trace Structure

A trace consists of spans. Each span represents a unit of work:

Trace ID: 4bf92f3577b34da6a3ce929d0e0e4736

├── [api-gateway] GET /api/orders (250ms)
│   ├── [auth-service] validate_token (15ms)
│   ├── [order-service] get_orders (200ms)
│   │   ├── [database] SELECT orders (45ms)
│   │   ├── [cache] GET user_preferences (3ms)
│   │   └── [inventory-service] check_stock (120ms)
│   │       └── [database] SELECT inventory (80ms)
│   └── [api-gateway] serialize_response (12ms)

This trace shows that the request took 250ms total, and the bottleneck is the inventory service’s database query at 80ms.

OpenTelemetry Instrumentation

OpenTelemetry is the standard for collecting traces (and metrics and logs). Here is how to instrument a Node.js application:

npm install @opentelemetry/api @opentelemetry/sdk-node \
  @opentelemetry/auto-instrumentations-node \
  @opentelemetry/exporter-trace-otlp-http

// tracing.js - load this before your application
const { NodeSDK } = require('@opentelemetry/sdk-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');

const sdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter({
    url: 'http://jaeger:4318/v1/traces',
  }),
  instrumentations: [
    getNodeAutoInstrumentations({
      '@opentelemetry/instrumentation-http': {
        ignoreIncomingPaths: ['/health', '/metrics'],
      },
    }),
  ],
  serviceName: 'order-service',
});

sdk.start();

Run your application with tracing enabled:

node --require ./tracing.js app.js

Adding Custom Spans

Auto-instrumentation covers HTTP, database, and gRPC calls. For custom business logic, add manual spans:

const { trace } = require('@opentelemetry/api');

async function processOrder(order) {
  const tracer = trace.getTracer('order-service');

  return tracer.startActiveSpan('process_order', async (span) => {
    span.setAttribute('order.id', order.id);
    span.setAttribute('order.total', order.total);
    span.setAttribute('order.items_count', order.items.length);

    try {
      await validateInventory(order);
      await chargePayment(order);
      await sendConfirmation(order);
      span.setStatus({ code: trace.SpanStatusCode.OK });
    } catch (error) {
      span.setStatus({
        code: trace.SpanStatusCode.ERROR,
        message: error.message,
      });
      span.recordException(error);
      throw error;
    } finally {
      span.end();
    }
  });
}

Trace Visualization with Jaeger

Deploy Jaeger for trace collection and visualization:

# Add to docker-compose.yml
jaeger:
  image: jaegertracing/all-in-one:1.53
  ports:
    - "16686:16686"  # Jaeger UI
    - "4318:4318"    # OTLP HTTP
    - "4317:4317"    # OTLP gRPC
  environment:
    - COLLECTOR_OTLP_ENABLED=true

Access the Jaeger UI at http://localhost:16686 to search traces by service, operation, duration, and tags.

Correlating the Three Pillars

The real power of observability comes from connecting logs, metrics, and traces:

1. A metric alert fires: “p99 latency for the order service exceeded 2 seconds.”
2. You open the dashboard and see the latency spike started at 14:15 UTC.
3. You search traces for slow requests to the order service during that window.
4. A trace shows the inventory-service database query took 1.8 seconds instead of the usual 80ms.
5. You search logs for the trace ID and find: “Slow query detected: sequential scan on inventory table, 1.2M rows examined.”

The key to correlation is shared identifiers. Include trace_id in your logs. Use exemplars in Prometheus to link metrics to specific traces. Use consistent labels across all three pillars.

{
  "timestamp": "2026-07-07T14:15:23Z",
  "level": "warn",
  "service": "inventory-service",
  "trace_id": "4bf92f3577b34da6",
  "span_id": "00f067aa0ba902b7",
  "message": "Slow database query",
  "query_duration_ms": 1823,
  "table": "inventory",
  "rows_examined": 1200000
}

Wrapping Up

The three pillars of observability each serve a distinct purpose: metrics tell you something is wrong, traces show you where in the request path it went wrong, and logs tell you why. Implementing all three with correlation between them gives you the ability to investigate any production issue systematically. Start by adding structured logging with trace IDs, instrument key services with OpenTelemetry for distributed tracing, and use Prometheus for metrics collection. As your observability practice matures, focus on reducing the time from “something is wrong” to “here is what happened and why” by tightening the links between all three pillars.