OpenTelemetry for Distributed Tracing

Why OpenTelemetry

When a request hits your API gateway, passes through three microservices, queries two databases, and calls an external payment provider, debugging latency or errors becomes a nightmare. Logs tell you what happened in one service. Metrics tell you aggregate counts. But neither tells you the story of a single request flowing through the entire system.

Distributed tracing solves this by assigning a unique trace ID to each request and propagating it across service boundaries. Every service records spans (units of work) with timing, status, and metadata. When you assemble the spans, you get a waterfall view showing exactly where time was spent.

OpenTelemetry (OTel) is the CNCF standard for collecting traces, metrics, and logs. It provides vendor-neutral SDKs and a collector that can export to Jaeger, Zipkin, Tempo, Datadog, New Relic, or any OTLP-compatible backend. You instrument once, and switch backends without changing application code.

Core Concepts

Trace: The entire journey of a request through your system. Identified by a trace ID.

Span: A single operation within a trace. A span has a name, start time, duration, status, attributes, and optionally a parent span. A trace is a tree of spans.

Context propagation: The mechanism that passes trace context (trace ID, span ID) across process boundaries. When service A calls service B over HTTP, the trace context is serialized into request headers. Service B extracts it and creates child spans under the same trace.

Exporter: The component that sends collected telemetry data to a backend (Jaeger, OTLP endpoint, etc.).

Collector: A standalone process that receives, processes, and exports telemetry data. It decouples your applications from the backend.

Instrumenting a Node.js Service

Install the packages:

npm install @opentelemetry/sdk-node \
  @opentelemetry/auto-instrumentations-node \
  @opentelemetry/exporter-trace-otlp-http \
  @opentelemetry/exporter-metrics-otlp-http \
  @opentelemetry/resources \
  @opentelemetry/semantic-conventions

Create a tracing setup file that runs before your application code:

// tracing.ts
import { NodeSDK } from "@opentelemetry/sdk-node";
import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";
import { OTLPMetricExporter } from "@opentelemetry/exporter-metrics-otlp-http";
import { PeriodicExportingMetricReader } from "@opentelemetry/sdk-metrics";
import { Resource } from "@opentelemetry/resources";
import {
  ATTR_SERVICE_NAME,
  ATTR_SERVICE_VERSION,
  ATTR_DEPLOYMENT_ENVIRONMENT_NAME,
} from "@opentelemetry/semantic-conventions";

const sdk = new NodeSDK({
  resource: new Resource({
    [ATTR_SERVICE_NAME]: "api-service",
    [ATTR_SERVICE_VERSION]: process.env.APP_VERSION || "0.0.0",
    [ATTR_DEPLOYMENT_ENVIRONMENT_NAME]: process.env.NODE_ENV || "development",
  }),
  traceExporter: new OTLPTraceExporter({
    url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT || "http://localhost:4318/v1/traces",
  }),
  metricReader: new PeriodicExportingMetricReader({
    exporter: new OTLPMetricExporter({
      url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT || "http://localhost:4318/v1/metrics",
    }),
    exportIntervalMillis: 30000,
  }),
  instrumentations: [
    getNodeAutoInstrumentations({
      "@opentelemetry/instrumentation-http": {
        ignoreIncomingPaths: ["/healthz", "/readyz"],
      },
      "@opentelemetry/instrumentation-fs": {
        enabled: false,
      },
    }),
  ],
});

sdk.start();

process.on("SIGTERM", () => {
  sdk.shutdown().then(() => process.exit(0));
});

Load it before your app:

node --require ./tracing.js ./app.js
# or with ts-node:
node --require ./tracing.ts ./app.ts

Auto-instrumentation automatically creates spans for HTTP requests, database queries (pg, mysql, mongodb), Redis calls, gRPC, and more. You get traces without changing your application code.

Adding Custom Spans

Auto-instrumentation covers framework-level operations. For business logic, add custom spans:

import { trace, SpanStatusCode } from "@opentelemetry/api";

const tracer = trace.getTracer("api-service");

async function processPayment(orderId: string, amount: number) {
  return tracer.startActiveSpan("process-payment", async (span) => {
    try {
      span.setAttribute("order.id", orderId);
      span.setAttribute("payment.amount", amount);
      span.setAttribute("payment.currency", "USD");

      // Validate the order
      await tracer.startActiveSpan("validate-order", async (validateSpan) => {
        const order = await db.orders.findById(orderId);
        if (!order) {
          validateSpan.setStatus({
            code: SpanStatusCode.ERROR,
            message: "Order not found",
          });
          throw new Error(`Order ${orderId} not found`);
        }
        validateSpan.setAttribute("order.status", order.status);
        validateSpan.end();
      });

      // Charge the payment provider
      const result = await tracer.startActiveSpan("charge-provider", async (chargeSpan) => {
        chargeSpan.setAttribute("provider", "stripe");
        const charge = await stripe.charges.create({ amount, currency: "usd" });
        chargeSpan.setAttribute("charge.id", charge.id);
        chargeSpan.end();
        return charge;
      });

      span.setAttribute("charge.id", result.id);
      span.setStatus({ code: SpanStatusCode.OK });
      return result;
    } catch (error) {
      span.setStatus({
        code: SpanStatusCode.ERROR,
        message: error instanceof Error ? error.message : "Unknown error",
      });
      span.recordException(error as Error);
      throw error;
    } finally {
      span.end();
    }
  });
}

Each startActiveSpan creates a child span under the current active span. The result is a trace tree: HTTP POST /checkout > process-payment > validate-order + charge-provider.

The OpenTelemetry Collector

In production, applications should not export directly to the tracing backend. The OTel Collector acts as a proxy that receives telemetry, processes it, and exports it to one or more backends:

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 5s
    send_batch_size: 1024

  memory_limiter:
    check_interval: 1s
    limit_mib: 512
    spike_limit_mib: 128

  attributes:
    actions:
      - key: environment
        value: production
        action: upsert

  tail_sampling:
    decision_wait: 10s
    policies:
      - name: errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: slow-requests
        type: latency
        latency:
          threshold_ms: 2000
      - name: probabilistic
        type: probabilistic
        probabilistic:
          sampling_percentage: 10

exporters:
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true

  otlp/metrics:
    endpoint: prometheus:4317
    tls:
      insecure: true

  debug:
    verbosity: basic

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, tail_sampling, batch, attributes]
      exporters: [otlp/tempo]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp/metrics]

Tail Sampling

The tail_sampling processor is critical for production. Without sampling, high-traffic services generate enormous volumes of trace data. Tail sampling makes intelligent decisions: keep all error traces, keep all traces slower than 2 seconds, and sample 10 percent of everything else. This dramatically reduces storage costs while preserving the traces you actually need for debugging.

Deploying the Collector in Kubernetes

# collector-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-collector
spec:
  replicas: 2
  selector:
    matchLabels:
      app: otel-collector
  template:
    metadata:
      labels:
        app: otel-collector
    spec:
      containers:
        - name: collector
          image: otel/opentelemetry-collector-contrib:0.100.0
          args: ["--config=/etc/otel/config.yaml"]
          ports:
            - containerPort: 4317
            - containerPort: 4318
          resources:
            requests:
              cpu: 200m
              memory: 256Mi
            limits:
              cpu: "1"
              memory: 512Mi
          volumeMounts:
            - name: config
              mountPath: /etc/otel
      volumes:
        - name: config
          configMap:
            name: otel-collector-config
---
apiVersion: v1
kind: Service
metadata:
  name: otel-collector
spec:
  selector:
    app: otel-collector
  ports:
    - name: otlp-grpc
      port: 4317
    - name: otlp-http
      port: 4318

Applications send telemetry to http://otel-collector:4318. The collector handles batching, sampling, and forwarding.

Context Propagation Across Services

When service A calls service B, trace context must be propagated. OpenTelemetry does this automatically for instrumented HTTP clients by injecting W3C Trace Context headers:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
tracestate: vendor=value

For message queues (Kafka, RabbitMQ), you need to propagate context through message headers. Most OTel instrumentation libraries handle this, but if you use a custom transport, you must inject and extract context manually:

import { context, propagation } from "@opentelemetry/api";

// Producer: inject context into message headers
const headers: Record<string, string> = {};
propagation.inject(context.active(), headers);
await queue.publish({ data: payload, headers });

// Consumer: extract context from message headers
const extractedContext = propagation.extract(context.active(), message.headers);
context.with(extractedContext, () => {
  tracer.startActiveSpan("process-message", (span) => {
    // This span is now a child of the producer's span
    processMessage(message);
    span.end();
  });
});

What to Instrument

Start with the boundaries: incoming HTTP requests, outgoing HTTP calls, database queries, and cache operations. Auto-instrumentation covers most of these. Add custom spans for business-critical operations: payment processing, order fulfillment, authentication flows.

Do not instrument every function. Too many spans create noise and increase overhead. Focus on operations that help you answer the question: “Where did the time go, and what went wrong?”