Docker Logging Drivers and Container Monitoring

By default, Docker captures everything a container writes to stdout and stderr and stores it as JSON files on the host. This works fine for a single container on a developer laptop. In production, with dozens or hundreds of containers writing logs continuously, those JSON files fill disks, lack structure, and offer no search capability. Logging drivers and monitoring tools solve this.

How Docker Logging Works

When a container process writes to stdout or stderr, the Docker daemon intercepts the stream and routes it through a logging driver. The driver determines where logs are stored and in what format. You can query logs with docker logs only when the driver supports it (json-file and journald do; most remote drivers do not).

# View logs from a running container
docker logs myapp

# Follow logs in real time
docker logs -f --tail 100 myapp

# Show timestamps
docker logs -t myapp

Available Logging Drivers

Docker ships with several logging drivers. Configure the default in /etc/docker/daemon.json or override per container.

Driver	Destination	docker logs support
json-file	Local JSON files (default)	Yes
local	Optimized local files	Yes
journald	systemd journal	Yes
syslog	Syslog server	No
fluentd	Fluentd collector	No
awslogs	AWS CloudWatch	No
gcplogs	Google Cloud Logging	No
splunk	Splunk HTTP Event Collector	No

The local driver is a better default than json-file for most setups. It uses a compressed binary format that takes less disk space and supports automatic rotation out of the box.

Configuring the Default Driver

Edit /etc/docker/daemon.json to change the default for all containers:

{
  "log-driver": "local",
  "log-opts": {
    "max-size": "50m",
    "max-file": "5"
  }
}

Restart the Docker daemon after changing this file. Existing containers keep their original driver; only new containers use the new default.

Per-Container Driver Configuration

Override the default when running a specific container:

docker run -d \
  --log-driver syslog \
  --log-opt syslog-address=udp://logserver:514 \
  --log-opt tag="myapp" \
  myapp:latest

In Docker Compose:

services:
  app:
    image: myapp:latest
    logging:
      driver: json-file
      options:
        max-size: "25m"
        max-file: "3"
        tag: "{{.Name}}"

Preventing Disk Exhaustion

The most common production incident caused by Docker logging is disk exhaustion. The json-file driver has no rotation by default. A chatty container can fill a disk in hours.

Always set max-size and max-file:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "50m",
    "max-file": "5"
  }
}

This caps each container at 250 MB of logs (5 files of 50 MB each). The driver rotates files automatically, deleting the oldest when the limit is reached.

Shipping Logs to a Centralized System

For production workloads, you typically want logs in a searchable system like the Elastic Stack, Loki, or a cloud provider’s logging service. There are two approaches.

Approach 1: Use a remote logging driver. Configure each container to send logs directly to the destination. This is simple but means docker logs stops working, which complicates debugging.

Approach 2: Use the local driver and ship with a sidecar. Keep the local or json-file driver, and run a log shipper (Promtail, Filebeat, Fluentd) that tails the log files and forwards them. This preserves docker logs access while still centralizing.

Here is a Compose example using Promtail to ship logs to Grafana Loki:

services:
  app:
    image: myapp:latest
    logging:
      driver: json-file
      options:
        max-size: "25m"
        max-file: "3"

  promtail:
    image: grafana/promtail:latest
    volumes:
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - ./promtail-config.yml:/etc/promtail/config.yml
    command: -config.file=/etc/promtail/config.yml

Container Resource Monitoring with cAdvisor

Logging covers application output. Monitoring covers resource consumption: CPU, memory, network, and disk I/O. Google’s cAdvisor exposes per-container metrics in Prometheus format.

services:
  cadvisor:
    image: gcr.io/cadvisor/cadvisor:latest
    ports:
      - "8080:8080"
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
    restart: unless-stopped

Once running, cAdvisor exposes metrics at http://localhost:8080/metrics. Key metrics include:

• container_cpu_usage_seconds_total - cumulative CPU time consumed
• container_memory_usage_bytes - current memory usage
• container_network_receive_bytes_total - network ingress
• container_fs_usage_bytes - filesystem usage

Prometheus and Grafana Stack

A complete monitoring stack combines cAdvisor, Prometheus, and Grafana:

services:
  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus

  cadvisor:
    image: gcr.io/cadvisor/cadvisor:latest
    ports:
      - "8080:8080"
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    volumes:
      - grafana_data:/var/lib/grafana

volumes:
  prometheus_data:
  grafana_data:

The Prometheus configuration to scrape cAdvisor:

# prometheus.yml
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: cadvisor
    static_configs:
      - targets: ["cadvisor:8080"]

Docker’s Built-in Stats

For quick checks without additional tooling, Docker provides real-time stats:

# Live resource usage for all containers
docker stats

# Specific containers with custom format
docker stats --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}" app db

This is useful for ad hoc debugging but not suitable for alerting or historical analysis.

Practical Recommendations

Set log-driver: local with rotation limits as your daemon default. This gives you docker logs access and prevents disk issues. For centralized logging, run a sidecar shipper rather than using remote log drivers directly. For monitoring, deploy cAdvisor with Prometheus even in small setups. The overhead is minimal, and having historical resource data is invaluable when diagnosing performance issues after the fact.