Log Aggregation with ELK Stack: Complete Setup Guide

What Is the ELK Stack?

The ELK Stack is a collection of three open-source tools that together provide a complete log aggregation and analysis platform:

• Elasticsearch stores and indexes log data, making it searchable in near real-time.
• Logstash collects logs from various sources, transforms them, and sends them to Elasticsearch.
• Kibana provides a web interface for searching, visualizing, and building dashboards from the data in Elasticsearch.

A common addition is Filebeat, a lightweight log shipper that runs on your application servers and forwards logs to Logstash or directly to Elasticsearch. Together, this is sometimes called the Elastic Stack.

When you have dozens of services generating logs across multiple servers, you need a central place to search and correlate them. The ELK Stack solves this by pulling all logs into one searchable system.

Architecture Overview

A typical ELK deployment follows this flow:

Application Servers          Central Processing          Storage & UI
┌─────────────┐             ┌──────────┐              ┌───────────────┐
│  App Logs   │──Filebeat──▶│ Logstash │──────────────▶│ Elasticsearch │
│  Syslog     │             │ (parse,  │              └───────┬───────┘
│  Nginx Logs │             │ filter)  │                      │
└─────────────┘             └──────────┘              ┌───────▼───────┐
                                                      │    Kibana     │
                                                      │ (dashboards)  │
                                                      └───────────────┘

Filebeat is lightweight and runs on every server. Logstash handles heavy parsing and transformation. Elasticsearch indexes everything. Kibana lets you explore and visualize.

Setting Up with Docker Compose

The fastest way to get the full stack running locally is with Docker Compose. Create a docker-compose.yml:

version: "3.8"

services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.14.0
    container_name: elasticsearch
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false
      - "ES_JAVA_OPTS=-Xms1g -Xmx1g"
    ports:
      - "9200:9200"
    volumes:
      - es_data:/usr/share/elasticsearch/data
    networks:
      - elk

  logstash:
    image: docker.elastic.co/logstash/logstash:8.14.0
    container_name: logstash
    volumes:
      - ./logstash/pipeline:/usr/share/logstash/pipeline
      - ./logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml
    ports:
      - "5044:5044"
      - "5000:5000/tcp"
      - "5000:5000/udp"
    depends_on:
      - elasticsearch
    networks:
      - elk

  kibana:
    image: docker.elastic.co/kibana/kibana:8.14.0
    container_name: kibana
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
    ports:
      - "5601:5601"
    depends_on:
      - elasticsearch
    networks:
      - elk

volumes:
  es_data:
    driver: local

networks:
  elk:
    driver: bridge

Create the Logstash configuration directory and files:

mkdir -p logstash/pipeline logstash/config

Create logstash/config/logstash.yml:

http.host: "0.0.0.0"
xpack.monitoring.elasticsearch.hosts: ["http://elasticsearch:9200"]

Writing Logstash Pipelines

Logstash pipelines have three stages: input, filter, and output. Create logstash/pipeline/main.conf:

input {
  beats {
    port => 5044
  }

  tcp {
    port => 5000
    codec => json_lines
  }
}

filter {
  if [type] == "nginx" {
    grok {
      match => {
        "message" => '%{IPORHOST:client_ip} - %{DATA:user} \[%{HTTPDATE:timestamp}\] "%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:http_version}" %{NUMBER:status_code} %{NUMBER:bytes}'
      }
    }
    date {
      match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]
    }
    mutate {
      convert => {
        "status_code" => "integer"
        "bytes" => "integer"
      }
    }
    geoip {
      source => "client_ip"
    }
  }

  if [type] == "application" {
    json {
      source => "message"
    }
    date {
      match => ["timestamp", "ISO8601"]
    }
    mutate {
      remove_field => ["message"]
    }
  }
}

output {
  elasticsearch {
    hosts => ["http://elasticsearch:9200"]
    index => "logs-%{[type]}-%{+YYYY.MM.dd}"
  }
}

This pipeline accepts input from Filebeat on port 5044 and JSON over TCP on port 5000. The filter section uses grok patterns to parse unstructured nginx logs into structured fields, and parses JSON-formatted application logs directly. The output sends everything to Elasticsearch with date-based index names.

Understanding Grok Patterns

Grok is Logstash’s most powerful filter. It uses named patterns to extract structured data from unstructured text:

# Pattern format: %{PATTERN_NAME:field_name}

# Common patterns:
# %{IP:client_ip}          - matches an IP address
# %{WORD:method}           - matches a single word (GET, POST, etc.)
# %{NUMBER:status:int}     - matches a number, converts to integer
# %{GREEDYDATA:message}    - matches everything remaining

You can test grok patterns at the Kibana Dev Tools Grok Debugger before deploying them.

Setting Up Filebeat

Filebeat runs on your application servers and ships logs to Logstash. Create a filebeat.yml:

filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /var/log/nginx/access.log
    fields:
      type: nginx
    fields_under_root: true

  - type: log
    enabled: true
    paths:
      - /var/log/myapp/*.log
    fields:
      type: application
    fields_under_root: true
    multiline.pattern: '^\d{4}-\d{2}-\d{2}'
    multiline.negate: true
    multiline.match: after

output.logstash:
  hosts: ["logstash-server:5044"]

logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat.log
  keepfiles: 7

The multiline configuration handles stack traces by joining lines that do not start with a date pattern onto the previous log entry. Add Filebeat to your Docker Compose or install it directly on application servers:

# Install on Ubuntu
curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.14.0-amd64.deb
sudo dpkg -i filebeat-8.14.0-amd64.deb
sudo systemctl enable filebeat
sudo systemctl start filebeat

Launching the Stack

Start everything with Docker Compose:

docker compose up -d

Wait about 30 seconds for Elasticsearch to initialize, then verify each component:

# Check Elasticsearch
curl http://localhost:9200/_cluster/health?pretty

# Check Logstash (should show pipeline running)
curl http://localhost:9600/_node/stats/pipelines?pretty

# Kibana should be available at http://localhost:5601

Send a test log message to verify the pipeline:

echo '{"timestamp":"2026-07-07T10:00:00Z","level":"info","service":"auth","message":"User login successful","user_id":"12345"}' | nc localhost 5000

Check Elasticsearch for the indexed document:

curl "http://localhost:9200/logs-*/_search?pretty&size=1"

Building Kibana Dashboards

Once logs are flowing, open Kibana at http://localhost:5601 and set up your data views and dashboards.

Create a Data View

Navigate to Stack Management, then Data Views. Create a new data view with the pattern logs-* to match all your log indices.

Build Visualizations

Common visualizations for log dashboards:

Request rate over time: Use a Lens area chart with @timestamp on the x-axis and count on the y-axis, filtered to type: nginx.

HTTP status code distribution: Create a pie chart breaking down status_code values to quickly spot error rates.

Top error messages: Use a table visualization showing the most frequent log entries where level is error.

Response time percentiles: If your application logs include response times, create a line chart showing p50, p95, and p99 latency.

Saved Searches

Create saved searches with commonly used filters:

# Find all 5xx errors in the last hour
status_code >= 500 AND @timestamp >= now-1h

# Find slow requests
response_time > 2000 AND type: "application"

# Find errors for a specific service
level: "error" AND service: "payment-api"

Index Lifecycle Management

Logs accumulate fast. Use Elasticsearch Index Lifecycle Management (ILM) to automatically manage old data:

curl -X PUT "http://localhost:9200/_ilm/policy/logs-policy" -H 'Content-Type: application/json' -d'
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_size": "10gb",
            "max_age": "1d"
          }
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "shrink": { "number_of_shards": 1 },
          "forcemerge": { "max_num_segments": 1 }
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}'

This policy keeps logs on fast storage for 7 days, moves them to warm storage with reduced resources, and deletes them after 30 days.

Alerting on Log Patterns

Kibana includes built-in alerting. Navigate to Stack Management, then Rules and Connectors, and create rules for critical patterns:

• Error spike: Alert when the count of level: error logs exceeds a threshold within a time window.
• Service down: Alert when a service stops sending logs entirely.
• Security events: Alert on authentication failures or suspicious patterns.

Alerts can be sent to Slack, email, PagerDuty, or webhooks.

Production Considerations

Resource sizing: Elasticsearch is memory-hungry. Allocate at least 4GB of heap for a development setup and 16GB or more for production. The general rule is to give Elasticsearch half of available RAM, up to 32GB.

Cluster setup: Run at least three Elasticsearch nodes in production for high availability. Use dedicated master nodes if your cluster handles heavy indexing.

Security: Enable Elasticsearch security features (TLS, authentication) in production. The default Docker setup disables security for convenience, but you must enable it in any real deployment.

Scaling Logstash: Logstash is CPU-intensive during parsing. Run multiple Logstash instances behind a load balancer if you hit bottlenecks. Alternatively, use Elasticsearch ingest pipelines to offload some parsing.

Consider alternatives: For simpler setups, Filebeat can ship directly to Elasticsearch without Logstash, using ingest pipelines for parsing. For cloud environments, consider the Elastic Cloud managed service.

Wrapping Up

You now have a working ELK Stack that collects, parses, and visualizes logs from multiple sources. The key pieces are Filebeat for shipping, Logstash for parsing with grok patterns, Elasticsearch for storage and search, and Kibana for dashboards. From here, add more log sources, refine your grok patterns for application-specific formats, set up alerting for critical error patterns, and implement ILM policies to manage storage costs as your log volume grows.