Kubernetes Logging with Fluent Bit and Elasticsearch

Every container in Kubernetes writes to stdout and stderr. By default, these logs are stored on each node’s filesystem and lost when the node is replaced. For any serious workload, you need centralized logging — collecting logs from every container, enriching them with metadata, and storing them in a searchable system.

Fluent Bit is the standard tool for this job. It is a lightweight log processor written in C that runs as a DaemonSet, reading logs from every node and forwarding them to backends like Elasticsearch, Loki, CloudWatch, or S3.

Architecture Overview

The logging pipeline has three stages:

1. Collection — Fluent Bit runs on every node as a DaemonSet, reading container log files from /var/log/containers/.
2. Processing — Fluent Bit parses, filters, and enriches logs with Kubernetes metadata (pod name, namespace, labels).
3. Storage — Logs are forwarded to Elasticsearch (or another backend) for indexing and search. Kibana provides the UI.

┌─────────────┐     ┌─────────────┐     ┌───────────────┐     ┌─────────┐
│  Container  │────▶│  Fluent Bit  │────▶│ Elasticsearch │────▶│ Kibana  │
│  (stdout)   │     │  (DaemonSet) │     │   (storage)   │     │  (UI)   │
└─────────────┘     └─────────────┘     └───────────────┘     └─────────┘

Deploying Elasticsearch and Kibana

For a quick setup, deploy Elasticsearch and Kibana in your cluster. For production, consider managed services like AWS OpenSearch or Elastic Cloud.

# elasticsearch.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: logging
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: elasticsearch
  namespace: logging
spec:
  serviceName: elasticsearch
  replicas: 1
  selector:
    matchLabels:
      app: elasticsearch
  template:
    metadata:
      labels:
        app: elasticsearch
    spec:
      containers:
        - name: elasticsearch
          image: docker.elastic.co/elasticsearch/elasticsearch:8.13.0
          env:
            - name: discovery.type
              value: single-node
            - name: xpack.security.enabled
              value: "false"
            - name: ES_JAVA_OPTS
              value: "-Xms512m -Xmx512m"
          ports:
            - containerPort: 9200
              name: http
          resources:
            requests:
              cpu: 500m
              memory: 1Gi
            limits:
              cpu: "1"
              memory: 2Gi
          volumeMounts:
            - name: data
              mountPath: /usr/share/elasticsearch/data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 20Gi
---
apiVersion: v1
kind: Service
metadata:
  name: elasticsearch
  namespace: logging
spec:
  selector:
    app: elasticsearch
  ports:
    - port: 9200
      targetPort: 9200
  clusterIP: None

# kibana.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kibana
  namespace: logging
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kibana
  template:
    metadata:
      labels:
        app: kibana
    spec:
      containers:
        - name: kibana
          image: docker.elastic.co/kibana/kibana:8.13.0
          env:
            - name: ELASTICSEARCH_HOSTS
              value: http://elasticsearch:9200
          ports:
            - containerPort: 5601
          resources:
            requests:
              cpu: 200m
              memory: 512Mi
            limits:
              cpu: 500m
              memory: 1Gi
---
apiVersion: v1
kind: Service
metadata:
  name: kibana
  namespace: logging
spec:
  selector:
    app: kibana
  ports:
    - port: 5601
      targetPort: 5601

Deploying Fluent Bit

Fluent Bit needs permissions to read pod metadata from the Kubernetes API and access log files on each node.

RBAC Setup

# fluent-bit-rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluent-bit
  namespace: logging
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fluent-bit
rules:
  - apiGroups: [""]
    resources:
      - namespaces
      - pods
      - pods/logs
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: fluent-bit
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: fluent-bit
subjects:
  - kind: ServiceAccount
    name: fluent-bit
    namespace: logging

Fluent Bit Configuration

# fluent-bit-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: logging
data:
  fluent-bit.conf: |
    [SERVICE]
        Flush         5
        Log_Level     info
        Daemon        off
        Parsers_File  parsers.conf
        HTTP_Server   On
        HTTP_Listen   0.0.0.0
        HTTP_Port     2020

    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/*.log
        Parser            cri
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Refresh_Interval  10

    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        Merge_Log           On
        Merge_Log_Key       log_processed
        K8S-Logging.Parser  On
        K8S-Logging.Exclude On
        Labels              On
        Annotations         Off

    [FILTER]
        Name    grep
        Match   kube.*
        Exclude $kubernetes['namespace_name'] kube-system

    [OUTPUT]
        Name            es
        Match           kube.*
        Host            elasticsearch.logging.svc.cluster.local
        Port            9200
        Logstash_Format On
        Logstash_Prefix kubernetes
        Retry_Limit     5
        Suppress_Type_Name On

  parsers.conf: |
    [PARSER]
        Name        cri
        Format      regex
        Regex       ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<log>.*)$
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L%z

    [PARSER]
        Name        json
        Format      json
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L%z

    [PARSER]
        Name        nginx
        Format      regex
        Regex       ^(?<remote>[^ ]*) - (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)
        Time_Key    time
        Time_Format %d/%b/%Y:%H:%M:%S %z

DaemonSet

# fluent-bit-daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: logging
  labels:
    app: fluent-bit
spec:
  selector:
    matchLabels:
      app: fluent-bit
  template:
    metadata:
      labels:
        app: fluent-bit
    spec:
      serviceAccountName: fluent-bit
      tolerations:
        - key: node-role.kubernetes.io/control-plane
          operator: Exists
          effect: NoSchedule
      containers:
        - name: fluent-bit
          image: fluent/fluent-bit:3.0
          ports:
            - containerPort: 2020
              name: metrics
          resources:
            requests:
              cpu: 50m
              memory: 64Mi
            limits:
              cpu: 200m
              memory: 256Mi
          volumeMounts:
            - name: varlog
              mountPath: /var/log
              readOnly: true
            - name: config
              mountPath: /fluent-bit/etc/
          readinessProbe:
            httpGet:
              path: /api/v1/health
              port: 2020
            initialDelaySeconds: 5
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /api/v1/health
              port: 2020
            initialDelaySeconds: 10
            periodSeconds: 30
      volumes:
        - name: varlog
          hostPath:
            path: /var/log
        - name: config
          configMap:
            name: fluent-bit-config

Deploy everything:

# kubectl apply -f fluent-bit-rbac.yaml
# kubectl apply -f fluent-bit-configmap.yaml
# kubectl apply -f fluent-bit-daemonset.yaml

# Verify pods are running on each node
# kubectl get pods -n logging -l app=fluent-bit -o wide

Understanding the Pipeline

Input: Tail

The tail input reads log files from /var/log/containers/. The DB option tracks the read position so Fluent Bit resumes correctly after restarts. Mem_Buf_Limit caps memory usage per input — if logs arrive faster than they can be sent, Fluent Bit pauses reading rather than consuming unbounded memory.

Filter: Kubernetes

The kubernetes filter enriches each log line with pod metadata:

{
  "log": "User logged in",
  "kubernetes": {
    "pod_name": "web-app-7f4d5b5b9-abc12",
    "namespace_name": "production",
    "container_name": "web-app",
    "labels": {
      "app": "web-app",
      "version": "v2"
    }
  }
}

Merge_Log On attempts to parse the log field as JSON. If your application outputs structured JSON logs, the fields are merged into the top-level record, making them searchable in Elasticsearch.

Filter: Grep

The grep filter excludes or includes logs based on patterns. The example above excludes all logs from kube-system to reduce noise:

[FILTER]
    Name    grep
    Match   kube.*
    Exclude $kubernetes['namespace_name'] kube-system

You can add more filters:

# Only include error-level logs from a specific app
[FILTER]
    Name    grep
    Match   kube.*
    Regex   $kubernetes['labels']['app'] my-critical-app

[FILTER]
    Name    grep
    Match   kube.*
    Regex   log (ERROR|FATAL|CRITICAL)

Output: Elasticsearch

Logstash_Format On creates daily indices like kubernetes-2026.07.07. This makes index lifecycle management straightforward — you can set retention policies to delete indices older than a certain age.

Parsing Application Logs

JSON Logs

If your application writes structured JSON to stdout, Fluent Bit parses it automatically with Merge_Log On:

// Application writes:
// {"level":"info","message":"User logged in","user_id":42,"duration_ms":15}

In Elasticsearch, level, message, user_id, and duration_ms become separate searchable fields.

Custom Parsers

For applications with custom log formats, define a parser and annotate the pod:

# In parsers.conf
[PARSER]
    Name        app-log
    Format      regex
    Regex       ^\[(?<level>\w+)\] (?<time>[^ ]+) (?<message>.*)$
    Time_Key    time
    Time_Format %Y-%m-%d %H:%M:%S

# In the pod spec
metadata:
  annotations:
    fluentbit.io/parser: app-log

Excluding Specific Pods

To stop collecting logs from a noisy pod:

metadata:
  annotations:
    fluentbit.io/exclude: "true"

Adding Log Enrichment

Add custom fields to all logs for easier filtering:

[FILTER]
    Name    modify
    Match   kube.*
    Add     cluster production-us-east-1
    Add     environment production

[FILTER]
    Name    modify
    Match   kube.*
    Rename  log message

Production Tuning

Buffer and Retry Settings

[SERVICE]
    Flush         5
    storage.path  /var/log/flb-storage/
    storage.sync  normal
    storage.checksum off
    storage.backlog.mem_limit 5M

Filesystem buffering (storage.path) ensures logs survive Fluent Bit restarts. Without it, logs in the memory buffer are lost.

Resource Limits

Monitor Fluent Bit memory and CPU usage:

# kubectl top pods -n logging -l app=fluent-bit

If Fluent Bit hits its memory limit, it pauses input collection. Increase Mem_Buf_Limit or add filesystem buffering.

Index Lifecycle Management

In Elasticsearch, configure ILM to manage index size and retention:

{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "10gb",
            "max_age": "1d"
          }
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Monitoring Fluent Bit

Fluent Bit exposes Prometheus metrics on port 2020:

# kubectl port-forward -n logging ds/fluent-bit 2020:2020
# curl http://localhost:2020/api/v1/metrics/prometheus

Key metrics to watch:

• fluentbit_input_records_total — records read per input.
• fluentbit_output_retries_total — retries indicate downstream issues.
• fluentbit_output_errors_total — errors mean logs are being dropped.

Multi-Output Setup

Send logs to multiple destinations:

[OUTPUT]
    Name            es
    Match           kube.*
    Host            elasticsearch.logging.svc.cluster.local
    Port            9200
    Logstash_Format On
    Logstash_Prefix kubernetes

[OUTPUT]
    Name            s3
    Match           kube.*
    region          us-east-1
    bucket          my-log-archive
    total_file_size 100M
    upload_timeout  10m
    s3_key_format   /logs/%Y/%m/%d/$TAG/%H_%M_%S.gz
    compression     gzip

This sends logs to both Elasticsearch (for search) and S3 (for long-term archive).

Wrapping Up

Centralized logging with Fluent Bit and Elasticsearch gives you visibility into everything running in your cluster. The setup process is:

1. Deploy Elasticsearch and Kibana (or use a managed service).
2. Create RBAC resources for Fluent Bit.
3. Configure the pipeline: inputs read from /var/log/containers/, filters enrich with Kubernetes metadata, outputs send to Elasticsearch.
4. Deploy Fluent Bit as a DaemonSet so it runs on every node.

For production readiness:

• Enable filesystem buffering to survive restarts.
• Set memory limits to prevent runaway resource usage.
• Configure index lifecycle management for retention.
• Monitor Fluent Bit metrics to catch pipeline issues early.
• Use structured JSON logging in your applications for the best search experience.