Chaos Engineering: Test Your System's Resilience

What Is Chaos Engineering?

Chaos engineering is the discipline of experimenting on a system to build confidence in its ability to withstand turbulent conditions in production. Rather than waiting for failures to happen at the worst possible time, you deliberately introduce failures in a controlled way to discover weaknesses before they become outages.

The idea is simple: if your system is supposed to handle a database failover gracefully, prove it by actually failing over the database. If your service is supposed to tolerate high latency from a dependency, inject latency and watch what happens.

Netflix pioneered the practice with Chaos Monkey, which randomly terminates production instances. The reasoning: if you know instances will be killed randomly, you design your services to handle it. The result is a system that gracefully handles the unexpected rather than collapsing when a single component fails.

The Chaos Engineering Process

Chaos engineering follows a scientific method:

Step 1: Define Steady State

Before breaking anything, define what “normal” looks like. Steady state is measured through business-level metrics, not just technical ones:

• Request success rate (e.g., 99.9%)
• p99 latency (e.g., under 300ms)
• Orders processed per minute
• Active user sessions

steady_state:
  metrics:
    - name: request_success_rate
      query: "sum(rate(http_requests_total{status!~'5..'}[5m])) / sum(rate(http_requests_total[5m]))"
      expected: ">= 0.999"
    - name: p99_latency
      query: "histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))"
      expected: "<= 0.3"
    - name: orders_per_minute
      query: "sum(rate(orders_completed_total[1m])) * 60"
      expected: ">= 50"

Step 2: Form a Hypothesis

State what you expect to happen when you introduce a failure:

“When we terminate one of three API pods, the remaining pods will absorb the traffic. The request success rate will stay above 99.9% and p99 latency will stay below 500ms. Kubernetes will reschedule the terminated pod within 60 seconds.”

Step 3: Run the Experiment

Introduce the failure in a controlled way. Start small: one pod, one availability zone, one network partition. Monitor your steady-state metrics throughout.

Step 4: Analyze Results

Did the system behave as expected? If yes, you have increased confidence in that failure mode. If not, you have found a weakness to fix before it causes a real outage.

Step 5: Fix and Repeat

Address any weaknesses discovered, then run the experiment again to verify the fix. Gradually increase the scope and severity of experiments.

Types of Chaos Experiments

Infrastructure Failures

experiments:
  pod_termination:
    description: "Kill random application pods"
    blast_radius: "1 pod at a time"
    expected: "No user-visible impact"

  node_failure:
    description: "Drain and cordon a Kubernetes node"
    blast_radius: "All pods on one node"
    expected: "Pods reschedule to other nodes within 2 minutes"

  zone_failure:
    description: "Simulate an availability zone outage"
    blast_radius: "All resources in one AZ"
    expected: "Traffic fails over to remaining AZs"

Network Failures

experiments:
  latency_injection:
    description: "Add 500ms latency to database calls"
    expected: "Circuit breaker trips, fallback responses served"

  packet_loss:
    description: "Drop 10% of packets between services"
    expected: "Retry logic handles dropped requests"

  dns_failure:
    description: "Make DNS resolution fail for an external service"
    expected: "Cached responses served, graceful degradation"

  network_partition:
    description: "Block traffic between two services"
    expected: "Dependent service detects failure and serves degraded response"

Application Failures

experiments:
  cpu_stress:
    description: "Spike CPU to 100% on application pods"
    expected: "Autoscaler adds capacity, no dropped requests"

  memory_pressure:
    description: "Consume available memory until OOM"
    expected: "Pod is killed and restarted, no cascading failure"

  disk_fill:
    description: "Fill the disk to 95% capacity"
    expected: "Alerts fire, log rotation frees space"

  dependency_failure:
    description: "Make a downstream API return 500 errors"
    expected: "Circuit breaker opens, cached/default response served"

LitmusChaos for Kubernetes

LitmusChaos is an open-source chaos engineering platform designed for Kubernetes. It provides a catalog of pre-built experiments and a framework for creating custom ones.

Installation

# Add the Litmus Helm repository
helm repo add litmuschaos https://litmuschaos.github.io/litmus-helm/
helm repo update

# Install LitmusChaos
kubectl create namespace litmus
helm install litmus litmuschaos/litmus \
  --namespace litmus \
  --set portal.frontend.service.type=NodePort

Install the chaos experiments for your target namespace:

kubectl apply -f https://hub.litmuschaos.io/api/chaos/3.0.0?file=charts/generic/experiments.yaml -n production

Running a Pod Delete Experiment

Create a ChaosEngine resource to run a pod deletion experiment:

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: api-pod-delete
  namespace: production
spec:
  engineState: active
  appinfo:
    appns: production
    applabel: app=api
    appkind: deployment
  chaosServiceAccount: litmus-admin
  experiments:
    - name: pod-delete
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: "60"
            - name: CHAOS_INTERVAL
              value: "15"
            - name: FORCE
              value: "false"
            - name: PODS_AFFECTED_PERC
              value: "30"
        probe:
          - name: check-api-health
            type: httpProbe
            mode: Continuous
            httpProbe/inputs:
              url: http://api.production.svc:8080/health
              method:
                get:
                  criteria: ==
                  responseCode: "200"
            runProperties:
              probeTimeout: 5s
              interval: 5s
              retry: 3

Apply and monitor:

kubectl apply -f api-pod-delete.yaml

# Watch the experiment status
kubectl get chaosengine api-pod-delete -n production -w

# Check results
kubectl get chaosresult api-pod-delete-pod-delete -n production -o yaml

Network Chaos Experiment

Inject network latency between services:

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: api-network-latency
  namespace: production
spec:
  engineState: active
  appinfo:
    appns: production
    applabel: app=api
    appkind: deployment
  chaosServiceAccount: litmus-admin
  experiments:
    - name: pod-network-latency
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: "120"
            - name: NETWORK_LATENCY
              value: "500"
            - name: NETWORK_INTERFACE
              value: "eth0"
            - name: DESTINATION_IPS
              value: "10.0.5.0/24"
            - name: PODS_AFFECTED_PERC
              value: "50"

This adds 500ms of latency to 50% of the API pods when communicating with IPs in the 10.0.5.0/24 range (your database subnet, for example). Watch how your circuit breakers, timeouts, and retry logic handle the degradation.

Chaos Mesh as an Alternative

Chaos Mesh is another popular Kubernetes-native chaos engineering platform. Install it with Helm:

helm repo add chaos-mesh https://charts.chaos-mesh.org
helm install chaos-mesh chaos-mesh/chaos-mesh \
  --namespace chaos-mesh --create-namespace \
  --set chaosDaemon.runtime=containerd \
  --set chaosDaemon.socketPath=/run/containerd/containerd.sock

Define experiments using Chaos Mesh CRDs:

apiVersion: chaos-mesh.org/v1alpha1
kind: StressChaos
metadata:
  name: api-cpu-stress
  namespace: production
spec:
  mode: one
  selector:
    namespaces:
      - production
    labelSelectors:
      app: api
  stressors:
    cpu:
      workers: 2
      load: 80
  duration: "5m"
  scheduler:
    cron: "@every 2h"

This runs a CPU stress test on one API pod every two hours for five minutes, continuously validating that your system handles CPU pressure gracefully.

Building a Chaos Engineering Program

Start Small

Begin with non-production environments and simple experiments:

1. Week 1-2: Kill individual pods and verify they restart correctly.
2. Week 3-4: Inject network latency between services.
3. Month 2: Simulate node failures and zone outages.
4. Month 3: Run experiments in production during business hours with the team watching.
5. Month 4+: Automate experiments to run regularly.

Safety Controls

Always have safety mechanisms in place:

Blast radius limits: Start with one pod, one node, one zone. Never affect everything at once.

Abort conditions: Define automated rollback triggers. If the error rate exceeds 5% or latency exceeds 10 seconds, stop the experiment immediately.

Time limits: Every experiment has a maximum duration. If the system does not recover within the expected time, halt the experiment.

Communication: Announce experiments in advance. Even in mature organizations, surprise chaos experiments create unnecessary anxiety.

# Safety controls configuration
safety:
  abort_conditions:
    - metric: "error_rate"
      threshold: 0.05
      action: "abort_experiment"
    - metric: "p99_latency"
      threshold: 10
      action: "abort_experiment"
  max_duration: "10m"
  allowed_hours: "09:00-16:00"
  allowed_days: ["Monday", "Tuesday", "Wednesday", "Thursday"]
  excluded_dates: ["2026-12-25", "2026-01-01"]
  notification_channel: "#chaos-engineering"

GameDays

GameDays are scheduled events where the team practices responding to simulated failures. They are like fire drills for your infrastructure:

1. Schedule a 2-4 hour window with the team.
2. The chaos engineer (or a rotation) designs scenarios in advance.
3. The on-call team responds to the incidents as if they were real.
4. Observers take notes on the response process.
5. Debrief afterward to identify improvements.

GameDays build confidence, test runbooks, and train new team members in incident response.

Tracking Results

Maintain a chaos experiment registry:

experiment_registry:
  - id: CE-2026-042
    name: "Database failover"
    date: 2026-07-07
    environment: staging
    hypothesis: "Application handles primary database failover with < 5s of errors"
    result: FAILED
    findings:
      - "Connection pool did not detect failover for 45 seconds"
      - "Health check endpoint did not test database connectivity"
    action_items:
      - "Reduce connection pool health check interval to 5s"
      - "Add database connectivity check to /health endpoint"
    status: remediated
    retest_date: 2026-07-14
    retest_result: PASSED

Wrapping Up

Chaos engineering is about building confidence through controlled experimentation. Start with a clear steady-state definition, form specific hypotheses, and run experiments with appropriate safety controls. Tools like LitmusChaos and Chaos Mesh make it straightforward to inject failures into Kubernetes environments. The goal is not to break things for the sake of it but to discover weaknesses before your users do. Begin with simple pod deletions in staging, graduate to network failures, and eventually run regular automated experiments in production. Each experiment either confirms your system’s resilience or reveals something to fix, and both outcomes make your systems stronger.