Chaos Engineering: Principles and First Experiments

What Chaos Engineering Is

Chaos engineering is the practice of deliberately injecting failures into a system to discover weaknesses before they cause outages. It sounds counterintuitive — you are breaking things on purpose. But the goal is not destruction. It is building confidence that your system can withstand real-world turbulence.

Netflix pioneered the discipline in 2010 when they moved to AWS. They asked: “What happens when an EC2 instance dies?” Rather than theorize, they built Chaos Monkey to randomly kill instances in production. The failures it uncovered led to architectural improvements that made Netflix one of the most resilient systems on the internet.

Chaos engineering is not the same as testing. Testing verifies known behaviors: “Given input X, the output should be Y.” Chaos engineering explores unknown behaviors: “What happens to the system when this component fails? Do we even know?”

The Four-Step Process

Every chaos experiment follows this structure:

Step 1: Define Steady State

Before breaking anything, define what “normal” looks like. This is your steady-state hypothesis. It should be measurable:

• “The API responds to 99.9% of requests within 300ms.”
• “Users can complete checkout with less than 0.1% error rate.”
• “The dashboard loads within 2 seconds.”

Use metrics you already monitor: latency percentiles, error rates, throughput, and business KPIs.

Step 2: Hypothesize

Form a hypothesis about what will happen during the experiment:

• “If we kill one of three API replicas, the load balancer will redistribute traffic and the error rate will stay below 0.5%.”
• “If the database becomes unreachable for 10 seconds, the circuit breaker will open and the API will return cached data.”

The hypothesis should be specific enough that you can clearly confirm or refute it.

Step 3: Inject Failure

Introduce the failure condition in a controlled way:

• Kill a process or container.
• Add network latency or packet loss.
• Fill up a disk.
• Exhaust a connection pool.
• Return errors from a dependency.

Start small. Run the experiment in a staging environment first. Limit the blast radius.

Step 4: Observe and Learn

Compare the system’s actual behavior against your hypothesis:

• Did the error rate stay within acceptable bounds?
• Did failover happen automatically?
• Did alerts fire correctly?
• Did the system recover when the failure was removed?

If the hypothesis held, you have gained confidence. If it did not, you have found a weakness to fix — which is even more valuable.

Your First Experiment: Killing a Pod

If you run on Kubernetes, the simplest experiment is killing a pod and watching what happens.

Hypothesis

“If we delete one pod from the api deployment (which has 3 replicas), requests will continue to be served with no user-visible errors.”

The Experiment

# Check current state
kubectl get pods -l app=api
# NAME        READY   STATUS    RESTARTS   AGE
# api-abc12   1/1     Running   0          2h
# api-def34   1/1     Running   0          2h
# api-ghi56   1/1     Running   0          2h

# Kill one pod
kubectl delete pod api-abc12

# Watch recovery
kubectl get pods -l app=api -w

While the pod restarts, monitor your metrics:

• Are requests returning 5xx errors?
• Did the load balancer stop routing to the dead pod?
• How long until the replacement pod is ready?

What You Might Find

Common discoveries from this simple experiment:

• No readiness probe: The replacement pod receives traffic before the application finishes starting, causing errors.
• No graceful shutdown: In-flight requests are dropped when the pod is killed.
• Single replica: If you only have one pod, the entire service goes down.
• Slow startup: The application takes 60 seconds to start, causing a noticeable outage window.

Each discovery leads to a concrete improvement: add readiness probes, implement graceful shutdown, increase replica count, optimize startup time.

Experiment: Network Latency with Toxiproxy

Toxiproxy is a tool by Shopify that simulates network conditions. It sits between your application and its dependencies as a proxy:

# Install Toxiproxy
docker run -d --name toxiproxy -p 8474:8474 -p 5433:5433 shopify/toxiproxy

# Create a proxy for PostgreSQL
toxiproxy-cli create postgres -l 0.0.0.0:5433 -u postgres-host:5432

# Add 500ms latency
toxiproxy-cli toxic add -t latency -a latency=500 postgres

# Point your app at localhost:5433 instead of postgres-host:5432

Hypothesis

“If the database has 500ms of added latency, API response times will increase but stay below our 2-second timeout, and no requests will fail.”

What You Might Find

• No timeouts configured: Your application waits indefinitely for slow queries.
• Connection pool exhaustion: Slow queries hold connections longer, eventually starving new requests.
• No circuit breaker: The application keeps hammering the slow database instead of failing fast.
• Cascading failures: Upstream services time out waiting for your slow API, causing their own failures.

Experiment: Dependency Failure

What happens when an external service your application depends on is completely unavailable?

# Using Toxiproxy: make the payment service return errors
toxiproxy-cli toxic add -t timeout -a timeout=0 payment-service

# Or using iptables (Linux): block traffic to the payment service
iptables -A OUTPUT -d payment-service-ip -j DROP

Hypothesis

“If the payment service is unavailable, the checkout API returns a user-friendly error message and does not crash. Other endpoints (browsing, search) continue to work normally.”

Common Findings

• No fallback behavior: The entire application crashes or hangs when one dependency fails.
• Missing error handling: The application returns a stack trace instead of a user-friendly message.
• Tight coupling: A failure in the payment service brings down the product catalog because they share a thread pool or connection pool.
• No retry with backoff: The application retries immediately and relentlessly, making the situation worse.

Experiment: Disk Full

# Fill the disk (in a container, safely)
dd if=/dev/zero of=/tmp/fill bs=1M count=1000

# Or use fallocate for faster results
fallocate -l 10G /tmp/fill

Hypothesis

“If the disk fills up, the application logs a warning, stops writing to disk gracefully, and continues serving requests from memory/cache.”

Common Findings

• Silent failures: The application silently drops log entries, making debugging impossible later.
• Database crashes: The local database runs out of space for WAL files and stops accepting writes.
• No monitoring: Nobody gets alerted until a customer reports an issue.

Tools for Chaos Engineering

Chaos Monkey (Netflix)

The original chaos tool. Randomly terminates EC2 instances in production. Part of the Simian Army suite.

Litmus (Kubernetes)

Open-source chaos engineering framework for Kubernetes. Define experiments as YAML:

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: api-chaos
spec:
  appinfo:
    appns: default
    applabel: "app=api"
  experiments:
    - name: pod-delete
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: "30"
            - name: CHAOS_INTERVAL
              value: "10"

Gremlin

Commercial chaos engineering platform with a web UI. Supports CPU, memory, disk, network, and process-level experiments. Good for teams new to chaos engineering because it provides guardrails and automatic rollback.

Chaos Mesh (Kubernetes)

Another Kubernetes-native chaos platform. Supports pod failure, network chaos, I/O chaos, and time skew experiments.

Toxiproxy (Shopify)

Simulates network conditions (latency, bandwidth, timeouts) at the TCP level. Language-agnostic. Excellent for testing database and API client resilience.

Safety Practices

Chaos engineering can cause real damage if done carelessly. Follow these principles:

Start in staging. Run experiments in non-production environments until you are confident in your safety mechanisms.

Limit blast radius. Affect a small percentage of traffic or a single availability zone. Use feature flags to control experiments.

Have a kill switch. Every experiment must be instantly reversible. If things go wrong, you need to stop the experiment immediately.

Monitor in real time. Watch dashboards during the experiment. Do not run chaos experiments and walk away.

Get buy-in. Chaos engineering works best when the whole team understands and supports it. Surprise failures without context create panic, not learning.

Document everything. Record the hypothesis, the experiment, the observations, and the action items. This turns each experiment into institutional knowledge.

Building a Chaos Practice

Start small and build gradually:

1. Week 1-2: Run your first pod-kill experiment in staging. Fix any issues you find.
2. Month 1: Add network latency experiments with Toxiproxy. Verify your timeouts and circuit breakers work.
3. Month 2: Run dependency failure experiments. Verify fallback behavior and graceful degradation.
4. Month 3: Move experiments to production with limited blast radius (1% of traffic).
5. Ongoing: Run regular game days where the team practices incident response during controlled chaos.

Key Takeaways

Chaos engineering is not about breaking things for fun. It is a disciplined practice of running controlled experiments to discover how your system behaves under failure conditions. Start with simple experiments: kill a pod, add network latency, block a dependency. Define your steady-state hypothesis before the experiment, observe what actually happens, and fix the gaps you discover. The goal is to shift from “I hope it works” to “I know it works because I tested it under failure.” Every experiment that reveals a weakness is a production outage you prevented.