Circuit Breaker Pattern for Resilient Microservices

In a microservices architecture, Service A calls Service B, which calls Service C. When Service C goes down, naive retry logic in Service B hammers the failing service, exhausting threads and connections. Service B’s response times spike, which causes Service A to time out, which causes the API gateway to time out, which causes the user to see an error. One failing service takes down the entire chain. This is a cascading failure, and the circuit breaker pattern exists to prevent it.

The Analogy

An electrical circuit breaker trips when it detects excessive current, preventing a fire. A software circuit breaker trips when it detects excessive failures, preventing cascading damage. When the breaker is open, calls fail immediately without contacting the downstream service, giving it time to recover.

The Three States

                  failure threshold
                     exceeded
     ┌────────┐  ─────────────>  ┌────────┐
     │ CLOSED │                  │  OPEN  │
     │        │  <─────────────  │        │
     └────────┘   (never auto)   └───┬────┘
         ^                           │
         │                     timeout expires
         │                           │
         │        ┌──────────┐       │
         └────────│HALF-OPEN │<──────┘
          success └──────────┘
          threshold   │
          reached     │ failure
                      │ detected
                      v
                  ┌────────┐
                  │  OPEN  │
                  └────────┘

Closed: everything is normal. Requests pass through to the downstream service. The breaker monitors failures. If the failure rate exceeds a threshold within a time window, the breaker trips to Open.

Open: all requests fail immediately with a fallback response or error. No calls reach the downstream service. After a configured timeout (the recovery window), the breaker transitions to Half-Open.

Half-Open: a limited number of probe requests are sent to the downstream service. If they succeed, the breaker resets to Closed. If any fail, the breaker returns to Open for another recovery window.

Implementation in Python

Here is a minimal but functional circuit breaker:

import time
from enum import Enum

class State(Enum):
    CLOSED = "closed"
    OPEN = "open"
    HALF_OPEN = "half_open"

class CircuitBreaker:
    def __init__(
        self,
        failure_threshold=5,
        recovery_timeout=30,
        half_open_max_calls=3
    ):
        self.failure_threshold = failure_threshold
        self.recovery_timeout = recovery_timeout
        self.half_open_max_calls = half_open_max_calls
        self.state = State.CLOSED
        self.failure_count = 0
        self.last_failure_time = None
        self.half_open_calls = 0

    def call(self, func, *args, **kwargs):
        if self.state == State.OPEN:
            if time.time() - self.last_failure_time > self.recovery_timeout:
                self.state = State.HALF_OPEN
                self.half_open_calls = 0
            else:
                raise CircuitOpenError("Circuit is open")

        if self.state == State.HALF_OPEN:
            if self.half_open_calls >= self.half_open_max_calls:
                raise CircuitOpenError("Half-open call limit reached")
            self.half_open_calls += 1

        try:
            result = func(*args, **kwargs)
            self._on_success()
            return result
        except Exception as e:
            self._on_failure()
            raise

    def _on_success(self):
        if self.state == State.HALF_OPEN:
            self.state = State.CLOSED
        self.failure_count = 0

    def _on_failure(self):
        self.failure_count += 1
        self.last_failure_time = time.time()
        if self.failure_count >= self.failure_threshold:
            self.state = State.OPEN

class CircuitOpenError(Exception):
    pass

Usage:

import httpx

breaker = CircuitBreaker(failure_threshold=3, recovery_timeout=60)

def get_user_profile(user_id):
    try:
        return breaker.call(
            httpx.get,
            f"https://user-service/users/{user_id}",
            timeout=5
        )
    except CircuitOpenError:
        # Return cached or default response
        return {"user_id": user_id, "name": "Unknown", "cached": True}

Configuration Parameters

Choosing the right configuration is more important than the implementation itself.

Failure threshold

How many failures before the breaker trips. Too low and transient errors trip the breaker unnecessarily. Too high and the downstream service suffers prolonged hammering.

A percentage-based threshold is often better than a raw count: “trip when the failure rate exceeds 50 percent in a 10-second window.” This avoids tripping on a single failure during low-traffic periods.

Recovery timeout

How long the breaker stays open before trying half-open probes. Start with 30 to 60 seconds. If the downstream service takes minutes to recover (for example, during a deployment rollback), increase this.

Half-open probe count

How many requests to allow in half-open state. One is aggressive. Three to five is more common, giving a better signal of actual recovery.

Sliding window

Track failures in a sliding time window rather than cumulatively. A breaker that counts failures “forever” will eventually trip on any long-lived service. A 10 to 30 second sliding window is typical.

Circuit Breaker in Java with Resilience4j

Resilience4j is the standard library for circuit breakers in Java:

CircuitBreakerConfig config = CircuitBreakerConfig.custom()
    .failureRateThreshold(50)              // 50% failure rate
    .waitDurationInOpenState(Duration.ofSeconds(30))
    .slidingWindowSize(10)                 // last 10 calls
    .slidingWindowType(SlidingWindowType.COUNT_BASED)
    .permittedNumberOfCallsInHalfOpenState(3)
    .build();

CircuitBreaker breaker = CircuitBreaker.of("userService", config);

Supplier<UserProfile> supplier = CircuitBreaker
    .decorateSupplier(breaker, () -> userServiceClient.getProfile(userId));

Try<UserProfile> result = Try.ofSupplier(supplier)
    .recover(CallNotPermittedException.class,
             e -> UserProfile.defaultProfile(userId));

Fallback Strategies

The circuit breaker is only useful if you have a meaningful fallback when the circuit is open:

• Cached response: return the last known good response from a local cache.
• Default value: return a safe default. A product page without personalized recommendations is better than an error page.
• Degraded service: skip the failing dependency and return partial data. Show the order without the shipping estimate.
• Queue for later: accept the user’s request and process it when the downstream service recovers.

Request flow with fallback:

  User -> API Gateway -> Order Service
                              │
                         ┌────┴────┐
                    ┌────┤ Circuit  ├────┐
                    │    │ Breaker  │    │
                    │    └─────────┘    │
                CLOSED              OPEN
                    │                   │
            ┌───────┴───────┐   ┌──────┴──────┐
            │ Payment Svc   │   │ Return      │
            │ (live call)   │   │ cached/     │
            └───────────────┘   │ default     │
                                └─────────────┘

Circuit Breaker vs Retry vs Timeout

These three patterns are complementary, not competing:

Pattern	Purpose	Without it
Timeout	Bound how long you wait for a response	Threads blocked indefinitely
Retry	Handle transient failures	Single blip causes user error
Circuit breaker	Stop calling a failing service	Cascading failure across services

The typical composition is: set a timeout on each call, retry transient failures two to three times with exponential backoff, and wrap the whole thing in a circuit breaker.

Circuit Breaker
  └─ Retry (max 3, exponential backoff)
       └─ Timeout (5 seconds)
            └─ HTTP call to downstream service

Observability

A circuit breaker without monitoring is a black box. You need to track:

• State transitions: when does the breaker trip and recover? Alert on transitions to Open.
• Failure rate: the percentage of calls that fail within the sliding window.
• Rejected calls: how many calls were short-circuited while the breaker was open.
• Recovery time: how long the downstream service took to recover.

Export these as metrics to Prometheus or your monitoring system. A dashboard showing circuit breaker state across all services gives immediate visibility into which dependencies are unhealthy.

Common Pitfalls

Sharing a breaker across unrelated endpoints: if one endpoint on a service is slow but another is healthy, a shared breaker blocks both. Use separate breakers per endpoint or per operation.

Not distinguishing error types: a 400 Bad Request is a client error, not a downstream failure. Only count 5xx errors and timeouts as failures.

Missing fallbacks: a circuit breaker that throws an exception when open is only marginally better than no breaker at all. The value is in graceful degradation.

Testing only the happy path: simulate downstream failures in staging. Chaos engineering tools like Chaos Monkey or Toxiproxy are designed for this.

Key Takeaways

The circuit breaker pattern is essential in any microservices architecture. It prevents a single failing service from taking down the entire system by failing fast and giving the downstream service time to recover. Configure your thresholds based on traffic patterns, always provide meaningful fallbacks, and monitor state transitions so you know when services are struggling.