A circuit breaker is a resilience pattern that wraps calls to a remote dependency and stops sending requests when failures or slow calls exceed a threshold. It moves between three states: closed (calls pass), open (calls fail fast) and half-open (a few trial calls test whether the dependency has recovered).
What a circuit breaker is
A circuit breaker protects a service from a dependency that is failing or responding slowly. Without one, every caller keeps sending requests, waits for timeouts and ties up threads, connections and memory. The slowness spreads upstream until services that were never broken run out of capacity too: a cascading failure. The pattern, popularized by Michael Nygard's book Release It!, borrows its name from the electrical component that cuts the current before wiring overheats.
The breaker sits in the client, in a library such as Resilience4j (Java) or Polly (.NET), or in the network layer, for example in a service mesh proxy such as Envoy.
The three states: closed, open, half-open
| State | Behavior | Transition |
|---|---|---|
| Closed | Calls pass through; outcomes are recorded in a sliding window. | Opens when the failure or slow-call rate exceeds the threshold. |
| Open | Calls are rejected immediately, typically with a fallback. | Moves to half-open after a configured wait duration. |
| Half-open | A limited number of trial calls is allowed. | Closes if trials succeed, opens again if they fail. |
The key benefit of the open state is failing fast: the caller gets an answer in microseconds instead of waiting for a timeout, and the struggling dependency gets breathing room to recover instead of being hammered by retries.
Example configuration
An illustrative Resilience4j configuration for a payment client:
resilience4j.circuitbreaker:
instances:
payments:
slidingWindowType: COUNT_BASED
slidingWindowSize: 50
minimumNumberOfCalls: 20
failureRateThreshold: 50
slowCallDurationThreshold: 2s
slowCallRateThreshold: 80
waitDurationInOpenState: 30s
permittedNumberOfCallsInHalfOpenState: 5Read it as: evaluate the last 50 calls, but only once at least 20 have been made; open the circuit if half of them fail or 80% take longer than two seconds; stay open for 30 seconds, then let five trial calls decide. The minimum call count prevents a single failure at low traffic from tripping the breaker.
Why it matters for observability
Circuit breakers change what your telemetry shows. While a breaker is open, the downstream service may look healthy and quiet, because it receives no traffic, while the caller reports a spike in rejected calls. When you investigate an incident, you need to see both sides: the breaker state, the rejection count and the original error rate or latency that tripped it. Useful signals are:
- Breaker state per instance and dependency, as a gauge or as state-change events.
- Calls by outcome: successful, failed, slow, rejected (not permitted).
- Failure and slow-call rates compared with the configured thresholds.
- Fallback executions, which hide errors from users but still mean degraded service.
In distributed traces, mark spans that were short-circuited, for example with an attribute or a span event, so a fast rejection is not mistaken for a healthy call.
Circuit breaking in a service mesh
Proxies such as Envoy implement the idea differently from client libraries. Envoy's "circuit breakers" are limits on concurrent connections, pending requests, active requests and retries per upstream cluster: once a limit is reached, further requests are rejected immediately. Ejecting individual unhealthy hosts is handled by outlier detection, which removes an endpoint from the load-balancing pool after, for example, a number of consecutive 5xx responses and returns it after an ejection period. In Istio both are configured in a DestinationRule through connectionPool and outlierDetection. The mesh approach works without code changes and applies consistently across languages, while library-based breakers understand application semantics better, such as which exceptions should count as failures and which fallback makes sense.
Best practices
- Combine with timeouts. A breaker only counts slow calls if calls have a timeout in the first place.
- Retry carefully. Put retries inside the breaker's budget and use exponential backoff with jitter, or retries will keep the breaker open.
- Scope per dependency. One breaker per downstream service or endpoint, not one for all outbound traffic.
- Design the fallback. Cached data, a default value or a clear error message; decide this deliberately per use case.
- Alert on state changes. An open breaker is an early, precise signal that a dependency is in trouble.
Common mistakes
Typical mistakes are thresholds copied from examples without load testing, breakers that never open because errors are swallowed and returned as HTTP 200, and fallbacks that silently serve stale data for hours without anyone noticing. Another trap is counting client errors such as HTTP 400 as failures, which lets a single misbehaving caller open the circuit for everyone.
How Obseria helps
Obseria correlates the caller's rejected and failed spans with the downstream service's own telemetry in one trace view, and Obseria TopoGraph shows which services depend on the affected one. Breaker metrics exported via OpenTelemetry or Prometheus can drive alerts, and error tracking groups the exceptions that tripped the breaker.
