Chaos engineering is the discipline of experimenting on a system by deliberately injecting failures, such as killed instances, network latency or unavailable dependencies, to build confidence in its ability to withstand turbulent conditions. Each experiment starts from a hypothesis about steady-state behavior and uses observability data to confirm or refute it.
What chaos engineering is
Chaos engineering treats resilience as something to verify experimentally rather than assume. Instead of waiting for a node to crash or a dependency to slow down at 3 a.m., teams cause that failure on purpose, in a controlled way, and watch how the system reacts. The goal is not to break things for its own sake but to discover weaknesses, such as a missing timeout, a retry storm or a failover that was never actually tested, while the team is prepared and the blast radius is small.
The practice became widely known through Netflix, which built Chaos Monkey around 2010 to randomly terminate production instances during its move to AWS and later published the Principles of Chaos Engineering. Today the approach is used well beyond large streaming platforms, from Kubernetes clusters to payment systems.
Core principles
The Principles of Chaos Engineering describe an experiment in five steps:
- Define steady state as a measurable output that indicates normal behavior, for example successful checkouts per minute or an SLO, rather than internal metrics such as CPU.
- Hypothesize that steady state will continue in both a control group and an experimental group.
- Vary real-world events: server crashes, disk full, network partitions, latency, clock skew, dependency errors, traffic spikes.
- Run experiments in production where possible, because staging rarely reproduces real traffic and configuration.
- Automate experiments to run continuously and minimize the blast radius, so that a failed hypothesis harms as few users as possible.
The experiment succeeds if the hypothesis is disproved, because the team has learned something before an incident taught it the hard way.
Why observability comes first
An experiment is only as good as the ability to see its effect. Without a reliable baseline of steady state you cannot tell whether a 200 ms latency injection degraded the user experience or not. Without traces you cannot see which downstream service amplified the fault. Before running chaos experiments, make sure you have:
- User-facing SLOs and dashboards that show steady state in real time.
- Distributed tracing across the affected call paths to follow how a fault propagates.
- Alerting that works, because an experiment that should have paged and did not is itself a finding.
- A clear abort condition, such as an error budget burn rate threshold, that automatically stops the experiment.
Example experiment
An illustrative experiment for a checkout service that calls a payment provider:
| Element | Definition |
|---|---|
| Steady state | Checkout success ratio at least 99.5%, p99 latency under 1.2 s |
| Hypothesis | If the payment API adds 2 s latency for 10% of calls, timeouts and the circuit breaker keep checkout within steady state |
| Fault | Inject 2 s latency on egress to the payment API for one canary pod |
| Blast radius | One pod, about 5% of traffic, 15 minutes |
| Abort | Checkout success below 99% for 2 minutes |
A typical finding would be that the HTTP client timeout was longer than the upstream gateway timeout, so users saw errors although the service eventually got a response. The fix is small; finding it during an outage would have been expensive.
Tools and formats
Common open source tools for Kubernetes are Chaos Mesh and LitmusChaos, both CNCF projects. Cloud providers offer managed services such as AWS Fault Injection Service and Azure Chaos Studio. Toxiproxy simulates network conditions between services in tests. Many organizations start with game days: scheduled sessions where a team runs a scenario manually, follows its runbooks and checks whether alerts, dashboards and escalation paths work as expected.
How to get started
A practical progression looks like this. First, pick one critical user journey and make its steady state visible. Second, run a manual game day in a non-production environment with a simple fault such as terminating one instance, and check that alerts fire and dashboards explain what happened. Third, repeat the same experiment in production with a tiny blast radius during business hours, with everyone who owns the service available. Only then automate it, for example as a scheduled job or as a step in the release pipeline, and widen the scope one dependency at a time. Each step should produce a short written result: hypothesis, outcome, what was fixed.
Common mistakes
Starting with large, uncontrolled experiments in production without an abort mechanism damages trust in the practice. Measuring internal metrics instead of user-facing steady state hides the actual impact. And running an experiment without recording the results, fixes and follow-ups in something like a postmortem wastes the learning.
How Obseria helps
Obseria provides the steady-state view chaos experiments depend on: SLOs, live service maps from Obseria TopoGraph and traces that show how an injected fault travels. Synthetic monitoring keeps a constant probe on critical journeys during the experiment, and burn-rate alerts can serve as the automatic abort signal.
