A canary deployment is a release strategy in which a new software version is first rolled out to a small subset of users or traffic, while the stable version keeps serving the rest. Error rates, latency and business metrics of the canary are compared with the stable baseline, and the rollout is expanded or rolled back based on that analysis.
What a canary deployment is
A canary deployment, or canary release, limits the blast radius of a change. Instead of replacing every instance of a service at once, you run the new version next to the stable one and route a small share of real traffic to it, for example 5%. If the canary behaves as well as the stable version, the share is increased step by step until the new version serves everything. If it misbehaves, traffic is shifted back and only a small fraction of users ever saw the problem.
The name comes from the canaries coal miners once carried underground: the bird reacted to toxic gas before people did. In software, the small group of requests hitting the new version plays the same role as an early warning system.
How a canary rollout works
- Deploy the canary alongside the stable version, with identical configuration apart from the change under test.
- Shift a small share of traffic, using a load balancer, service mesh, ingress controller or feature flag. Some teams route by percentage; others route specific cohorts such as internal users or one region.
- Analyze the canary against the stable version over a defined window: error rate, latency percentiles, saturation and key business metrics.
- Promote or roll back. If the canary passes, increase the weight, for example 5% to 25% to 50% to 100%. If it fails, route all traffic back to the stable version.
In Kubernetes, tools such as Argo Rollouts and Flagger automate these steps, querying metrics providers between each step and aborting automatically when analysis fails.
Why canary deployments matter
Most production incidents are triggered by change: new code, new configuration, new infrastructure. Tests and staging environments catch many problems, but not those that only appear with real traffic, real data and real dependencies. A canary turns production into a controlled experiment. Compared with a blue-green deployment, where all traffic switches at once, a canary exposes far fewer users to a bad release. Compared with a plain rolling update, it adds an explicit decision point based on data.
Example: a canary analysis
An illustrative checkout service runs ten stable pods. The team deploys one canary pod and routes 10% of traffic to it for 15 minutes. The analysis compares both versions:
| Signal | Stable | Canary | Rule | Result |
|---|---|---|---|---|
| HTTP 5xx rate | 0.12% | 0.15% | at most +0.1 points | Pass |
| p95 latency | 210 ms | 340 ms | at most +20% | Fail |
| Orders per minute per pod | 38 | 37 | at most -10% | Pass |
The p95 latency regression fails the analysis, so the rollout is aborted. A trace comparison between both versions shows that the new code path issues an additional database query per request. Only 10% of traffic for 15 minutes was affected.
Best practices
- Compare canary with stable at the same time, not with yesterday. Traffic patterns change during the day, and a concurrent baseline removes that noise.
- Use enough traffic and time. With very low request counts, statistical comparisons are meaningless. Low-traffic services need longer analysis windows.
- Tag telemetry with the version. OpenTelemetry's
service.versionresource attribute lets you split every metric, log and trace by release. - Analyze user-facing signals, ideally the same indicators that back your SLOs.
- Keep changes backward compatible. Both versions run in parallel, so database schemas, message formats and APIs must work for both.
- Automate rollback. A canary that requires a human to notice a problem at 3 a.m. loses much of its value.
Common mistakes
Teams sometimes only check that canary pods pass their health checks. A pod can be live and ready while returning wrong results or responding slowly. Another mistake is sticky routing that sends one user's requests to different versions, which causes confusing behavior in sessions and caches. Finally, canaries do not protect against changes that affect all versions at once, such as a shared database migration or a configuration pushed globally.
How Obseria helps
Obseria deployment tracking records every release and splits error rate, latency and throughput by version, so canary and stable can be compared side by side. Anomaly detection highlights regressions, and the Obseria API and Automation workflows can serve as a metrics source for automated promotion or rollback decisions in Kubernetes rollouts.