What is canary testing? Canary releases explained with an example

What is canary testing?

Canary testing, or a canary release, sends a new version of software to a small share of real production users or traffic first, compares its health with the current version running alongside it, and rolls it out further only if it holds up. If the canary looks worse, traffic goes back to the old version before most users ever see the problem.

The name comes from the canaries miners carried underground: the birds reacted to toxic gas before people did. The canary group is the early warning.

Canary testing vs other release practices

In practice these combine: code deploys behind a canary, and features inside it are switched on with flags.

How canary testing works

1. Deploy the new version next to the current one. Both serve production traffic.
2. Route a small share to the canary, often 1–5%, assigned randomly and kept sticky per user.
3. Compare canary with baseline over the same window. Compare against the current version running now, not against yesterday, so time-of-day and traffic mix cancel out. Google's SRE guidance recommends exactly this side-by-side comparison.
4. Check a short list of metrics: error rate, latency (p95/p99), crash-free users for apps, and one or two business signals such as checkout completion.
5. Promote in steps (for example 5% → 25% → 50% → 100%) or roll back automatically if a threshold is crossed.

A worked example: promote or roll back?

Sample data for one hour with a 5% canary. The rule, set in advance: roll back if the canary's error rate is more than 1.2× the baseline's.

Ratio = 0.42 ÷ 0.31 = 1.35, above the 1.2 threshold, so the canary is rolled back. Only 5% of users were exposed, for one hour. Note that the canary still had only 42 errors: with 1% of traffic it would have had about 8, too few to tell apart from noise. Choose the canary share so each window produces enough events to decide.

Mobile apps: phased and staged rollouts

Apps cannot be rolled back on users' phones, so the stores provide the canary mechanism:

• Apple phased release spreads an update to users with automatic updates on over seven days (1%, 2%, 5%, 10%, 20%, 50%, 100%). You can pause it for up to 30 days in total.
• Google Play staged rollouts release to a percentage of users you choose and can be increased, or halted if problems appear.

The comparison works the same way: the new version's crash-free users and error counts against the previous version's, over the same days.

Common pitfalls

• Comparing with the past instead of with the baseline running now.
• Canary too small to produce enough events in the window.
• Only technical metrics. A release can be error-free and still break checkout; include one business guardrail metric.
• Non-sticky assignment, where a user bounces between versions mid-session.
• Long-lived canaries that drift into an unplanned permanent split.

Measuring a canary with TapCub

TapCub does not deploy code or route traffic; it measures the result.

• Apps: App analytics › Users › Versions shows each version's share of active users and crash count, and App analytics › Quality & attribution › Stability gives the crash-free share filtered by version. See crash-free rate and track app versions.
• Web: the performance report shows JavaScript error count and error rate (errors ÷ pageviews). See find JavaScript errors.
• Alerts: a monitor on the built-in error_rate or errors measure can notify you when a release pushes errors past a threshold, and an annotation marks the deploy on every chart.

Sources

• Sato, D. (2014). Canary Release. martinfowler.com.
• Google, The Site Reliability Workbook, Chapter 16: Canarying Releases.
• Apple, Release a version update in phases.
• Google Play Console Help, Release app updates with staged rollouts.