Quick answer
Statistical power is the probability that a test will find a statistically significant result when a real effect of a specified size exists; it equals 1 minus the probability of a type II error (a false negative).
Formula: Power = 1 − β; n per group ≈ (z₁₋α/₂ + z₁₋β)² × [p₁(1 − p₁) + p₂(1 − p₂)] / (p₁ − p₂)²
What is statistical power?
Statistical power is the probability that a test detects an effect if the effect is really there. A test with 80% power will find a true difference of the size you planned for in 80 out of 100 repeats; in the other 20 it misses it (a type II error, or false negative).
Power = 1 − β, where β is the false-negative rate. For the other kind of error, see type I vs type II errors.
Four quantities that move together
| Quantity | Effect on power |
|---|---|
| Sample size | More data → more power |
| Effect size | Bigger true difference → more power |
| Significance level (α) | Stricter α (0.01 instead of 0.05) → less power |
| Variability | Noisier metric → less power |
Fix any three and the fourth is determined. That is what a power analysis does: you choose α, the power you want and the smallest effect worth detecting, and it tells you the sample size.
A worked example: conversion from 4% to 5%
You want to detect a lift from a 4% to a 5% conversion rate, α = 0.05 (two-sided):
| Visitors per group | Power |
|---|---|
| 1,000 | 19% |
| 2,000 | 33% |
| 4,000 | 58% |
| 6,742 | 80% |
| 10,000 | 93% |
With 2,000 visitors per group, a real one-point lift would be missed two times out of three.
The sample size for 80% comes from:
n ≈ (1.96 + 0.84)² × [0.04 × 0.96 + 0.05 × 0.95] / (0.05 − 0.04)² ≈ 6,742 per group
Five ways to raise power
- Run longer or include more traffic. The direct route.
- Test bigger changes. Small tweaks produce small effects that need huge samples.
- Use a less noisy metric. A click on the next step varies less than revenue per visitor.
- Reduce variance with pre-period data. Methods such as CUPED adjust for each user's past behaviour.
- Test fewer things at once. Many comparisons need a stricter α, such as a Bonferroni correction, which lowers power for each.
Why underpowered tests are costly
- They miss real improvements, so good ideas get dropped.
- When they do reach significance, the observed effect is usually too large (sometimes called the winner's curse), so forecasts based on it disappoint.
- "Not significant" gets reported as "no effect", which it is not. Report the confidence interval as well as the p-value.
Estimating the inputs from TapCub
TapCub does not run A/B tests, but it gives you the inputs for a power analysis: the baseline conversion rate of the goal you will test, and the traffic per week on the page or segment involved. Divide the required sample by weekly visitors to see how long a test would need. If the answer is many months, test a bigger change or a higher-traffic step. See also minimum detectable effect.
Was this explanation clear?
We use your answers to fix and expand this page.
Couldn’t send. Please try again in a minute.
Thanks for the feedback. We’ll use it to improve this page.You’ve already sent feedback on this page. Thanks.
Product Analyst
Focuses on event analysis, conversion funnels, and retention, using clear metric definitions and structured analysis to explain product performance.