What is CUPED? Variance reduction for A/B tests explained

What is CUPED?

CUPED (Controlled-experiment Using Pre-Experiment Data) is a way to make A/B test results less noisy. It was published by researchers at Microsoft in 2013 and is now built into most experimentation platforms.

The idea: much of the difference between users in a test is just who they are. Heavy users spend more in both groups. If you know how much each user spent before the test, you can subtract out that predictable part and compare what is left.

The formula

For each user, with outcome Y during the test and the same (or a related) metric X before it:

Y_adj = Y − θ (X − mean(X))

θ = cov(X, Y) / var(X)

θ is the slope you would get regressing Y on X. Compare the average Y_adj between groups instead of the average Y. Since the treatment cannot change X (it happened earlier), the expected difference between groups stays the same; only the noise shrinks.

A small worked example

Five users, pre-period sessions X, test-period sessions Y, with θ = 0.8 and mean(X) = 10 (sample numbers):

The raw values range from 4 to 18. The adjusted ones range from 10.4 to 11.6. The average barely changes (11.0 in both cases), but the spread is far smaller, so a real treatment effect stands out sooner.

How much does it help?

Variance falls by about ρ², where ρ is the correlation between X and Y:

Engagement and revenue metrics for returning users often correlate strongly week to week, so gains of 30–50% are common for them. Conversion of first-time visitors, with no history, gains little.

Choosing the covariate and pitfalls

• Use the same metric from before the test when you can; it usually correlates best.
• The window must end before assignment. Anything measured after users saw the change can be affected by it and will bias the result.
• New users have no history; give them the mean value (no adjustment) or handle them as a separate stratum.
• Estimate θ on both groups pooled, not separately per group.
• CUPED raises power; it does not fix peeking, many comparisons (see Bonferroni correction) or a broken randomisation.

In TapCub

TapCub does not run experiments or apply CUPED. If you run tests with another tool, you can compute the pre-period covariate from TapCub data: export per-person events for the weeks before the test started and count sessions or purchases per user, then join that with your experiment assignments in a notebook. Person-level analysis needs identified mode, because anonymous visitors have no profile to join on.