What is regression to the mean? Examples and how to avoid being fooled

What is regression to the mean?

Regression to the mean is the tendency of an unusually high or low result to be followed by one closer to the average, even when nothing has changed. Francis Galton described it in 1886: very tall parents tended to have children who were tall, but less extremely so.

It is not a force pulling things back. It is a selection effect.

Why it happens

Every measurement is a mix of a stable level and random noise:

observed = true level + noise

When you pick the most extreme results (the worst week, the best campaign, the lowest-converting pages), you are mostly picking cases where the noise happened to be large. Next period, the noise is drawn fresh, so those cases drift back towards their true level.

The more noise in a metric (small samples, short periods), the stronger the effect.

A sample example: fixing the worst pages

An SEO team finds the 20 product pages with the lowest conversion rate last month and rewrites them (sample numbers):

Without the control, the rewrite looks like it raised conversion by 0.8 points. With it, most of that recovery happened anyway; the rewrite's likely effect is about +0.2 points. The site average was 2.0% in both months.

Where it fools teams

• Interventions on the worst performers. Coaching the worst support agents, fixing the lowest pages, re-targeting the weakest campaigns. They usually improve partly on their own.
• Celebrating the best. Last quarter's top channel disappoints this quarter.
• Alerting. A rule that fires after a very low day will often be "resolved" the next day without any action. Pair it with recency bias and teams over-react.
• Before/after studies. Users who joined a programme because they had a bad month will look better afterwards.

How to guard against it

1. Use a control group chosen the same way (also extreme) but not changed.
2. Select on one period, measure on another. Never judge on the same data you used to pick the cases.
3. Use longer windows and larger samples so noise is a smaller part of each value.
4. Compare against a baseline, not against the extreme point.

In TapCub

When you pick pages or channels to fix from a report, note a second group of similar ones you leave alone, and compare both a month later using the previous-period comparison. Add an annotation on the day you made the change. For alerts, the monitor conditions with a baseline help: the forecast band compares with the last 7 days and the same weekday, and the anomaly condition uses the same weekday over 28 days, so one noisy day matters less. See also selection bias.