What is Simpson's paradox? Examples and how to spot it

What is Simpson's paradox?

Simpson's paradox is when a pattern that holds in every group reverses, or disappears, once the groups are added together. It is named after Edward Simpson, who described it in 1951, although Karl Pearson and Udny Yule noted it earlier.

It is not a trick of arithmetic. It happens whenever groups have different mixes and the mixing factor also affects the outcome.

A conversion-rate example

Two traffic channels, split by device (sample numbers):

Channel B converts better on desktop and on mobile. Yet overall, Channel A looks clearly better. The reason: two thirds of A's visitors are on desktop, where everyone converts well, while most of B's visitors are on mobile, where everyone converts poorly. Device is a confounding variable that drives both the mix and the outcome.

Why it happens

An overall rate is a weighted average of the group rates:

overall rate = Σ (share of visitors in group × rate in group)

When two things being compared have very different shares in a high-rate group, the weights can outweigh the differences within each group.

Two classic cases

• UC Berkeley admissions, 1973. Overall, men were admitted at a higher rate than women. Split by department, most departments admitted women at similar or higher rates; women had applied more often to departments that rejected most applicants.
• Kidney stone treatments, 1986. One treatment had a higher success rate for both small and large stones, but a lower one overall, because it was used more often on the harder, large stones.

Which number to trust

• Trust the split when the splitting factor (device, stone size, department) existed before and affects the outcome, and is not caused by what you are comparing. That is the usual case in analytics.
• Trust the combined number when the splitting factor is a result of the treatment. Splitting on it hides the effect.
• If the mix itself is what you can change (for example, sending more desktop traffic), the overall number still matters for planning.

Catching it in TapCub

Before concluding that one channel, page or campaign "converts better", click a device, source or country row to add it as a filter and repeat the comparison inside each value. Check new vs returning visitors too; they often have very different conversion rates and very different shares by channel. A stratified comparison like this takes a few clicks and catches the paradox before it reaches a slide. See also conversion rate.