Effect size: what it is, formulas and why p-values are not enough

What is effect size?

Effect size is a measure of how big a difference or relationship is, separate from whether it is statistically significant. A p-value tells you whether an effect is likely to be real; effect size tells you whether it is large enough to matter.

Effect sizes come in two forms:

• Unstandardized: in the metric's own units. "+0.6 percentage points of conversion", "+$4 average order value", "+2.5% relative lift". These are what most business decisions need.
• Standardized: scaled by variability so effects can be compared across metrics and studies. Cohen's d, Cohen's h and the correlation coefficient r are the common ones.

Effect size formulas

Cohen's d for two means:

d = (mean₂ − mean₁) ÷ pooled standard deviation
pooled SD ≈ √((SD₁² + SD₂²) ÷ 2)   (equal group sizes)

Cohen's h for two proportions such as conversion rates:

h = 2 × arcsin(√p₂) − 2 × arcsin(√p₁)

Correlation r for the strength of a linear relationship between two numeric variables, from −1 to 1.

Cohen's conventional benchmarks:

Cohen offered these for behavioral science when nothing better is known. In product and marketing data most real effects are "small" by these labels and still worth millions, so judge size against the business, not the labels.

Worked example: significance is not size

Three comparisons, all sample data:

How to read it:

• The first test is "significant" only because of the enormous sample. A 2.5% relative lift may or may not justify the cost of the change; the p-value cannot tell you.
• The second test is not significant, but the observed lift is ten times larger. The sample is simply too small to be sure; the honest conclusion is "possibly a big effect, needs more data", not "no effect". See statistical power.
• The third shows why standardized labels mislead in business: d = 0.08 is "negligible" on Cohen's scale, yet +$4 per order on 2,000 orders a week is $416,000 a year.

Why effect size matters

• Decisions need magnitude. "Is it worth shipping?" depends on the size of the gain against cost and risk.
• Planning tests. The smallest effect worth detecting, the minimum detectable effect, sets the required sample size.
• Comparing results. Effect sizes let you compare a change in signups with a change in revenue, or combine several studies.
• Avoiding p-value traps. Large samples make trivial effects significant; small samples hide big ones. Reporting only p-values invites both mistakes, and searching for a p below 0.05 leads to p-hacking.

How to report effect size

Report three things together:

1. The effect in business units: absolute and relative change, e.g. "+0.6 points (+15%)".
2. Its uncertainty: a confidence interval for the difference, e.g. "95% interval +0.2 to +1.0 points".
3. The p-value, if a decision rule needs one.

"Conversion rose from 4.0% to 4.6% (+0.6 points, 95% interval −0.2 to +1.4)" says far more than "p = 0.14". See statistical significance and p-value.

Effect size in analytics data

In analytics you rarely need Cohen's d. The useful effect sizes are concrete: change in conversion rate in points, revenue per visitor, retained users per cohort, minutes saved per task. When you compare periods or segments in TapCub, compute the absolute and relative difference from the counts, then add an interval. For controlled tests run in an experimentation tool, also track guardrail metrics so that a positive effect on one metric does not hide a negative one elsewhere.

Sources

• Sullivan, G. M. and Feinn, R. (2012). Using effect size, or why the P value is not enough. Journal of Graduate Medical Education, 4(3).
• Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences, 2nd ed. Lawrence Erlbaum.
• Lakens, D. (2013). Calculating and reporting effect sizes to facilitate cumulative science. Frontiers in Psychology, 4.