What is the false discovery rate? Benjamini-Hochberg explained

What is the false discovery rate?

The false discovery rate (FDR) is the expected proportion of false positives among all the results you declare significant. If you run 40 tests and call 10 of them significant, an FDR of 5% means you expect, on average, about half a false one among those 10.

It matters whenever you test many things at once: twenty metrics in an A/B test, every country in a breakdown, hundreds of features in a model. Test enough hypotheses at p < 0.05 and some will cross the line by chance. The Benjamini-Hochberg (BH) procedure, published in 1995, is the standard way to control the FDR.

FDR vs family-wise error rate

The Benjamini-Hochberg procedure

With m tests and a target FDR level q (often 0.05):

1. Sort the p-values from smallest to largest: p₍₁₎ ≤ p₍₂₎ ≤ … ≤ p₍ₘ₎.
2. For each rank i, compute the threshold (i ÷ m) × q.
3. Find the largest i for which p₍ᵢ₎ ≤ (i ÷ m) × q.
4. Declare that test and all tests with smaller p-values significant, even if some of them sat above their own threshold.

A worked example

Sample data: an A/B test of a new onboarding flow reports ten secondary metrics. q = 0.05, m = 10.

The largest rank below its threshold is 3, so three metrics are discoveries. Compare the three approaches:

Without correction, "time to first report", "support chats" and "docs views" would have been reported as wins; with ten metrics, a couple of chance results are expected. Bonferroni keeps only activation and misses two effects that are probably real.

Adjusted p-values

Instead of thresholds, many tools report BH-adjusted p-values (sometimes called q-values): for rank i, the smallest value of (m ÷ j) × p₍ⱼ₎ over all j ≥ i, capped at 1. A result is significant at FDR q when its adjusted p-value is ≤ q. In the table, the three discoveries are exactly the ones with adjusted p ≤ 0.05.

Assumptions and limits

• BH controls the FDR when the tests are independent or positively dependent, which covers most metric sets in practice. For arbitrary dependence, the Benjamini-Yekutieli variant is valid but more conservative.
• FDR is an average over many studies; any single list of discoveries may contain more or fewer false ones.
• It does not fix peeking or p-hacking; decide which tests are in the family before looking.

When to use it

Use BH when you screen many hypotheses and will follow up on the discoveries: secondary metrics in experiments, many segments in a breakdown, many features or keywords. Use a family-wise method such as Bonferroni when a single false claim is costly, such as the primary metric of a pricing decision. Pre-register one primary metric either way.

FDR and TapCub

TapCub reports breakdowns and comparisons, not p-values, and it does not apply multiple-comparison corrections. The idea still applies: when Why did it change or a breakdown across dozens of countries or pages highlights a few unusual values, some will be unusual by chance. Treat them as hypotheses, as described in diagnostic analytics, and if you test them formally, adjust for how many you looked at. For background on single tests, see p-value.

Sources

• Benjamini, Y. and Hochberg, Y. (1995). Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. Journal of the Royal Statistical Society B, 57(1), 289–300.
• Benjamini, Y. and Yekutieli, D. (2001). The control of the false discovery rate in multiple testing under dependency. Annals of Statistics, 29(4), 1165–1188.