Base rate fallacy: definition, examples and how to avoid it

What is the base rate fallacy?

The base rate fallacy (also called base rate neglect) is the mistake of judging how likely something is from specific evidence while ignoring how common it is in the first place: its base rate. When the base rate is low, even a fairly accurate signal produces mostly false alarms, and people consistently underestimate how many.

It was described by Daniel Kahneman and Amos Tversky in the 1970s; their "cab problem" asked people to judge a witness's report of a taxi's color while ignoring that 85% of the city's cabs were one color. In analytics it shows up whenever a rare event meets an imperfect detector: fraud, bots, churn predictions, "winning" experiments.

The math: Bayes' theorem

The probability that something is true given a positive signal is:

P(true | flagged) = P(flagged | true) × P(true)
                    ÷ [ P(flagged | true) × P(true) + P(flagged | false) × P(false) ]

In plain words: of everything flagged, what share is real? That share is the precision, or positive predictive value. It depends heavily on the base rate P(true), not only on the detector's accuracy.

Example 1: a bot detector

Suppose 2% of sessions on a site are bots that slipped past basic filters (sample numbers). A behavioral rule catches 95% of those bots and wrongly flags 3% of human sessions.

Out of 100,000 sessions:

Only 1,900 of the 4,840 flagged sessions are bots: 39%. A rule that sounds "95% accurate" is wrong more often than right when it fires, because humans vastly outnumber bots. Deleting every flagged session would remove 2,940 real visitors.

This is why bot handling usually layers signals and keeps borderline cases reviewable rather than dropping them; see how bot detection works and reporting a false positive.

Example 2: "winning" A/B tests

Assume 10% of the ideas a team tests genuinely improve the metric (base rate), tests have 80% statistical power, and the significance level is 5%.

Out of 100 tests:

• 10 real winners × 80% detected = 8 true wins
• 90 non-winners × 5% false positives = 4.5 false wins

So of 12.5 "significant" results, only 64% are real. If the base rate of good ideas is 5%, it falls to about 46%. A significant result is not a 95% guarantee; the share of real effects among significant results depends on how often your ideas work. Peeking and testing many metrics push it lower still; see p-hacking.

Example 3: segment comparisons

"Users who open the pricing page convert at 12%, versus 2% overall, so we should push everyone to pricing." The base rate trap here is subtler: people who choose to visit pricing were already more likely to buy. Compare like with like, and remember that the 12% describes a self-selected group; see confounding variable and correlation vs causation.

How to avoid the base rate fallacy

1. Ask for the base rate first. How common is the thing before any evidence? Write it down.
2. Draw the 2×2 table with real counts instead of percentages. Natural frequencies make the answer obvious.
3. Report precision, not just accuracy. For any detector or prediction, ask: of everything flagged, what share is correct?
4. Expect more false alarms for rare events. The rarer the event, the more precise the signal must be to be useful.
5. Replicate surprising wins. A second test or a holdout period separates real effects from lucky ones.

Base rate fallacy vs related biases

• Representativeness heuristic: judging probability by how much something resembles a stereotype, which often causes base rate neglect.
• Prosecutor's fallacy: confusing P(evidence | innocent) with P(innocent | evidence), a specific form of the same error.
• Simpson's paradox: a different trap, where group mix reverses a trend.

Sources

• Kahneman, D. and Tversky, A. (1973). On the psychology of prediction. Psychological Review, 80(4).
• Bar-Hillel, M. (1980). The base-rate fallacy in probability judgments90046-3). Acta Psychologica, 44(3).
• Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Medicine, 2(8).