What is survivorship bias? Definition and examples in analytics

What is survivorship bias?

Survivorship bias is drawing conclusions from only the cases that made it through some filter, while the ones that dropped out are missing from the data. Because you never see the failures, the traits of the survivors look like the reason they survived.

The best-known example comes from WWII. Analysts mapped bullet holes on bombers that returned and proposed adding armour where the holes clustered. Statistician Abraham Wald argued the opposite: the planes hit in the engines never came back, so the areas with no holes on the survivors were the vulnerable ones.

Four places it shows up in analytics

1. "What do our best customers do?" You study power users and find most of them use integrations. But many churned users also tried integrations; you just don't look at them.
2. Surveys and NPS. You survey current customers. People who left over price or bugs are not on the list, so the score looks better than the product is.
3. Top content. Old blog posts that still get traffic all have long titles, so you conclude long titles work. The long-titled posts that got no traffic were pruned or never ranked.
4. Case studies and playbooks. "Every unicorn did X" ignores the companies that did X and failed.

A sample cohort example

A team looks at users still active 60 days after signup (sample numbers):

Looking only at survivors, "72% used feature A" sounds like a strong signal. The full cohort tells you more: of the 300 who used feature A, 130 stayed (43%); of the 700 who did not, 50 stayed (7%). That gap is the real comparison, and it still needs a check for confounding variables before you call it causal.

How to correct for survivorship bias

• Start from the beginning of the funnel. Define the group at entry (every signup, every visitor from a campaign), not at the end.
• Keep the drop-outs. Build a group for people who churned or never activated and run the same analysis on them.
• Compare rates, not shares of winners. Ask "what share of people who did X stayed?" instead of "what share of people who stayed did X?".
• Watch for deleted or filtered data. Pruned pages, archived campaigns and expired data retention all remove failures quietly.

Checking it in TapCub

A retention analysis starts from a cohort at its start date and shows how many are still active each period, so the people who left stay in the denominator. In Segments, build one rule segment for users who did your key action in week 1 and one for users who did not, then compare their retention. Segments can also use "did not" conditions, which makes a churned-users segment easy to keep. See also correlation vs causation.