TapCub is live — analytics, insights and live chat, free on one platform
Analytics

Causation Examples: What Evidence Separates a Cause From a Coincidence

Examples of causation from a randomized hormone trial, Bing's headline experiment and Facebook's ad studies, plus a worked product-analytics case where a 21.9-point retention gap is really 2 points, and a checklist for what you can claim.

Lin Zhou Published Nov 20, 2026 11 min read Analytics
Sample data: 10,000 users; 3,200 tried the feature. Retention is 47.5% for users and 25.6% for non-users, but engaged users both adopt and stay. Within each group the gap is 2 points, so the raw gap overstates the effect 10.9 times.Sample data

An example of causation is a randomized test: show a new checkout button to a random half of visitors and see higher sales in that half. Random assignment makes the groups alike apart from the change, so the difference can be attributed to it. This guide sets out the evidence ladder, three published cases where experiments and observation disagreed, and a worked example in which a 21.9-point retention gap turns out to be 2 points.

What you will learn

  • Only a randomized test (or a close equivalent) justifies the word cause; adjusted and raw comparisons justify weaker wording
  • In the sample, feature users retained 21.9 points better, but within comparable groups the effect was 2 points, a 10.9x overstatement
  • Check order, third factors, who chose the exposure, chance and mechanism before writing caused

What counts as causation, and what evidence shows it?

Causation means that changing X, with everything else held equal, changes Y. A shop's checkout button turning green and sales rising the same week is a correlation. Showing the green button to a random half of visitors and finding more sales in that half is evidence of causation, because the random split makes the two halves alike in everything except the button.

This guide is about the evidence, not the vocabulary. For the difference between the two ideas and a list of confusing pairs, see correlation vs causation examples and the causation glossary entry. Here we take real cases, one product-analytics example you can recompute, and a short test for telling a cause from a coincidence.

The evidence ladder

EvidenceWhat it rules outExample in a product team
Randomized experimentConfounders, selection, reverse causation (assignment is random and comes first)Show the new onboarding to a random 50% of new sign-ups
Holdout or natural experimentMost confounders, if assignment is as good as randomA feature launched to one country first, a region chosen by a coin flip
Observational data with adjustmentOnly the confounders you measuredComparing feature users with non-users within the same activity level
Raw correlationNothing"Users of feature X retain better"

Moving down the table, the claim you are allowed to make gets weaker. Everything in this guide is about what happens when people use a lower rung and speak as if they stood on the top one.

EvidenceWhat it rules outExample
Randomized experimentConfounders, selection, reverse causationShow new onboarding to a random 50%
Holdout or natural experimentMost confounders if assignment is near randomFeature launched to one country first
Adjusted observational dataOnly the confounders you measuredCompare users and non-users at the same activity level
Raw correlationNothingUsers of feature X retain better

The claim you may make weakens as you move down the table.

Example 1: hormone therapy, randomized trial versus years of observation

The Women's Health Initiative randomized 16,608 healthy postmenopausal women to estrogen plus progestin or placebo. The trial was stopped early, after a mean of 5.2 years, because the treatment increased the risks of coronary heart disease, stroke, pulmonary embolism and invasive breast cancer. The authors concluded that overall risks exceeded benefits and that the regimen should not be started to prevent heart disease (Rossouw et al., JAMA 2002). The trial was run because observational evidence alone had left the balance of risks and benefits uncertain.

What makes this a causal example is the design: women were assigned by chance, so the groups did not differ in health habits, income or access to doctors. That is the property raw comparisons lack.

Example 2: a headline change measured at Bing

A small change to how ad headlines were displayed at Bing sat in the backlog until an engineer ran a quick online controlled experiment. It found a revenue lift of 12%, which would be more than $100 million a year in the United States alone, without hurting key user-experience metrics (Kohavi and Thomke, Harvard Business Review). The book by Kohavi and colleagues adds that Bing runs over 10,000 experiments a year and that a simple change with such a big effect happens only once every few years (Trustworthy Online Controlled Experiments).

Two lessons. First, nobody could have guessed from dashboards that this change mattered; a controlled test measured it. Second, most ideas do nothing, which is why you want the control group.

Example 3: when observational ad measurement disagreed with experiments

Gordon, Zettelmeyer, Bhargava and Chapsky compared observational methods with randomized experiments on the same advertising campaigns. They used 15 US experiments at Facebook with 500 million user-experiment observations, treated the experiments as the benchmark, and found that commonly used observational approaches often failed to measure the true effect of advertising (Marketing Science, 2019). Some studies came close and others did not, and the authors note that even fairly comprehensive data may be insufficient.

The point for marketers is not that observational data is useless. It is that you cannot tell from the observational result alone whether it is one of the close ones.

Example 4: a feature and retention, worked from numbers

Sample data: 10,000 new users, a new "saved views" feature, and 30-day retention. Users who are already engaged are much more likely to try the feature, and they also stay anyway.

GroupUsersTried the feature30-day retention, users30-day retention, non-users
Already engaged2,0001,600 (80%)70%68%
Everyone else8,0001,600 (20%)25%23%
All10,0003,20047.5%25.6%

Compare feature users with non-users across the whole base and the gap is 47.5% − 25.6% = 21.9 points. Compare them within each group and the gap is 2 points in both (70% vs 68%, 25% vs 23%). The raw gap overstates the effect by a factor of 10.9, because engaged users are over-represented among feature users (1,600 of 3,200) and under-represented among non-users (400 of 6,800).

A randomized holdout, where a random 10% of users are not shown the feature, would balance the engaged users across both groups, so the measured difference would be expected to land near 2 points. The correlation in the dashboard is real. The causal claim "the feature lifts retention by 22 points" is not.

app.tapcub.com

30-day retention gap: raw versus within groups

sample data, points
Raw gap, users vs non-users21.9
Within already-engaged users2.0
Within everyone else2.0
Sample data
Sample data: 10,000 users; 3,200 tried the feature. Retention is 47.5% for users and 25.6% for non-users, but engaged users both adopt and stay. Within each group the gap is 2 points, so the raw gap overstates the effect 10.9 times.

When you cannot randomize: Hill's nine viewpoints

When randomized trials are impossible, evidence is built from several directions. Austin Bradford Hill listed nine "viewpoints" to ask of an association in his 1965 address, and he stressed that none can bring indisputable proof and none is a required checklist item (Hill, Proceedings of the Royal Society of Medicine, 1965). They translate well to product questions:

Hill's viewpointQuestion for a product team
StrengthHow large is the effect compared with ordinary noise?
ConsistencyDoes it show up in different segments, countries and months?
SpecificityDoes the feature move the metric it should, and not everything?
TemporalityDid usage start before retention improved?
Biological gradientDo heavier users of the feature retain better?
PlausibilityCan you explain the mechanism in one sentence?
CoherenceDoes it fit what you already know about your users?
ExperimentHas anyone actually withheld the feature and measured?
AnalogyDo similar changes elsewhere behave this way?

Five "yes" answers from observational data build a case; they do not replace the "experiment" row.

A quick test for cause versus coincidence

Before you write "caused", work through these checks in order:

  1. Order. Did X happen before Y? If Y came first, you may have reverse causation (churning users stop opening the app, so "less usage predicts churn" is partly the churn itself).
  2. Third factor. Is there something that makes both X and Y more likely? Engaged users in Example 4 are the pattern.
  3. Who picked X? If people chose the feature themselves, assume selection until a random assignment says otherwise.
  4. Chance. How many comparisons did you run before finding this one? Many tests make a lucky result likely, and a correction such as Bonferroni shows how much.
  5. Dose. Does more of X go with more of Y?
  6. Replication. Does it hold in a second period or segment?
  7. Mechanism. Can you tell the story of how X moves Y in one sentence?

If checks 1–3 fail, stop. The remaining four only matter when the first three pass.

Words to match the evidence

Evidence you haveSafe wordingAvoid
Randomized test, enough sample"The change increased conversion by X points (95% interval …)""Proves" without an interval
Holdout or natural experiment"The change likely increased …"Overstating how random the split was
Adjusted observational"Is associated with, after adjusting for …""Causes", "drives"
Raw comparison"Users who did X had Y"Any forward-looking claim ("will lift")

Dashboards, chat transcripts and slide decks are where wording drifts; fix it at the source.

What TapCub can and cannot tell you about cause

Use the saved-views example. TapCub shows who used the feature and who stayed; the causal step is a design decision outside it.

  1. See the raw gap. Build a retention report in All-platform analytics › Behavior › Behavior analysis for people who did the feature event and those who did not. This reproduces the 47.5% versus 25.6% comparison and nothing more.
  2. Compare within activity levels. Add a segment or breakdown for already-engaged users from All-platform analytics › Users › Segments & tags and repeat. A gap that shrinks is a sign of a confounder.
  3. Mark the launch. Add a date annotation under Configuration › Annotations so launches, campaigns and price changes sit on the chart next to the metric.
  4. Treat explanations as explanations. All-platform analytics › Operations › AI Assistant has a change-attribution view that breaks a metric change down by channel, referrer domain, country, device and landing page. It shows where a change happened, not why.
  5. Read chat conversion carefully. Live chat › Reports › Conversion attribution compares visitors who chatted with those who did not, and shows relative lift. It warns when fewer than 30 visitors chatted. People who start a chat are often already closer to buying, so the lift is an upper bound, not a causal estimate.

TapCub does not run experiments, split traffic or hold out users. Assign the holdout in your own code, send the group as an event property, and use TapCub to compare outcomes.

FAQ

What is an example of causation?

Giving a random half of visitors a new checkout button and seeing higher sales in that half is causal evidence, because random assignment makes the groups alike apart from the button. A hormone-therapy trial that randomized 16,608 women and found higher stroke and breast cancer risk is another.

How do you prove causation?

Randomize who gets the change and compare outcomes. When you cannot, combine several lines of evidence: the cause precedes the effect, the effect grows with exposure, it holds across groups, and a mechanism makes sense. Hill's 1965 viewpoints are the classic list, and none is a guarantee.

What is the difference between correlation and causation?

Correlation means two measures move together. Causation means changing one moves the other. The feature-and-retention example shows the gap: users of the feature retained 21.9 points better, yet within comparable groups the difference was 2 points, because engaged users both adopt and stay.

Can observational data show causation?

Sometimes, with care. Adjusting for known confounders and checking consistency and dose-response can strengthen a case, but unmeasured factors stay unaddressed. In Facebook's 15 advertising experiments, common observational methods often failed to match the randomized result, which is why a holdout is worth running.

Does a significant p-value prove causation?

No. A p-value tells you how surprising the data would be if there were no effect. It says nothing about why two things move together. A significant difference between self-selected groups can come entirely from who chose to be in each group.

Sources

Filed underAnalytics

Was this article helpful?

We use your answers to fix and expand this page.

Lin Zhou

Product Experience Specialist

Writes about product experience, practical workflows, and growth, turning complex processes into clear steps for everyday decisions.

See the raw gap, then test it

Compare retention for people who did and did not use a feature, mark launches on the chart and send your holdout group as an event property.

See your data clearly. Find your growth.

Every click, backed by data. Install one line of code and see your first numbers in a minute.

No credit card · Free plans for analytics and chat · Cookieless analytics