TapCub is live — analytics, insights and live chat, free on one platform
Analytics

Confounding Variables: Examples and How to Spot Them in Your Data

Three recomputable examples of confounding variables from product and marketing data: a feature that seems to double retention, an email lift that mostly vanishes and a channel that wins on every device yet loses overall. Plus how to split, standardise and avoid common mistakes.

Chen Yu Published Nov 18, 2026 12 min read Analytics
Sample data: paid converts 2.8% against organic's 4.3% overall, yet beats organic on mobile (2.0% vs 1.5%) and on desktop (6.0% vs 5.0%). Paid traffic is 80% mobile, which converts at a third of desktop's rate or less.Sample data

A confounding variable is a third factor that differs between the groups you compare and also affects the outcome, so it makes a gap look like cause and effect. The fix is one move: split the result by the suspect (channel, device, new or returning, plan) and check whether the gap survives inside each slice. Below are three recomputable examples and the methods that follow.

What you will learn

  • Split any gap by the usual suspects before believing it: channel, device, new vs returning, plan, country, signup date
  • A confounder can shrink, erase or even reverse an effect, so compare the mix of each group, not only the rates
  • Only randomising covers confounders you did not measure; otherwise say what you adjusted for, not that you proved a cause

What has to be true for a variable to confound?

A variable confounds a comparison when three things hold at once:

  1. It differs between the groups you compare. Feature users are mostly on the paid plan; non-users are mostly on the free plan.
  2. It affects the outcome on its own. Paid users retain better whether or not they touch the feature.
  3. It is not a step between the cause and the effect. Plan was chosen before the feature was used; it is not something the feature produced.

When all three hold, part of the gap you see belongs to the third variable, and the comparison credits it to the thing you were looking at. Statisticians call this a failure of exchangeability: the two groups would have behaved differently even if the feature had not existed. The formal treatment is in Greenland, Robins and Pearl (1999); the plain-language definition is already covered in confounding variable, so this guide spends its time on examples you can recompute.

The practical test is short: take the gap, split it by the suspect variable, and see whether it survives inside each slice. Everything below is that one move, repeated.

Example 1: people who use a feature retain better

Sample data: 10,000 new users, 2,000 of whom used a new "saved views" feature in their first week. Day-30 retention looks like this:

GroupUsersRetained at day 30Retention
Used the feature2,0001,04052%
Did not use it8,0001,92024%

The gap is 28 points, and the natural reading is that the feature doubles retention. Now split by plan, the obvious suspect, since paying customers are both more likely to explore features and more likely to stay:

PlanGroupUsersRetainedRetention
PaidUsed the feature1,60096060%
PaidDid not use it80048060%
FreeUsed the feature4008020%
FreeDid not use it7,2001,44020%

Inside each plan the gap is zero. Retention is 60% for paid users and 20% for free users, with or without the feature. The 28-point gap came entirely from the mix: 80% of feature users are paid, against 10% of non-users (800 of 8,000). The mix is exactly what the checklist above describes: the variable differs between the groups, it drives the outcome, and it came first.

That is the extreme case, chosen so the arithmetic is easy to follow. Real data rarely collapses to zero; more often the gap shrinks to a fraction of its raw size. The habit that matters is the same: never report the raw gap until you have seen the sliced one.

Example 2: visitors from email convert better

Sample data: 20,000 sessions in a month, 2,000 of them from an email campaign. The raw numbers:

SourceSessionsOrdersConversion rate
Email2,0001547.7%
Everything else18,0005403.0%

Email converts at 2.57 times the rate of everything else. Before moving budget, split by new versus returning visitors. Email lists contain people who already bought, so they are mostly returning visitors.

Visitor typeSourceSessionsOrdersRate
ReturningEmail1,6001449.0%
ReturningEverything else3,0002408.0%
NewEmail400102.5%
NewEverything else15,0003002.0%

Email still wins in both slices, but the lead is 1 point among returning visitors and half a point among new ones, nowhere near 4.7 points. To put one number on it, ask what the non-email group would have converted at if it had email's visitor mix (80% returning, 20% new): 0.8 × 8.0% + 0.2 × 2.0% = 6.8%. Against that baseline, email's 7.7% is a lift of 0.9 points, not 4.7. This is called direct standardisation, and it is the cheapest way to compare groups with different compositions.

The campaign did not do nothing. It did about a fifth of what the raw comparison suggested, and the rest was the audience it was sent to.

Sample data: 10,000 paid sessions and 10,000 organic sessions.

ChannelOverall rateMobileDesktop
Paid2.8% (280 of 10,000)2.0% (160 of 8,000)6.0% (120 of 2,000)
Organic4.3% (430 of 10,000)1.5% (30 of 2,000)5.0% (400 of 8,000)

Overall, organic beats paid by 1.5 points. On mobile, paid beats organic (2.0% against 1.5%). On desktop, paid beats organic (6.0% against 5.0%). The channel that looks worse is better in every slice, because 80% of paid sessions are on mobile, which converts at a third of desktop's rate or less in both channels. This reversal is Simpson's paradox, and it is the most dramatic form of confounding; the pattern is explained further in Simpson's paradox. Bickel, Hammel and O'Connell (1975) analysed a real case in university admissions, where an apparent bias in the combined figures largely disappeared once applications were compared department by department.

If you had cut the paid budget on the strength of the overall numbers, you would have cut the channel that performs better on both devices. The right comparison is by device, or standardised to a common mix: at 50/50 mobile and desktop, paid is 4.0% and organic 3.25%.

app.tapcub.com

Conversion rate, paid vs organic

sample sessions, %
Organic, overall4.3%
Paid, overall2.8%
Paid, mobile2.0%
Organic, mobile1.5%
Paid, desktop6.0%
Organic, desktop5.0%
Sample data
Sample data: paid converts 2.8% against organic's 4.3% overall, yet beats organic on mobile (2.0% vs 1.5%) and on desktop (6.0% vs 5.0%). Paid traffic is 80% mobile, which converts at a third of desktop's rate or less.

How do you spot a confounder before it misleads you?

Three checks, in the order they pay off:

  • Ask who ends up in each group. Any comparison between people who chose something (used a feature, opened an email, clicked an ad) has a selection step. If people sorted themselves, assume they differ on something else too. See selection bias.
  • Split by the usual suspects. In product and marketing data the list is short: acquisition channel, device, new versus returning, plan or customer size, country, and signup date (seasonality and releases). If the gap survives each split, it is far more believable.
  • Compare the mix, not just the rates. If the shares of mobile, returning or paid users differ a lot between the groups, you have found the candidate. In Example 1 the mix was 80% against 10%.

A gap that changes a lot when you split, shrinks, vanishes or flips, is confounded. A gap that holds roughly steady in every slice is not proof of cause, but you have removed the most common explanation.

What can you do about it?

MethodWhat it doesWorks whenWeak spot
StratifyCompare inside slices (plan, device) and report eachFew confounders, enough users per sliceSlices get thin fast
StandardiseReweight to one common mix and compare ratesYou want a single comparable numberHides differences between slices
MatchPair each treated user with a similar untreated oneMany confounders you have measuredOnly handles what you measured
RandomiseAssign the treatment by chanceYou control the treatmentNeeds the ability to run an experiment

Randomisation is the only method that also handles confounders you did not think of or cannot measure, which is why it is the standard for causal claims. Propensity-score methods (Rosenbaum and Rubin, 1983) are a formal way to do matching, but they too can only adjust for variables that are recorded. If you cannot run an experiment, the honest wording is "after accounting for plan and device, feature users retained X points better", not "the feature raised retention by X points". For the wider causal question see correlation vs causation.

MethodWhat it doesWeak spot
StratifyCompare inside slices such as plan or deviceSlices get thin fast
StandardiseReweight groups to one common mixHides differences between slices
MatchPair similar treated and untreated usersOnly handles measured variables
RandomiseAssign the treatment by chanceNeeds the ability to run an experiment

Randomisation is the only method that also covers variables you did not measure.

Which mistakes make confounding worse?

  • Controlling for something the treatment causes. If the feature leads people to log in more, and you adjust for login frequency, you remove part of the real effect. Only adjust for variables that were fixed before the treatment (plan at signup, channel, device, country).
  • Slicing until nothing is left. Splitting by plan, device, country and week gives cells with five users each. Rates from tiny cells swing wildly. Start with one or two splits and check the counts.
  • Treating "I adjusted for it" as "I removed it". You removed the part you measured. Say what you did not measure.
  • Assuming a confounder always shrinks the effect. It can hide a real effect or reverse its sign, as in Example 3.
  • Looking only for the hidden variable. The confounder is often in plain sight in your own table (device, channel); it only needs a split.

How do you do these splits in TapCub?

The examples above are the three questions teams most often ask of an analytics tool. Here is how the device example runs in TapCub, using only breakdown and segment features that exist today.

  1. Start with the raw comparison. In All-platform analytics › Behavior › Behavior analysis, build a funnel from landing to order and look at it by channel. Funnel breakdown uses one layer and keeps the 10 groups with the most entries, so make that layer the channel.
  2. Add the suspect variable. In All-platform analytics › Behavior › Event analysis, count the order event with a breakdown of up to three layers, such as channel then device then new or returning visitor. Compare the conversion rate by channel inside each device, as in the table above.
  3. Retention needs a different route. Retention analysis has no breakdown, so build one segment per slice in All-platform analytics › Users › Segments & tags (for example "mobile") and apply each in turn, or use a filter. Anonymous visitors never join segments, so this works for identified users only.
  4. Ask what moved. When a number changed between two periods, the AI assistant's "why did it change" view in All-platform analytics › Operations › AI Assistant splits channel, referrer domain, country, device and entry page, and shows each value's change divided by the total change. It shows which dimension carried the movement; it does not say whether anything caused it.
  5. Testing the cause is outside TapCub. TapCub does not run experiments or split traffic; it only measures. Run the randomised test in your own tooling, and bring the results back as events to read by segment.

FAQ

What is an example of a confounding variable?

Plan type in a feature-retention comparison. Paid users both use more features and stay longer for other reasons, so comparing feature users with non-users mixes the feature's effect with the plan's. In the sample above, a 28-point retention gap fell to zero inside each plan.

What are confounding variables in simple terms?

A confounding variable is a third factor that is linked to both the thing you think is causing a result and the result itself, so it creates or hides a link between them. Check for one by splitting the data by that factor.

How is a confounding variable different from a lurking variable?

Many sources use the terms interchangeably. A lurking variable is usually one nobody measured; a confounder is any variable that distorts the comparison, measured or not. The fix differs: measured confounders can be adjusted for, unmeasured ones need randomisation.

Can a confounder make a real effect disappear?

Yes. A confounder can inflate an effect, hide it or reverse its direction. In the paid-versus-organic example, device mix pushed paid to the bottom overall even though it won on both devices.

How do you control for confounding variables?

Stratify (compare inside slices), standardise (reweight to a common mix), match similar users, or randomise assignment. Randomisation is the only one that also covers variables you did not measure.

See also: Confounding vs Lurking Variable: The Difference, With Worked Examples.

Sources

Filed underAnalytics

Was this article helpful?

We use your answers to fix and expand this page.

Chen Yu

Technical Engineer

Focuses on tracking code, SDKs, and API integrations, with practical guidance on implementation checks and troubleshooting.

Split any result by channel, device or visitor type

Breakdowns, segments and a why-did-it-change view help you check whether a gap survives inside each slice.

See your data clearly. Find your growth.

Every click, backed by data. Install one line of code and see your first numbers in a minute.

No credit card · Free plans for analytics and chat · Cookieless analytics