TapCub is live — analytics, insights and live chat, free on one platform
Glossary

What is self-selection bias? Product examples, calculations and how to reduce it

Self-selection bias is what happens when people choose their own group, so a gap between groups reflects who chose. Three product examples with numbers: feature users, beta volunteers and chat openers, and how to shrink the bias.

Chen Yu Reviewed by Sophia Bennett Also called self-selection effect Last updated Nov 19, 2026 Glossary

Quick answer

Self-selection bias is a distortion that appears when people put themselves into a group, so the group already differed from those who did not choose it. A gap in retention or conversion then reflects who opted in as well as the effect of the thing you are studying.

What self-selection bias is, and how it differs from nearby biases

Self-selection bias appears when the people in your comparison put themselves into a group instead of being assigned to it. The ones who opted in were already different from the ones who did not, in motivation, need, skill or budget, so any gap between the groups mixes the effect of the thing you are studying with the effect of who chose it.

In product work the classic form is: "users who use feature X retain better, so X improves retention." The people who find and use X are often the ones who were going to stay anyway. The Catalogue of Bias, maintained by Oxford's Centre for Evidence-Based Medicine, describes the same pattern in medicine: in its hormone therapy example, women who chose the treatment were more health-conscious, more active and better off, which confounded the comparison.

Self-selection biasSelection bias (general)Survivorship bias
Who decides who is in the sample?The people themselves (they opt in)Anyone: the researcher, the process, the participantsThe process of dropping out (only survivors remain)
Typical product exampleFeature users vs non-users; beta volunteers; survey respondersAnalyzing only logged-in usersAnalyzing retention of users who are still active today
Why the gap is misleadingThe groups differ before the feature existsThe sample differs from the population you care aboutThose who left are missing from the data
Main fixCompare people with the same starting point; randomizeRedefine the population; weight or widen the sampleInclude everyone who started, not who finished

The wider family is covered in selection bias and survivorship bias. The underlying mechanism, a third factor that drives both the choice and the outcome, is a confounding variable. This page stays on the case where the choice is the user's own.

Example 1: feature users retain better, but who chose the feature?

Sample data: 1,000 users in their first month. 300 used the reports feature at least once; 700 did not. Retention is whether a user was still active 30 days after sign-up.

GroupUsersRetained30-day retention
Used the feature30019063.3%
Did not use it70017525.0%
Gap38.3 points

A 38-point gap looks like proof. Now split everyone by a proxy for intent that exists before the feature: whether the user finished setup (created a project, connected a data source) on day 1.

Day-1 setupGroupUsersRetainedRetention
FinishedUsed the feature26018270.0%
FinishedDid not use it1409165.0%
Not finishedUsed the feature40820.0%
Not finishedDid not use it5608415.0%

Within each setup group the gap is 5 points (70 vs 65, and 20 vs 15). Weighting the two groups by their share of the 1,000 users (400 finished, 600 not) gives 0.4 × 5 + 0.6 × 5 = 5 points. So about 5 of the 38.3 points can be attributed to the feature once setup behavior is held fixed, and the rest, 33.3 points or 87%, comes from who chose it: 260 of 300 feature users had finished setup, against 140 of 700 non-users. Day-1 setup is only a proxy, so even 5 points is an upper estimate of what the feature does; other differences you cannot see remain.

Example 2: beta volunteers

Sample data: 3,000 users are invited to a beta program. 900 opt in; 2,100 do not. In the 90 days after the beta starts, 80% of volunteers are retained (720 of 900) against 50% of the others (1,050 of 2,100). That is a 30-point gap.

The same users have history. In the 90 days before the beta, 78% of the volunteers (702 of 900) and 49% of the others (1,029 of 2,100) were retained: a 29-point gap that existed before anyone saw the beta.

PeriodVolunteersOthersGap
90 days before78%49%29 points
90 days after80%50%30 points
Change in gap+2+1+1 point

The beta moved the gap by 1 point, not 30. Comparing a group with its own earlier behavior removes everything that was already different about the volunteers. It does not remove differences that start changing at the same moment as the beta (a volunteer who changes jobs, for example), but it is far closer than a single after-the-fact comparison.

Example 3: visitors who open live chat

Sample data: 1,800 visitors viewed the pricing page. 200 opened chat; 1,600 did not. Orders: 36 of 200 (18.0%) against 109 of 1,600 (6.8%), an 11.2-point gap. Split by intent, using whether the visitor viewed pricing two or more times:

IntentOpened chatUsersOrdersRate
High (2+ pricing views)Yes1503020.0%
HighNo4506314.0%
Low (1 view)Yes50612.0%
LowNo1,150464.0%

Inside each intent group the gap is 6 and 8 points instead of 11.2. Three quarters of the people who opened chat (150 of 200) were high-intent, against 450 of 1,600 non-chatters (28%). Part of the gap is intent; some may be real. A measured gap after splitting is an estimate to test, not a result to report.

How to reduce self-selection bias

MethodWhat you doWhat it fixesWhat it does not fix
Compare like with likeSplit users by a trait fixed before the choice (source, plan, day-1 behavior, signup week), compare inside each split, combine the resultsDifferences you measuredDifferences you did not measure
Compare before and afterLook at the same people's behavior before and after, and compare the change between groupsStable differences between groupsDifferences that change at the same time as the exposure
Delay the comparisonStart measuring from a fixed point (first week) rather than from "first use"Early dropouts looking like non-usersMotivation
RandomizeAssign the feature at random and compareEverything, if the groups are large enoughNeeds an experiment tool

Two habits help regardless of method. Write down the choice that put people in each group before looking at outcomes, and ask what else would drive that choice. And keep each comparison small enough to read: after three or four splits the groups are too small to trust.

Common mistakes

  • Reading a raw gap as a feature effect. The 38.3-point gap in Example 1 is a description of two groups, not of the feature.
  • Adjusting for something that happens after the feature. Splitting by "completed three sessions" when the feature itself causes sessions removes part of the real effect. Use traits fixed before exposure.
  • Counting the volunteers as the population. A survey, a beta or a feedback widget tells you about people who answered, not about everyone who used the product.
  • Treating one adjustment as proof. Matching on setup behavior (Example 1) still leaves unobserved differences; say so in the report.
  • Splitting until the groups are tiny. With 40 users in one cell (Example 1, not finished), one extra retained user moves the rate 2.5 points.

Self-selection bias in TapCub

Take Example 1 into TapCub. The goal is to find out whether people who use a feature stay longer than comparable people who do not, and how much of the gap survives when you compare like with like.

  1. Record the choice as an event. Send an event when a user first uses the feature (for instance report_viewed) so you can build a rule segment for people who did it and people who did not.
  2. Build the groups as segments. In All-platform analytics › Users › Segments & tags, create rule segments such as "first seen in the last 60 days and did report_viewed" and "first seen in the last 60 days and did not do report_viewed".
  3. Read retention per group. In All-platform analytics › Behavior › Behavior analysis, open Retention with the sign-up as the start and apply one segment at a time. Retention has no breakdown, so you run it once per segment and compare the rates.
  4. Split by something fixed beforehand. Add a second condition to each segment, such as acquisition channel (from Web analytics › Sources), new vs returning, or whether the user did a setup event on day 1, and repeat. If the gap shrinks from 38 points to 5, the first number was mostly who chose the feature.
  5. Know the limit. TapCub does not run A/B experiments or assign users to groups, so it cannot randomize. It shows the gap and how much of it survives a comparison; to settle causation you would run the experiment in your own tool and measure the result here.

Sources

Was this explanation clear?

We use your answers to fix and expand this page.

Chen Yu

Technical Engineer

Focuses on tracking code, SDKs, and API integrations, with practical guidance on implementation checks and troubleshooting.

FAQ

Questions people ask

Short answers to the follow-up questions that come up most.

Still have a question?

Chat with the team behind TapCub — we usually reply within a few hours on business days.

What is an example of self-selection bias?

Users who adopt a feature often retain better because motivated users adopt features. In the sample here, feature users retained at 63.3% against 25.0%, but within groups who finished setup on day 1 the gap was only 5 points, so most of the raw gap came from who chose the feature.

How is self-selection bias different from selection bias?

Selection bias is the broad family: the sample differs from the population for any reason. Self-selection is the case where the people choose to enter, for example by volunteering, opting in or using a feature. Survivorship bias is when only those who stayed remain in the data.

How do you reduce self-selection bias?

Compare people with the same starting point (split by a trait fixed before the choice), compare the same people before and after, start the measurement from a fixed point, or randomize who gets the feature. Splitting handles what you measured, not what you did not.

Can analytics tools remove self-selection bias?

No. Analytics can show the gap and how much of it survives a like-for-like split, but only random assignment removes it. TapCub does not run experiments, so run them in your own tool and measure the results in TapCub.

See this number on your own site

Add one script tag for web and apps, and the first reports fill in within minutes.

See your data clearly. Find your growth.

Every click, backed by data. Install one line of code and see your first numbers in a minute.

No credit card · Free plans for analytics and chat · Cookieless analytics