Quick answer
Selection bias is a distortion that happens when the people or events included in an analysis differ systematically from the population you want to draw conclusions about, because of how they were chosen or chose themselves.
What is selection bias?
Selection bias happens when the group you analyse differs systematically from the group you want to draw conclusions about. The difference comes from how people ended up in the data: they were picked in a skewed way, they chose themselves, or some of them dropped out.
It does not shrink with more data. A million self-selected users are still self-selected.
Five types, with analytics examples
| Type | What happens | Analytics example |
|---|---|---|
| Self-selection | People choose their own group | Users who turn on a feature are compared with those who did not |
| Sampling bias | The sampling method skips part of the population | A survey shown only on desktop |
| Non-response | Some people do not answer | Only very happy and very angry customers fill in the NPS survey |
| Attrition | People drop out unevenly over time | Long trials lose low-intent users, so later weeks look better |
| Survivorship | Only the cases that made it are left | Studying only customers who stayed |
There is also a quiet one built into tracking: coverage bias. Visitors with ad blockers, strict browsers or no consent may be missing from client-side data, and they are not a random slice of your audience.
A sample example: the newsletter effect
A team compares purchase rates over 90 days (sample numbers):
| Group | Purchase rate |
|---|---|
| Newsletter subscribers | 9.0% |
| Non-subscribers | 2.5% |
That looks like a 3.6× lift from the newsletter. But people who subscribe were already more interested. Looking at the same subscribers in the 90 days before they subscribed:
| Group | Before subscribing | After |
|---|---|---|
| Subscribers | 6.5% | 9.0% |
| Non-subscribers (same periods) | 2.4% | 2.5% |
Most of the gap existed before the newsletter. The change linked to subscribing is closer to +2.5 points for subscribers against +0.1 for everyone else, and even that may include other differences, so check for confounding variables.
How to reduce selection bias
- Randomise when you can. A random holdout removes self-selection.
- Compare before and after for the same people, against a group that did not change.
- Define the population at entry, not at the end of the process.
- Sample deliberately. Use stratified sampling so small groups are represented in surveys and interviews.
- Close tracking gaps. Send key business events (signups, orders) from your server so they are counted even when a browser blocks scripts.
In TapCub
TapCub does not bypass ad blockers or Do Not Track, so browsers that block the script are missing from website reports. The default website setup is anonymous with no cookie, which avoids the consent-driven gaps that cookie banners can create; identified mode, by contrast, writes a device id only after consent. Send purchases and signups as server-side events so totals do not depend on the browser. To compare a feature against a fair baseline, build a rule segment with a random bucket condition as a holdout, and use the period comparison to look at the same people before and after.
Was this explanation clear?
We use your answers to fix and expand this page.
Couldn’t send. Please try again in a minute.
Thanks for the feedback. We’ll use it to improve this page.You’ve already sent feedback on this page. Thanks.
Data Analyst
Explores traffic sources, conversion funnels, and retention, using clear metric definitions and concrete examples to explain what the numbers mean.