Quick answer
Sampling bias is a systematic error that occurs when the method of collecting a sample makes some members of the population more likely to be included than others, so the sample does not represent the population and estimates are skewed regardless of sample size.
What is sampling bias?
Sampling bias happens when the way a sample is collected makes some members of the population more or less likely to be included than others, so the sample does not represent the population. Any estimate from that sample (an average, a rate, a share) is then systematically off, and collecting more data the same way does not fix it.
That last point is what separates bias from noise. Random sampling error shrinks as the sample grows; sampling bias does not. A survey of 50,000 people who all chose to answer is still a survey of people who chose to answer.
Sampling bias vs selection bias
The terms overlap and are often used interchangeably. A useful distinction:
- Sampling bias is about who gets into the data: the sample does not reflect the population you want to describe. It threatens external validity (does the result generalize?).
- Selection bias is broader and includes how people are assigned to the groups you compare, for example users who opt into a feature being different from users who do not. It threatens internal validity (is the comparison fair?).
Common types of sampling bias
| Type | What goes wrong | Analytics example |
|---|---|---|
| Undercoverage | Part of the population cannot be sampled at all | Visitors whose browsers block the analytics script never appear |
| Self-selection (voluntary response) | People choose whether to be in the sample | Feedback widgets answered mostly by very happy or very angry users |
| Nonresponse | Some people sampled do not respond, and they differ from those who do | Exit surveys skipped by visitors in a hurry |
| Convenience sampling | Only easy-to-reach units are measured | User interviews only with customers on the highest plan |
| Survivorship bias | Only the ones that "survived" a process remain | Analyzing only retained users to learn why users stay |
| Time-window bias | The period sampled is unusual | Benchmarking conversion during a holiday sale |
Worked example: consent-based tracking
Suppose a site only records analytics for visitors who accept a cookie banner (sample data):
- 55% of visitors accept; they convert at 3.5%.
- 45% decline; they convert at 1.8%.
- The true conversion rate of all visitors is 0.55 × 3.5% + 0.45 × 1.8% = 2.74%.
- The dashboard, which only sees acceptors, reports 3.5%, about 28% too high.
Now acceptance rates change. If a new banner design lowers acceptance to 40% while each group converts as before, the true rate falls to 2.48%, but the dashboard still shows 3.5%. The bias is invisible from inside the data because the excluded visitors are, by definition, not in it.
Sources of sampling bias in web and app analytics
- Script and tracker blocking. Ad blockers and privacy browsers block many analytics scripts. Blocking rates differ by audience: developer and gaming audiences block far more than general consumers. See do ad blockers block TapCub.
- Consent-dependent tracking. When tracking waits for consent, only consenting visitors are counted, and they differ from those who decline.
- Cookie expiry and deletion. Returning visitors whose identifiers were cleared show up as new, biasing new vs returning ratios.
- Logged-in only analysis. Product analytics on signed-in users excludes everyone who never signed up.
- Sampled reports. Tools that sample large date ranges can over- or under-represent small segments; see data sampling.
- Bots. Unfiltered automated traffic adds a population you did not mean to measure.
How to detect sampling bias
- Compare against an independent source. Server logs, payment records or CRM counts should roughly match analytics trends. A widening gap means coverage is changing.
- Check the sample's composition. Compare device, country or plan mix in the sample with what you know about the whole population.
- Watch coverage over time. A sudden step change in sessions with no change in orders often means tracking coverage changed, not demand.
- Look at who is missing. Ask which visitors could not be recorded, and whether they are likely to behave differently.
How to reduce it
- Measure without depending on optional storage where you can. Counting that does not need cookies is not affected by banner acceptance; see cookieless analytics.
- Send key business events server-side (orders, sign-ups), so they are recorded even when the browser blocks scripts.
- Use stratified or weighted estimates when you know the population mix; see stratified sampling.
- Sample randomly for surveys and interviews instead of asking whoever is easiest to reach.
- State the population. "Conversion among consenting desktop visitors" is honest; "conversion" is not.
In TapCub
TapCub's default website counting is anonymous and writes no cookie or local storage, so visit counts do not depend on whether visitors accept a cookie banner (regional presets that wait for consent are the exception). Server-side events through the API are recorded regardless of browser blocking. Bot filtering removes known automated traffic before reports. Reports use all collected events rather than a sample. None of this removes bias from people who block the script entirely; compare with server-side order counts to see the size of that gap.
Sources
- Wikipedia. Sampling bias.
- Lane, D. M. et al. Online Statistics Education: sampling bias. Rice University.
- Pew Research Center. U.S. surveys: frequently asked questions.
Was this explanation clear?
We use your answers to fix and expand this page.
Couldn’t send. Please try again in a minute.
Thanks for the feedback. We’ll use it to improve this page.You’ve already sent feedback on this page. Thanks.
Data Analyst
Explores traffic sources, conversion funnels, and retention, using clear metric definitions and concrete examples to explain what the numbers mean.