Inferential statistics: definition, examples and common tests

What is inferential statistics?

Inferential statistics uses data from a sample to draw conclusions about something larger that you have not fully observed: a population, a process, or what will happen next. It always comes with a statement of uncertainty, such as a confidence interval or a p-value.

Descriptive statistics answer "what is in this data?" Inferential statistics answer "what does this data let me say about the world beyond it?" A dashboard showing last month's activation rate is descriptive. Saying the new onboarding flow raised activation, and by roughly how much, is inferential.

The two main tools

A third family, modelling, estimates how several variables relate at once: regression analysis, forecasting, classification. It rests on the same ideas.

Worked example: did activation really improve?

A product changed its onboarding at the start of the month. Sample data:

Estimation. This month's rate is 25.75% with a 95% interval of ±2.14 points, so roughly 23.6% to 27.9%.

Testing the difference. The difference is 3.75 points. Its standard error is √(0.2575 × 0.7425 ÷ 1,600 + 0.22 × 0.78 ÷ 1,500) = 1.53 points, so z = 3.75 ÷ 1.53 = 2.45 and the two-sided p ≈ 0.014. The 95% interval for the difference is +0.75 to +6.75 points.

What we can say: a difference this large would be unlikely if activation had not changed at all, and the plausible improvement ranges from under 1 point to almost 7. What we cannot say from this alone: that onboarding caused it. The two months had different traffic, campaigns and seasonality. To attribute the change, compare like with like (same channels, same weekdays) or run a controlled experiment; see correlation vs causation.

The assumptions behind every inference

1. The sample represents the population. If you sampled only desktop users or only one campaign, no formula fixes it. See sampling bias.
2. Observations are independent. Count users, not repeated events from the same user.
3. The model fits. Normal approximations need enough data; skewed metrics such as revenue may need medians, transformations or bootstrap intervals.
4. You decided the test before looking. Running many tests and reporting the one that worked inflates false positives.

Common errors

• Type I error: concluding there is an effect when there is none (false positive). Its rate is the significance level, often 5%.
• Type II error: missing a real effect (false negative). Its rate depends on sample size and effect size; one minus it is statistical power.

See type 1 vs type 2 error for worked examples.

• Confusing significance with importance. With huge samples, trivial differences become "significant". Always report the size of the difference; see effect size.
• Treating a non-significant result as proof of no effect. It may just mean the sample was too small.

Common inferential tests

Is web analytics data a sample?

If your tool records every visit, last month's numbers are not a sample of last month, so descriptive statistics fully describe it. Inference comes in when you treat that month as evidence about the site in general: whether a change worked, whether a segment really converts better, or what next month will look like. In those cases the month is effectively one sample from an ongoing process, and the same intervals and tests apply.

TapCub reports describe all collected events without sampling. Compare date ranges and segments in the reports, then apply the tests above to the counts, or export raw data and run them in a notebook or with SQL. For a refresher on reading results, see statistical significance.

Sources

• OpenStax. Introductory Statistics, chapters 8–10: confidence intervals and hypothesis testing.
• NIST/SEMATECH. e-Handbook of Statistical Methods: product and process comparisons.
• Wasserstein, R. L. and Lazar, N. A. (2016). The ASA statement on p-values: context, process, and purpose. The American Statistician, 70(2).