Quick answer
A t-test is a statistical test that compares a mean, or the difference between two means, with the spread of the data to judge whether the difference is likely to be more than random noise.
Formula: Welch: t = (mean₁ − mean₂) / √(s₁²/n₁ + s₂²/n₂)
What is a t-test?
A t-test compares averages. It asks whether the difference between two means, or between one mean and a target, is large compared with how much the data varies. The result is a t statistic and a p-value.
It is called "t" because the test statistic follows Student's t distribution, published by William Gosset under the pen name "Student" in 1908.
The three types
| Type | Compares | Example |
|---|---|---|
| One-sample | One mean with a fixed value | Is average session duration different from 120 seconds? |
| Two-sample (independent) | Means of two separate groups | Average order value in variant A vs variant B |
| Paired | Before and after for the same subjects | Pages per session for the same users before and after a redesign |
For two independent groups, use Welch's t-test by default. It does not assume the two groups have equal variance.
The formula
For Welch's t-test:
t = (mean₁ − mean₂) / √(s₁²/n₁ + s₂²/n₂)
where s is each group's standard deviation and n its size. The denominator is the standard error of the difference. Compare t with the t distribution at the right degrees of freedom to get the p-value.
A worked example
Average order value in two variants of a checkout page (sample numbers):
| Variant A | Variant B | |
|---|---|---|
| Orders | 400 | 400 |
| Mean order value | $52.0 | $48.0 |
| Standard deviation | $20 | $22 |
- Difference: 52.0 − 48.0 = 4.0
- Standard error: √(20²/400 + 22²/400) = √(1.00 + 1.21) = 1.487
- t = 4.0 / 1.487 = 2.69, with about 790 degrees of freedom
- Two-sided p ≈ 0.007
At a 0.05 threshold the difference is statistically significant. The 95% confidence interval is about 4.0 ± 2.9, so $1.1 to $6.9.
Assumptions and when to use something else
- Independent observations. One row per order or user, not repeated events from the same person counted as separate samples.
- Roughly normal means. With a few hundred values per group the mean is close to normal even if the data is skewed. Very heavy tails (a few huge orders) still distort it; consider trimming or winsorizing, or a Mann-Whitney test.
- Yes/no outcomes. For conversion rates use a two-proportion test.
- Many comparisons. Running a t-test on every segment inflates false positives; adjust with a Bonferroni correction.
Getting the inputs from TapCub
TapCub does not run A/B tests or calculate t-tests for you. What it gives you are the inputs: build a segment for each group, then use a distribution analysis on a numeric property (such as order amount) to see each group's mean, median and spread, and the sample size. For the full calculation, export the raw rows and compute the standard deviation in a spreadsheet or notebook.
Was this explanation clear?
We use your answers to fix and expand this page.
Couldn’t send. Please try again in a minute.
Thanks for the feedback. We’ll use it to improve this page.You’ve already sent feedback on this page. Thanks.
Product Analyst
Focuses on event analysis, conversion funnels, and retention, using clear metric definitions and structured analysis to explain product performance.