TapCub is live — analytics, insights and live chat, free on one platform
Glossary

What is a test statistic? Definition, formulas and worked examples

A test statistic is one number that measures how far your data are from the null hypothesis in standard-error units. Formulas for z, t and chi-square, with two worked examples.

Emma Collins Reviewed by Chloe Morgan Also called test stat Last updated Nov 21, 2026 Glossary

Quick answer

A test statistic is one number computed from sample data that measures how far the data sit from what the null hypothesis predicts. A z of 2 means the observed difference is about two standard errors from zero. It is compared with a critical value or turned into a p-value.

Formula: (observed − value under H0) ÷ standard error; two-proportion z = (p̂1 − p̂2) ÷ √(p̂(1 − p̂)(1/n1 + 1/n2))

What a test statistic is

A test statistic is one number computed from your sample that tells you how far the data sit from what the null hypothesis predicts, measured in units of the noise you would expect from sampling alone. The null hypothesis is the "nothing is going on" claim, such as "the two versions convert equally". If the test statistic is small, the data look like chance. If it is large, chance is a poor explanation.

It sits between the raw data and the decision. You collect counts or measurements, reduce them to one number, and then compare that number with a critical value or convert it into a p-value. Every standard test has its own statistic, but they share one idea: signal divided by noise. For how the cut-off is chosen and what "significant" means in practice, see statistical significance.

The general form: signal divided by noise

Most test statistics follow this pattern:

test statistic = (what you observed − what the null hypothesis predicts) ÷ standard error

The numerator is the signal: a difference between two rates, or between a sample mean and a target. The denominator is the standard error: how much that difference would wobble if you repeated the sampling. A result is impressive when the signal is large compared with the wobble, so a test statistic of 2 means the observed value is two standard errors away from what the null predicts.

Dividing by the standard error is also what makes results from different metrics comparable: a z of 2 for a conversion rate and a z of 2 for average order value carry the same weight of evidence, even though one is a percentage and the other is money.

Two consequences are worth remembering. The same difference gives a bigger statistic with more data, because the standard error shrinks as the sample grows. And the statistic has no units, which lets you compare results across different metrics.

Common test statistics at a glance

StatisticUsed forFormula (short)Compare with
z (one proportion)One rate against a target rate(p̂ − p0) ÷ √(p0(1 − p0)/n)Normal distribution
z (two proportions)Two rates, for example conversion A and B(p̂1 − p̂2) ÷ √(p̂(1 − p̂)(1/n1 + 1/n2))Normal distribution
t (one sample)A mean against a target, standard deviation unknown(Ȳ − μ0) ÷ (s/√n)t distribution with n − 1 degrees of freedom
Chi-squareCounts in categories against expected countsΣ (observed − expected)² ÷ expectedChi-square distribution
FVariances, or several group means at onceRatio of two variancesF distribution

The NIST/SEMATECH e-Handbook gives the two-proportion form with the pooled rate p̂ = (x1 + x2) ÷ (n1 + n2), and the one-sample t form T = (Ȳ − μ0) ÷ (s/√N) with N − 1 degrees of freedom.

Worked example: a two-proportion z statistic

Sample data: two versions of a sign-up page, 4,000 visitors each. Version A had 200 sign-ups (5.0%); version B had 244 (6.1%). The null hypothesis: both versions have the same true sign-up rate.

  1. Observed difference: 0.061 − 0.050 = 0.011
  2. Pooled rate: (200 + 244) ÷ 8,000 = 444 ÷ 8,000 = 0.0555
  3. Standard error: √(0.0555 × 0.9445 × (1/4,000 + 1/4,000)) = √(0.0000262) = 0.00512
  4. Test statistic: z = 0.011 ÷ 0.00512 = 2.15

A z of 2.15 means the difference is about 2.15 standard errors from zero. The two-sided p-value is 0.032. At the usual 5% level the critical value is 1.96, and 2.15 is beyond it, so you reject the null.

Now keep the same rates but double the sample to 8,000 visitors per version (400 and 488 sign-ups). The difference is still 1.1 points, but the standard error falls to 0.00362 and z = 3.04, p = 0.002. The effect did not change; the evidence did.

Worked example: a one-sample t statistic

Sample data: your sign-up rate has averaged 3.0% a day. Over the 8 days after a change it was 2.9, 3.4, 3.1, 3.8, 3.5, 3.0, 3.6 and 3.3 percent. The null hypothesis: the true mean is still 3.0%.

  1. Sample mean: 3.325
  2. Sample standard deviation: 0.3105
  3. Standard error: 0.3105 ÷ √8 = 0.1098
  4. Test statistic: t = (3.325 − 3.0) ÷ 0.1098 = 2.96, with 7 degrees of freedom

The critical value for a two-sided 5% test with 7 degrees of freedom is 2.365. Since 2.96 is larger, reject the null; the two-sided p-value is 0.021. A t statistic is used here instead of z because the standard deviation was estimated from only eight days. The t distribution has heavier tails to account for that.

One statistic, two ways to decide

Once you have the statistic, you decide in one of two equivalent ways:

  • Critical value. Choose a significance level α in advance (commonly 0.05). Look up the cut-off for your test and reject the null if the statistic is beyond it. For a two-sided z test at 5%, that is |z| > 1.96.
  • P-value. Convert the statistic into the probability of seeing a result at least this extreme if the null were true. Reject if it is below α.

They always agree. In the first example z = 2.15 exceeds 1.96, and the p-value of 0.032 is below 0.05. In a one-sided test the cut-off moves: it is 1.645 at 5%. Decide one-sided or two-sided before you look at the data, because choosing afterwards inflates your false positive rate.

A test statistic is not an effect size

A large statistic says the evidence for some difference is strong. It does not say the difference is large. Take the sign-up example at 40,000 visitors per version, with 2,000 and 2,440 sign-ups: the rates are 5.0% and 6.1%, the same 1.1-point gap, and z = 6.8. The statistic is more than three times bigger than at 4,000 visitors, yet the effect is exactly the same size.

So report both: the effect (the 1.1-point difference, with its confidence interval) and the statistic or p-value. Decisions about whether to ship a change depend on the effect; whether the effect is real depends on the statistic.

Common mistakes with test statistics

  • Using z when the sample is tiny. The normal approximation for proportions needs enough events in each group. The NIST handbook recommends an exact test for small samples. In the same sign-up example with only 400 visitors each (20 and 24 sign-ups), z = 0.62 and p = 0.54: far from significant, and too few events to trust the approximation anyway.
  • Peeking. Checking the statistic every day and stopping when it crosses 1.96 produces false positives much more often than 5%. Fix the sample size beforehand.
  • Treating 1.96 as a boundary of truth. A z of 1.9 and a z of 2.0 are nearly the same evidence.
  • Forgetting that the null can be wrong in the other direction. A two-sided test checks for a difference either way. If a change could only help, say so before the test.
  • Mixing up the unit. Visitors and sessions are different denominators, and the standard error depends on the one you count.

Test statistics in TapCub

TapCub does not run experiments or compute test statistics. What it provides is the counts that go into the formula.

  1. Counts per group. In Web analytics › Goals, read visitors and conversions for a goal; in All-platform analytics › Behavior › Behavior analysis, a funnel with a breakdown gives entered and converted counts per group, such as the two versions in the example.
  2. Do the arithmetic outside. Enter the counts into the two-proportion formula in a spreadsheet. The inputs are the converted count and the base of each group, with the same unit (people or sessions) on both sides.
  3. Guard small samples. Live chat › Reports › Conversion attribution shows a sample-size warning when fewer than 30 visitors chatted. Treat any group below that as too small to test.

The same logic applies to your own splits, since TapCub measures but does not assign groups.

Sources

  • NIST/SEMATECH e-Handbook of Statistical Methods, Comparing two proportions: pooled z test statistic, critical value rules and the exact test for small samples. Checked 2026-11-21.
  • NIST/SEMATECH e-Handbook of Statistical Methods, Test statistic for the one-sample t-test: T = (Ȳ − μ0) ÷ (s/√N), degrees of freedom N − 1 and the critical region. Checked 2026-11-21.

Was this explanation clear?

We use your answers to fix and expand this page.

Emma Collins

Product Analyst

Focuses on event analysis, conversion funnels, and retention, using clear metric definitions and structured analysis to explain product performance.

FAQ

Questions people ask

Short answers to the follow-up questions that come up most.

Still have a question?

Chat with the team behind TapCub — we usually reply within a few hours on business days.

What is a test statistic in simple terms?

It is a single number that shows how surprising your data are if nothing is really going on. You divide the observed difference by the amount of noise expected from sampling. The bigger the number, the harder it is to explain the result as chance.

What is the formula for a test statistic?

Most take the form (observed value − value under the null hypothesis) ÷ standard error. For two conversion rates, z = (p̂1 − p̂2) ÷ √(p̂(1 − p̂)(1/n1 + 1/n2)), where p̂ is the pooled rate across both groups.

What is the difference between a test statistic and a p-value?

The test statistic measures distance from the null in standard errors, such as z = 2.15. The p-value converts that distance into a probability, here 0.032. They give the same decision: reject when the statistic passes the critical value or the p-value is below the significance level.

Does a bigger test statistic mean a bigger effect?

No. It means stronger evidence of some difference. With more data the same 1.1-point gap gives z = 2.15 at 4,000 visitors per version and z = 6.8 at 40,000, while the effect stays the same. Report the effect and its interval alongside the statistic.

See this number on your own site

Add one script tag for web and apps, and the first reports fill in within minutes.

See your data clearly. Find your growth.

Every click, backed by data. Install one line of code and see your first numbers in a minute.

No credit card · Free plans for analytics and chat · Cookieless analytics