TapCub is live — analytics, insights and live chat, free on one platform
Glossary

What is hypothesis testing? Steps, errors and a worked conversion example

Hypothesis testing decides whether a difference in sample data is bigger than chance would produce. The six steps, a worked conversion-rate example, type I and II errors, and the mistakes that make a p-value meaningless.

Emma Collins Reviewed by Su He Also called statistical hypothesis test Last updated Nov 17, 2026 Glossary

Quick answer

Hypothesis testing is a procedure that uses sample data to decide whether a difference is larger than chance alone would plausibly produce. You fix a null hypothesis, a significance level and a sample size first, then compute a p-value and reject the null only if it falls below that level.

Formula: z = (p̂₁ − p̂₂) ÷ √( p̂ (1 − p̂) (1/n₁ + 1/n₂) ), where p̂ = (x₁ + x₂) ÷ (n₁ + n₂); reject H0 when the two-sided p-value is below α

What is hypothesis testing?

Hypothesis testing is a statistical procedure for deciding whether a difference seen in sample data is larger than chance alone would plausibly produce. You state a null hypothesis, usually "no difference" or "no effect", and an alternative. You compute a test statistic and a p-value: the probability of getting a result at least this extreme if the null were true. If the p-value is below a threshold α that you chose in advance, you reject the null. NIST describes the null as a statement the test tries to reject, and α as the risk of rejecting a null that is actually true.

Failing to reject the null does not prove it. It means this sample did not give enough evidence against it.

The idea behind the method is to set the target before shooting: the hypotheses, α, the test and the sample size are all fixed before you look at the results. Everything after that, especially the p-value, only means something if you kept to that order.

The six steps

StepWhat you decideExample: two conversion rates
1. State the hypothesesThe null and the alternativeH0: both rates are equal. H1: they differ (two-sided)
2. Set α and the directionHow much false-alarm risk you acceptα = 0.05, two-sided
3. Choose the testOne that fits the data typeTwo-proportion z-test (needs large samples)
4. Fix the sample sizeEnough to detect the smallest difference you care about5.0% → 6.0% with 80% power needs about 8,155 per group
5. Collect all the data, then computeThe test statistic and the p-valuez and p from the formula below
6. Decide and reportReject or not, plus the size of the difference and its interval"+1.0 point, 95% interval +0.3 to +1.7"

For two proportions, NIST gives the large-sample statistic z = (p̂₁ − p̂₂) ÷ √( p̂ (1 − p̂) (1/n₁ + 1/n₂) ), where p̂₁ and p̂₂ are the group rates and p̂ is the pooled rate, (x₁ + x₂) ÷ (n₁ + n₂).

Worked example: two checkout variants

Sample data: two checkout variants were set up in advance with H0 "equal conversion rates", a two-sided α of 0.05 and a smallest difference of interest of 1 point (5.0% → 6.0%). At 80% power that needs 8,155 visitors per group, rounded up to 8,200.

GroupVisitorsConversionsConversion rate
A (current)8,2004105.00%
B (new)8,2004926.00%
  • Pooled rate: (410 + 492) ÷ 16,400 = 0.055.
  • Standard error: √(0.055 × 0.945 × (1/8,200 + 1/8,200)) = 0.00356.
  • z = 0.010 ÷ 0.00356 = 2.81, so the two-sided p-value is 0.005. That is below 0.05, so reject H0.
  • The difference is +1.0 point, with a 95% interval of +0.30 to +1.70 points.

Now the same rates with far less data. Sample data: A has 100 conversions from 2,000 visitors (5.0%), B has 118 from 2,000 (5.9%). Then z = 1.25 and p = 0.21, so you cannot reject H0, and the 95% interval for the difference runs from −0.51 to +2.31 points. A test of this size had only about 28% power to detect a 1-point gain, so the honest reading is "not enough data", not "no effect".

Type I and type II errors

A real difference existsNo real difference exists
You reject H0Correct (this is power, 1 − β)Type I error, probability α (5% here)
You do not reject H0Type II error, probability βCorrect

α is the risk of a false alarm and β is the risk of a miss. NIST notes that the risk β increases as α decreases. For a fixed sample you cannot lower both; a larger sample is what lets you do that, which is why the sample size comes first. The 8,200 per group above gives about 80% power (80.2%) for a 1-point gain. The difference between the two errors is covered in type 1 vs type 2 error, the power side in statistical power, and the smallest gain a test can see in minimum detectable effect.

What a p-value does and does not say

The American Statistical Association's 2016 statement makes four points that matter here. A p-value shows how incompatible the data are with a stated model. It is not the probability that the hypothesis is true. It does not measure the size or importance of an effect. And a decision should not rest only on whether it crosses a threshold.

In the example, p = 0.005 does not mean there is a 99.5% chance that B is better. It means that if the two variants were really equal, a gap this large would show up about 0.5% of the time. The effect size and the interval are what you act on.

Choosing a test

QuestionDataTestFree calculator
Do two conversion rates differ?Conversions and visitors per groupTwo-proportion z-test; for a 2×2 table, z² equals the chi-square statisticChi-square calculator
Did this week's traffic differ from last week's?Daily visit countsPoisson test of two rates, adjusted for weekdaysTraffic change significance
Do two groups' average values differ, such as order value?A number per usert-testNone here
How many visitors do I need?Baseline rate and smallest differenceSample-size calculationA/B test sample size

Common mistakes

  • Peeking. Stopping the first time p falls below 0.05 inflates false alarms. Sample data from a simulation of 20,000 tests in which both groups truly converted at 5.0%: with one test at 10,000 per group, 4.9% came out significant. With ten looks, one every 1,000 visitors per group, 19.2% were significant at least once. With five looks, 14.2%.
  • Many comparisons. Test 20 segments at α = 0.05 and the chance of at least one false positive is 1 − 0.95²⁰ = 64.2%, if the tests are independent. Decide the comparisons in advance or tighten α.
  • Reading "not significant" as "no effect". Sample data: B converts at 5.51% (452 of 8,200) against A's 5.00% (410 of 8,200). Then z = 1.47 and p = 0.14, yet the 95% interval is −0.17 to +1.20 points, which still includes the 1-point gain you hoped for.
  • Confusing significant with important. With 1,000,000 visitors per group, 5.00% against 5.10% gives z = 3.23 and p = 0.001. It is significant, and the gain is a tenth of a point. Report the size, not only the p-value.
  • Changing the rules after seeing the data. Switching to a one-sided test once you see which way the gap points halves the p-value. Fix the direction, α and the test beforehand.
  • Treating a before/after comparison as an experiment. A test says a gap is unlikely to be chance alone, not what caused it; see correlation vs causation.

Hypothesis testing next to TapCub reports

Say you changed the pricing page last Tuesday and want to know whether conversion really moved.

  1. Know what TapCub does here. TapCub does not run A/B experiments or split traffic, and its reports show differences rather than test them. It measures the before and after; the test happens outside.
  2. Compare the right periods. On Web analytics › Goals, set the comparison in the report header to the previous period or to the same period last week. The same-week option keeps weekdays aligned, which matters when weekends differ.
  3. Mark the change. Add a date annotation in Configuration › Annotations for the release, so anyone reading the chart later sees what changed and when.
  4. Take the four numbers out and test them. Copy visitors and conversions for each period into the chi-square calculator or traffic change significance, and use the sample-size calculator to plan the next comparison.
  5. Keep the denominators honest. New projects count recognised bots and AI crawlers separately, so they do not inflate visitors or dilute the conversion rate you test.

Live chat › Reports › Conversion attribution is a good picture of the limits. It compares visitors who chatted with visitors who only browsed, flags the sample as small when fewer than 30 people chatted, and does not claim the chat caused the goal. Treat any test on a comparison like that the same way: it shows whether the gap is likely to be chance, not why it exists.

Sources

Was this explanation clear?

We use your answers to fix and expand this page.

Emma Collins

Product Analyst

Focuses on event analysis, conversion funnels, and retention, using clear metric definitions and structured analysis to explain product performance.

FAQ

Questions people ask

Short answers to the follow-up questions that come up most.

Still have a question?

Chat with the team behind TapCub — we usually reply within a few hours on business days.

What are the steps of hypothesis testing?

State the null and alternative hypotheses, set the significance level α and the direction, choose a test that fits the data, fix the sample size, collect all the data and compute the statistic and p-value, then decide and report the size of the difference with its interval.

What is the difference between the null and alternative hypothesis?

The null hypothesis is the default claim of no difference or no effect, and it is what the test tries to reject. The alternative hypothesis is the claim that a difference exists, either in one direction or in either direction.

What does it mean to fail to reject the null hypothesis?

It means the sample did not give enough evidence against the null, not that the null is true. A small sample can miss a real difference, so look at the interval and the test's power before concluding there is no effect.

Does TapCub run hypothesis tests?

No. TapCub measures visitors, conversions and other metrics and can compare periods, but it does not run A/B experiments or compute significance. Copy the counts into a calculator such as the free chi-square or traffic-change tools on this site.

See this number on your own site

Add one script tag for web and apps, and the first reports fill in within minutes.

See your data clearly. Find your growth.

Every click, backed by data. Install one line of code and see your first numbers in a minute.

No credit card · Free plans for analytics and chat · Cookieless analytics