Quick answer
Hypothesis testing is a procedure that uses sample data to decide whether a difference is larger than chance alone would plausibly produce. You fix a null hypothesis, a significance level and a sample size first, then compute a p-value and reject the null only if it falls below that level.
Formula: z = (p̂₁ − p̂₂) ÷ √( p̂ (1 − p̂) (1/n₁ + 1/n₂) ), where p̂ = (x₁ + x₂) ÷ (n₁ + n₂); reject H0 when the two-sided p-value is below α
What is hypothesis testing?
Hypothesis testing is a statistical procedure for deciding whether a difference seen in sample data is larger than chance alone would plausibly produce. You state a null hypothesis, usually "no difference" or "no effect", and an alternative. You compute a test statistic and a p-value: the probability of getting a result at least this extreme if the null were true. If the p-value is below a threshold α that you chose in advance, you reject the null. NIST describes the null as a statement the test tries to reject, and α as the risk of rejecting a null that is actually true.
Failing to reject the null does not prove it. It means this sample did not give enough evidence against it.
The idea behind the method is to set the target before shooting: the hypotheses, α, the test and the sample size are all fixed before you look at the results. Everything after that, especially the p-value, only means something if you kept to that order.
The six steps
| Step | What you decide | Example: two conversion rates |
|---|---|---|
| 1. State the hypotheses | The null and the alternative | H0: both rates are equal. H1: they differ (two-sided) |
| 2. Set α and the direction | How much false-alarm risk you accept | α = 0.05, two-sided |
| 3. Choose the test | One that fits the data type | Two-proportion z-test (needs large samples) |
| 4. Fix the sample size | Enough to detect the smallest difference you care about | 5.0% → 6.0% with 80% power needs about 8,155 per group |
| 5. Collect all the data, then compute | The test statistic and the p-value | z and p from the formula below |
| 6. Decide and report | Reject or not, plus the size of the difference and its interval | "+1.0 point, 95% interval +0.3 to +1.7" |
For two proportions, NIST gives the large-sample statistic z = (p̂₁ − p̂₂) ÷ √( p̂ (1 − p̂) (1/n₁ + 1/n₂) ), where p̂₁ and p̂₂ are the group rates and p̂ is the pooled rate, (x₁ + x₂) ÷ (n₁ + n₂).
Worked example: two checkout variants
Sample data: two checkout variants were set up in advance with H0 "equal conversion rates", a two-sided α of 0.05 and a smallest difference of interest of 1 point (5.0% → 6.0%). At 80% power that needs 8,155 visitors per group, rounded up to 8,200.
| Group | Visitors | Conversions | Conversion rate |
|---|---|---|---|
| A (current) | 8,200 | 410 | 5.00% |
| B (new) | 8,200 | 492 | 6.00% |
- Pooled rate: (410 + 492) ÷ 16,400 = 0.055.
- Standard error: √(0.055 × 0.945 × (1/8,200 + 1/8,200)) = 0.00356.
- z = 0.010 ÷ 0.00356 = 2.81, so the two-sided p-value is 0.005. That is below 0.05, so reject H0.
- The difference is +1.0 point, with a 95% interval of +0.30 to +1.70 points.
Now the same rates with far less data. Sample data: A has 100 conversions from 2,000 visitors (5.0%), B has 118 from 2,000 (5.9%). Then z = 1.25 and p = 0.21, so you cannot reject H0, and the 95% interval for the difference runs from −0.51 to +2.31 points. A test of this size had only about 28% power to detect a 1-point gain, so the honest reading is "not enough data", not "no effect".
Type I and type II errors
| A real difference exists | No real difference exists | |
|---|---|---|
| You reject H0 | Correct (this is power, 1 − β) | Type I error, probability α (5% here) |
| You do not reject H0 | Type II error, probability β | Correct |
α is the risk of a false alarm and β is the risk of a miss. NIST notes that the risk β increases as α decreases. For a fixed sample you cannot lower both; a larger sample is what lets you do that, which is why the sample size comes first. The 8,200 per group above gives about 80% power (80.2%) for a 1-point gain. The difference between the two errors is covered in type 1 vs type 2 error, the power side in statistical power, and the smallest gain a test can see in minimum detectable effect.
What a p-value does and does not say
The American Statistical Association's 2016 statement makes four points that matter here. A p-value shows how incompatible the data are with a stated model. It is not the probability that the hypothesis is true. It does not measure the size or importance of an effect. And a decision should not rest only on whether it crosses a threshold.
In the example, p = 0.005 does not mean there is a 99.5% chance that B is better. It means that if the two variants were really equal, a gap this large would show up about 0.5% of the time. The effect size and the interval are what you act on.
Choosing a test
| Question | Data | Test | Free calculator |
|---|---|---|---|
| Do two conversion rates differ? | Conversions and visitors per group | Two-proportion z-test; for a 2×2 table, z² equals the chi-square statistic | Chi-square calculator |
| Did this week's traffic differ from last week's? | Daily visit counts | Poisson test of two rates, adjusted for weekdays | Traffic change significance |
| Do two groups' average values differ, such as order value? | A number per user | t-test | None here |
| How many visitors do I need? | Baseline rate and smallest difference | Sample-size calculation | A/B test sample size |
Common mistakes
- Peeking. Stopping the first time p falls below 0.05 inflates false alarms. Sample data from a simulation of 20,000 tests in which both groups truly converted at 5.0%: with one test at 10,000 per group, 4.9% came out significant. With ten looks, one every 1,000 visitors per group, 19.2% were significant at least once. With five looks, 14.2%.
- Many comparisons. Test 20 segments at α = 0.05 and the chance of at least one false positive is 1 − 0.95²⁰ = 64.2%, if the tests are independent. Decide the comparisons in advance or tighten α.
- Reading "not significant" as "no effect". Sample data: B converts at 5.51% (452 of 8,200) against A's 5.00% (410 of 8,200). Then z = 1.47 and p = 0.14, yet the 95% interval is −0.17 to +1.20 points, which still includes the 1-point gain you hoped for.
- Confusing significant with important. With 1,000,000 visitors per group, 5.00% against 5.10% gives z = 3.23 and p = 0.001. It is significant, and the gain is a tenth of a point. Report the size, not only the p-value.
- Changing the rules after seeing the data. Switching to a one-sided test once you see which way the gap points halves the p-value. Fix the direction, α and the test beforehand.
- Treating a before/after comparison as an experiment. A test says a gap is unlikely to be chance alone, not what caused it; see correlation vs causation.
Hypothesis testing next to TapCub reports
Say you changed the pricing page last Tuesday and want to know whether conversion really moved.
- Know what TapCub does here. TapCub does not run A/B experiments or split traffic, and its reports show differences rather than test them. It measures the before and after; the test happens outside.
- Compare the right periods. On Web analytics › Goals, set the comparison in the report header to the previous period or to the same period last week. The same-week option keeps weekdays aligned, which matters when weekends differ.
- Mark the change. Add a date annotation in Configuration › Annotations for the release, so anyone reading the chart later sees what changed and when.
- Take the four numbers out and test them. Copy visitors and conversions for each period into the chi-square calculator or traffic change significance, and use the sample-size calculator to plan the next comparison.
- Keep the denominators honest. New projects count recognised bots and AI crawlers separately, so they do not inflate visitors or dilute the conversion rate you test.
Live chat › Reports › Conversion attribution is a good picture of the limits. It compares visitors who chatted with visitors who only browsed, flags the sample as small when fewer than 30 people chatted, and does not claim the chat caused the goal. Treat any test on a comparison like that the same way: it shows whether the gap is likely to be chance, not why it exists.
Sources
- NIST/SEMATECH e-Handbook of Statistical Methods, 7.1.3 What are statistical tests?
- NIST/SEMATECH e-Handbook of Statistical Methods, 7.3.3 How can we determine whether two processes produce the same proportion of defectives? (the two-proportion z-test used in the example).
- Wasserstein, R. L. & Lazar, N. A. (2016). The ASA Statement on p-Values: Context, Process, and Purpose. The American Statistician, 70(2), 129–133.
Was this explanation clear?
We use your answers to fix and expand this page.
Couldn’t send. Please try again in a minute.
Thanks for the feedback. We’ll use it to improve this page.You’ve already sent feedback on this page. Thanks.

Product Analyst
Focuses on event analysis, conversion funnels, and retention, using clear metric definitions and structured analysis to explain product performance.