TapCub is live — analytics, insights and live chat, free on one platform
Glossary

What is a Type II error in statistics? Definition, beta and an example

A Type II error is a miss: the test finds no difference although one exists. Its probability is beta, and power is 1 − beta. A worked 4.0% vs 4.6% example shows power of 10% at 1,000 per group and what sample is enough.

Chen Yu Reviewed by Xu An Also called false negative Last updated Nov 18, 2026 Glossary

Quick answer

A Type II error is a false negative: the test does not reject the null hypothesis although it is false, so a real difference goes unseen. Its probability is β, and power is 1 − β. Small samples, small effects and a strict significance level all raise it.

Formula: β = P(fail to reject H₀ | H₀ is false); power = 1 − β

What is a Type II error in statistics?

A Type II error is a false negative: the test does not reject the null hypothesis even though the null hypothesis is false. Put in plain words, there really is a difference or an effect, and the test fails to find it. Its probability is written β (beta). Statistical power is the opposite, the chance that the test finds a difference that is really there: power = 1 − β.

The test is a procedure, and β describes how often that procedure misses, not whether a particular result is wrong. A single experiment either missed or did not. What you can say beforehand is "if the true difference is this large, a test of this size will miss it about this often." That is why Type II error belongs in the planning of a comparison and not only in reading its result. If you have only ever heard "p > 0.05, no significant difference", this is the other half of that sentence: no significant difference does not mean no difference.

How is a Type II error different from a Type I error?

The two errors are the two ways a test can be wrong. The full side-by-side treatment is in Type I vs Type II error; the short version is the table below.

The difference is realThere is no real difference
Test says "difference"Correct (this is power, 1 − β)Type I error, a false alarm. Probability α
Test says "no difference"Type II error, a miss. Probability βCorrect (1 − α)

The easy way to keep them apart is by what went wrong. A Type I error is finding something that is not there; a Type II error is missing something that is there. The usual convention caps the false-alarm rate at α = 0.05 and is far more relaxed about misses: designs commonly aim for 80% power, which means accepting a 20% chance of missing a real effect of the size you planned for. Which error is more costly depends on the decision: shipping a change that does nothing is a Type I problem, while abandoning a change that works is a Type II problem.

What makes the Type II error rate go up?

Four things set β, and three of them are in your hands before the data arrives:

  • Effect size. The smaller the true difference, the harder it is to see through random variation. Halving the lift roughly quadruples the sample you need.
  • Sample size. More users per group shrink the random variation in each rate, so smaller differences stand out.
  • Variability. Noisier outcomes (revenue per user, with a few large buyers) need more users than yes/no outcomes with the same average.
  • The significance level α. Demanding more evidence to call a difference real (a smaller α) makes misses more likely. Moving from α = 0.05 to 0.01 raises β in the example below from about 90% to about 97%.

The relationship between β, α and the sample is a trade-off: for a fixed sample, lowering one error raises the other. The only way to lower both is more data or a less noisy measure.

Worked example: why a 4.0% vs 4.6% test usually finds nothing

Sample data: a page converts at 4.0% today and you suspect a change lifts it to 4.6%, a relative lift of 15%. You run an equal split, 1,000 visitors per group, and test at α = 0.05 (two-sided, two-proportion z-test).

Power for this design. With true rates 4.0% and 4.6%, the standard error under the null is √(2 × 0.043 × 0.957 ÷ 1,000) ≈ 0.0091 and under the true rates also ≈ 0.0091, while the true difference is 0.006. Working that through gives power ≈ 10.1%, so β ≈ 89.9%. A simulation of 3,000 repeated experiments under these true rates found a significant result in 9.0% of them, consistent with the calculation. In other words, the test misses a real 15% lift about nine times out of ten.

How many visitors are enough. The approximate sample per group is

n ≈ (z<sub>α/2</sub> × √(2p̄(1 − p̄)) + z<sub>β</sub> × √(p₁(1 − p₁) + p₂(1 − p₂)))² ÷ (p₂ − p₁)²

with z<sub>α/2</sub> = 1.96, z<sub>β</sub> = 0.84 for 80% power and 1.28 for 90% power, and p̄ = 0.043.

Visitors per groupPowerβ (chance of missing)
1,00010.1%89.9%
2,00015.5%84.5%
5,00031.5%68.5%
10,00055.2%44.8%
13,00066.4%33.6%
20,00084.1%15.9%

For 80% power you need about 17,943 visitors per group; for 90% power, about 24,020. The same exercise for other lifts at 1,000 visitors per group:

True rate vs 4.0%Relative liftPower at 1,000 per groupPer group for 80% power
4.6%15%10.1%17,943
5.0%25%19.0%6,745
6.0%50%53.7%1,863

The "same" experiment is a coin flip for a 50% lift and a near-certain miss for a 15% lift. The question to ask before running is not "how many visitors do I have?" but "what is the smallest lift worth acting on, and can I see it with the traffic I have?" Planning that threshold is the idea behind minimum detectable effect.

How do you reduce the chance of a Type II error?

In order of how often each works:

  1. Raise the sample. Run longer, or pool more traffic into each group. The table above shows the price: going from 1,000 to 20,000 per group takes power from 10% to 84%. Plan the run length in whole weeks so every weekday appears equally often in both groups.
  2. Test a bigger change. A redesigned page is easier to detect than a small tweak. Bundling small tweaks into one test is often better than five tests that each miss.
  3. Pick a less noisy outcome. Test the conversion rate, not revenue per visitor, if both answer the question. If you must use a noisy metric, plan for a larger sample.
  4. Reduce the number of groups. Splitting the same traffic across four variants gives each comparison a quarter of the sample.
  5. Loosen α only deliberately. Using α = 0.10 lifts power at 1,000 per group from 10.1% to 17.3%, but it also doubles the false-alarm rate to 10%. It is a reasonable choice when a wrong "go" is cheap and a missed win is expensive; it is not a free fix.

Note what does not help: waiting for the p-value to fall below 0.05 by looking every day. Repeatedly checking inflates the Type I error instead, which is a different problem (see p-hacking).

What do people get wrong about Type II errors?

  • "Not significant, so it does not work." With 10% power, a non-significant result is the expected outcome even when the lift is real. Report the confidence interval: in the example, a difference of 0.6 points on 1,000 per group has an interval that easily spans zero and also spans much larger effects. See how to calculate a confidence interval.
  • Calculating power after the fact from the observed effect. "Observed power" is a re-expression of the p-value and adds no information. Compute power for the effect size you planned to detect, before the test.
  • Treating 80% power as a guarantee. It means a 20% miss rate at the planned effect, and a higher miss rate for anything smaller.
  • Using the same sample for every question. A sample big enough for the headline metric can still be too small for a segment or secondary metric.
  • Pooling the non-results. Five tests that each missed at 20% power do not add up to "we tested five times and nothing works". Pool the raw counts and test once, or run a single larger test.
  • Ignoring which error costs more. If a wrong "ship" is cheap to undo, accept a smaller sample and a higher α; if a missed win is costly, plan for 90% power rather than 80%.
  • Confusing β with the p-value. A p-value is computed from your data under the null hypothesis; β is computed before the data, under a specific alternative. The reading of the former is in p-value, and the choice of the threshold in statistical significance.

How do you use TapCub to avoid Type II errors?

TapCub does not run A/B experiments, so it has no built-in test or power calculator. What it does is measure the inputs: the traffic and the base rate you need to plan a test. Take the 4.0% page from the example.

  1. Get the base rate and traffic. In Web analytics › Goals, read the goal's conversions and the page's visitors for a recent period, and divide one by the other. If the base rate is 4.0% and the page gets 1,500 visitors a week, an equal split gives 750 per group per week.
  2. Check the arithmetic against your traffic. For a 15% lift you need about 17,943 per group; at 750 per group per week that is roughly 24 weeks. For a 50% lift, 1,863 per group is about 2.5 weeks. If the weeks are not realistic, test a bigger change or accept that the test cannot answer the question.
  3. Count real people. TapCub counts human visitors and bots separately, so crawlers do not inflate the traffic you plan with or dilute the conversion rate.
  4. Run the test in your own tool and send the variant. Send your experiment's variant name as an event property so each group can be filtered. After the test, compare the two groups' goal conversion in TapCub.
  5. Read a non-result as unresolved. If the difference is not significant and your power was low, write that down as "not detectable at this sample", not "no effect".

See also: What is a Type 1 error? False positives, alpha and p-values.

Sources

Was this explanation clear?

We use your answers to fix and expand this page.

Chen Yu

Technical Engineer

Focuses on tracking code, SDKs, and API integrations, with practical guidance on implementation checks and troubleshooting.

FAQ

Questions people ask

Short answers to the follow-up questions that come up most.

Still have a question?

Chat with the team behind TapCub — we usually reply within a few hours on business days.

What is a Type II error in simple terms?

It is a miss. A real difference exists, but your test says there is none. A store that tests a change which really lifts conversion, sees no significant result and drops the change has made a Type II error.

What is the probability of a Type II error?

It is β, and it depends on the true effect size, sample size, variability and the significance level. For a 4.0% vs 4.6% comparison with 1,000 visitors per group at α = 0.05, β is about 90%, meaning the test misses the lift nine times out of ten.

What is the relationship between Type II error and power?

Power is the chance of finding a real effect, so power = 1 − β. Designs often aim for 80% power, a 20% chance of missing the planned effect. In the example here, about 17,943 visitors per group are needed to reach 80% power.

How do you reduce a Type II error?

Increase the sample size, test a larger change, use a less noisy outcome and avoid splitting traffic across many variants. Loosening α also lowers β but raises the false-alarm rate, so do it only on purpose.

See this number on your own site

Add one script tag for web and apps, and the first reports fill in within minutes.

See your data clearly. Find your growth.

Every click, backed by data. Install one line of code and see your first numbers in a minute.

No credit card · Free plans for analytics and chat · Cookieless analytics