Quick answer
A Type 1 error is a false positive: a statistical test rejects a null hypothesis that is actually true, reporting an effect that is not there. Its probability equals the significance level α, chosen before the test, commonly 0.05.
Formula: P(Type 1 error) = α; P(at least one in m independent tests) = 1 − (1 − α)^m
What is a Type 1 error?
A Type 1 error is a false positive: a statistical test rejects a null hypothesis that is actually true, so it announces an effect that is not there. A new checkout page that changes nothing gets declared a winner. A chart shows that visitors from one country convert better, and the gap is only noise.
The probability of making this error is the significance level, α (alpha), and you choose it before looking at the data. At α = 0.05, a test on a true null hypothesis will wrongly reject it about 1 time in 20. Neyman and Pearson (1933) introduced the two kinds of error; the other one, missing a real effect, is a Type 2 error. For the two side by side, see Type 1 vs Type 2 error; this page stays on the first kind.
How does α relate to the p-value?
They are easy to mix up because they are compared with each other, but they are different things.
| α (significance level) | p-value | |
|---|---|---|
| What it is | A threshold you set in advance | A number computed from your data |
| Question it answers | How often am I willing to cry wolf? | If there were no effect, how often would data this extreme appear? |
| Typical value | 0.05, 0.01, 0.10 | Anything from 0 to 1 |
| Decision rule | Reject the null if p ≤ α | Compare it with α, do not interpret it alone |
Example: an experiment gives p = 0.03. At α = 0.05 you reject the null; at α = 0.01 you do not. The data did not change; only your tolerance for false alarms did. A p-value is not the probability that the null hypothesis is true, and p = 0.03 does not mean "97% sure the effect is real" (see p-value and the American Statistical Association statement in Sources).
How do you choose α?
α trades false alarms against missed effects: lowering it makes Type 1 errors rarer and Type 2 errors more likely, unless you also collect more data. Choose it by what a wrong "yes" costs.
| α | Wrong "yes" happens about | Reasonable when |
|---|---|---|
| 0.10 | 1 time in 10 | The change is cheap and easy to reverse, and you are screening many ideas |
| 0.05 | 1 time in 20 | The common default for product and marketing tests |
| 0.01 | 1 time in 100 | The change is costly, hard to undo, or affects pricing or safety |
Two rules keep this honest. Fix α before the test starts, not after you see which threshold the result passes. And write down the decision it controls: "ship the new page if p ≤ 0.05 and the lift is at least 3%".
Why do many tests make false positives likely?
α applies to one test. Run several on true nulls and the chance of at least one false positive climbs. With m independent tests at α = 0.05:
P(at least one false positive) = 1 − (1 − α)^m
Sample data: you check 20 metrics (revenue, clicks, bounce and so on) in a test where nothing actually changed.
| Tests (m) | Chance of ≥ 1 false positive at α = 0.05 | With Bonferroni α = 0.05 ÷ m |
|---|---|---|
| 1 | 5.0% | 5.0% |
| 5 | 22.6% | 4.9% |
| 10 | 40.1% | 4.9% |
| 20 | 64.2% | 4.9% |
At 20 tests the expected number of false positives is 20 × 0.05 = 1, and the chance of seeing at least one is 64.2%. The Bonferroni correction divides α by the number of tests (0.05 ÷ 20 = 0.0025 each), which brings the family-wise chance back to 4.9%. It is simple but strict; the Benjamini–Hochberg procedure controls the share of false discoveries instead and loses less power when you test many things.
How does peeking create false positives?
Checking a running test and stopping when p dips under 0.05 is another form of multiple testing. In a simulation of 20,000 A/A runs (two identical groups, five looks at equal intervals, stop at the first |z| > 1.96), 13.9% declared a "winner", against 5.0% when looking only once at the end. That matches the roughly 14% reported by Armitage and colleagues (1969). The cure is to fix the sample size in advance, or to use a method designed for repeated looks.
How often are "significant wins" actually false?
α is the false-positive rate among tests of true nulls. It is not the share of your announced wins that are false, because that also depends on how many ideas really work.
Sample data: 1,000 ideas tested, 100 truly work, 900 do nothing. The test has 80% power and α = 0.05.
- True positives: 100 × 0.80 = 80.
- False positives: 900 × 0.05 = 45.
- Announced wins: 80 + 45 = 125, of which 45 ÷ 125 = 36% are false.
If only 20 of the 1,000 ideas really work, the same test gives 16 true and 49 false positives, so 75% of wins are false. Rare real effects plus a loose threshold mean a lot of false "wins".
What does a Type 1 error look like in practice?
The same mistake shows up in different settings. In each, the null hypothesis is "nothing changed", and the Type 1 error is believing otherwise.
| Setting | Null hypothesis | What a Type 1 error looks like | What it costs |
|---|---|---|---|
| Landing-page test | The new page converts the same as the old one | You ship the new page because of a lucky week | Engineering time, and a real conversion drop later |
| Marketing report | Two channels convert equally | You move budget to the channel that was ahead by chance | Spend on a channel that is no better |
| Medical screening | The patient does not have the condition | A healthy patient is told the test is positive | Anxiety and follow-up tests |
| Spam filter | The email is legitimate | A real email goes to the spam folder | A missed message |
The cost column is why α is a business decision as well as a statistical one.
Common mistakes with Type 1 errors
- Reading p ≤ 0.05 as "proven". It means the data are unusual if there were no effect, nothing more.
- Slicing until something is significant. Splitting one result by 10 countries × 10 channels is 100 comparisons. Decide the segments first, or correct for the number you looked at.
- Switching the metric after the fact. Name one primary metric before the test.
- Treating "not significant" as "no effect". That is a different risk, the Type 2 error.
- Ignoring effect size. With a very large sample, a 0.1% lift can be significant and still not worth shipping.
How does this apply to your analytics data?
TapCub does not run experiments, split traffic or calculate p-values; it measures what happened. The Type 1 risk lives in how you read its reports, so the discipline is to decide before you look.
- Write the question down first. For example "did the new pricing page raise goal completions?" with one goal from Web analytics › Goals and the dates.
- Mark the change. Add the release in Configuration › Annotations so the before and after periods are unambiguous.
- Limit your slices. Breaking one report into many segments (sources, countries, devices) creates many comparisons; pick two or three in advance and note how many you looked at.
- Compare like with like. Use Web analytics › Sources and Audience to check that the periods have similar traffic mixes; real people are counted apart from bots.
- Run the test elsewhere, then measure here. Use your experiment tool for the random split and the significance test; use TapCub for the goal numbers, and divide α by the number of comparisons you made, as in the table above.
Sources
- Neyman, J., & Pearson, E. S. (1933). On the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society A, 231, 289–337.
- NIST/SEMATECH e-Handbook of Statistical Methods, 7.1.3 What are statistical tests?
- Wasserstein, R. L., & Lazar, N. A. (2016). The ASA Statement on p-Values: Context, Process, and Purpose. The American Statistician, 70(2), 129–133.
- Benjamini, Y., & Hochberg, Y. (1995). Controlling the False Discovery Rate. Journal of the Royal Statistical Society B, 57(1), 289–300.
- Armitage, P., McPherson, C. K., & Rowe, B. C. (1969). Repeated Significance Tests on Accumulating Data. Journal of the Royal Statistical Society A, 132(2), 235–244.
Was this explanation clear?
We use your answers to fix and expand this page.
Couldn’t send. Please try again in a minute.
Thanks for the feedback. We’ll use it to improve this page.You’ve already sent feedback on this page. Thanks.

Technical Engineer
Focuses on tracking code, SDKs, and API integrations, with practical guidance on implementation checks and troubleshooting.