TapCub is live — analytics, insights and live chat, free on one platform
Glossary

What is a Type 1 error? False positives, alpha and p-values

A Type 1 error is a false positive: rejecting a true null hypothesis. Its probability is alpha. See how alpha relates to p, how multiple tests and peeking inflate it, and how to choose a threshold.

Chen Yu Reviewed by Sophia Bennett Also called Type I error Last updated Nov 19, 2026 Glossary

Quick answer

A Type 1 error is a false positive: a statistical test rejects a null hypothesis that is actually true, reporting an effect that is not there. Its probability equals the significance level α, chosen before the test, commonly 0.05.

Formula: P(Type 1 error) = α; P(at least one in m independent tests) = 1 − (1 − α)^m

What is a Type 1 error?

A Type 1 error is a false positive: a statistical test rejects a null hypothesis that is actually true, so it announces an effect that is not there. A new checkout page that changes nothing gets declared a winner. A chart shows that visitors from one country convert better, and the gap is only noise.

The probability of making this error is the significance level, α (alpha), and you choose it before looking at the data. At α = 0.05, a test on a true null hypothesis will wrongly reject it about 1 time in 20. Neyman and Pearson (1933) introduced the two kinds of error; the other one, missing a real effect, is a Type 2 error. For the two side by side, see Type 1 vs Type 2 error; this page stays on the first kind.

How does α relate to the p-value?

They are easy to mix up because they are compared with each other, but they are different things.

α (significance level)p-value
What it isA threshold you set in advanceA number computed from your data
Question it answersHow often am I willing to cry wolf?If there were no effect, how often would data this extreme appear?
Typical value0.05, 0.01, 0.10Anything from 0 to 1
Decision ruleReject the null if p ≤ αCompare it with α, do not interpret it alone

Example: an experiment gives p = 0.03. At α = 0.05 you reject the null; at α = 0.01 you do not. The data did not change; only your tolerance for false alarms did. A p-value is not the probability that the null hypothesis is true, and p = 0.03 does not mean "97% sure the effect is real" (see p-value and the American Statistical Association statement in Sources).

How do you choose α?

α trades false alarms against missed effects: lowering it makes Type 1 errors rarer and Type 2 errors more likely, unless you also collect more data. Choose it by what a wrong "yes" costs.

αWrong "yes" happens aboutReasonable when
0.101 time in 10The change is cheap and easy to reverse, and you are screening many ideas
0.051 time in 20The common default for product and marketing tests
0.011 time in 100The change is costly, hard to undo, or affects pricing or safety

Two rules keep this honest. Fix α before the test starts, not after you see which threshold the result passes. And write down the decision it controls: "ship the new page if p ≤ 0.05 and the lift is at least 3%".

Why do many tests make false positives likely?

α applies to one test. Run several on true nulls and the chance of at least one false positive climbs. With m independent tests at α = 0.05:

P(at least one false positive) = 1 − (1 − α)^m

Sample data: you check 20 metrics (revenue, clicks, bounce and so on) in a test where nothing actually changed.

Tests (m)Chance of ≥ 1 false positive at α = 0.05With Bonferroni α = 0.05 ÷ m
15.0%5.0%
522.6%4.9%
1040.1%4.9%
2064.2%4.9%

At 20 tests the expected number of false positives is 20 × 0.05 = 1, and the chance of seeing at least one is 64.2%. The Bonferroni correction divides α by the number of tests (0.05 ÷ 20 = 0.0025 each), which brings the family-wise chance back to 4.9%. It is simple but strict; the Benjamini–Hochberg procedure controls the share of false discoveries instead and loses less power when you test many things.

How does peeking create false positives?

Checking a running test and stopping when p dips under 0.05 is another form of multiple testing. In a simulation of 20,000 A/A runs (two identical groups, five looks at equal intervals, stop at the first |z| > 1.96), 13.9% declared a "winner", against 5.0% when looking only once at the end. That matches the roughly 14% reported by Armitage and colleagues (1969). The cure is to fix the sample size in advance, or to use a method designed for repeated looks.

How often are "significant wins" actually false?

α is the false-positive rate among tests of true nulls. It is not the share of your announced wins that are false, because that also depends on how many ideas really work.

Sample data: 1,000 ideas tested, 100 truly work, 900 do nothing. The test has 80% power and α = 0.05.

  • True positives: 100 × 0.80 = 80.
  • False positives: 900 × 0.05 = 45.
  • Announced wins: 80 + 45 = 125, of which 45 ÷ 125 = 36% are false.

If only 20 of the 1,000 ideas really work, the same test gives 16 true and 49 false positives, so 75% of wins are false. Rare real effects plus a loose threshold mean a lot of false "wins".

What does a Type 1 error look like in practice?

The same mistake shows up in different settings. In each, the null hypothesis is "nothing changed", and the Type 1 error is believing otherwise.

SettingNull hypothesisWhat a Type 1 error looks likeWhat it costs
Landing-page testThe new page converts the same as the old oneYou ship the new page because of a lucky weekEngineering time, and a real conversion drop later
Marketing reportTwo channels convert equallyYou move budget to the channel that was ahead by chanceSpend on a channel that is no better
Medical screeningThe patient does not have the conditionA healthy patient is told the test is positiveAnxiety and follow-up tests
Spam filterThe email is legitimateA real email goes to the spam folderA missed message

The cost column is why α is a business decision as well as a statistical one.

Common mistakes with Type 1 errors

  • Reading p ≤ 0.05 as "proven". It means the data are unusual if there were no effect, nothing more.
  • Slicing until something is significant. Splitting one result by 10 countries × 10 channels is 100 comparisons. Decide the segments first, or correct for the number you looked at.
  • Switching the metric after the fact. Name one primary metric before the test.
  • Treating "not significant" as "no effect". That is a different risk, the Type 2 error.
  • Ignoring effect size. With a very large sample, a 0.1% lift can be significant and still not worth shipping.

How does this apply to your analytics data?

TapCub does not run experiments, split traffic or calculate p-values; it measures what happened. The Type 1 risk lives in how you read its reports, so the discipline is to decide before you look.

  1. Write the question down first. For example "did the new pricing page raise goal completions?" with one goal from Web analytics › Goals and the dates.
  2. Mark the change. Add the release in Configuration › Annotations so the before and after periods are unambiguous.
  3. Limit your slices. Breaking one report into many segments (sources, countries, devices) creates many comparisons; pick two or three in advance and note how many you looked at.
  4. Compare like with like. Use Web analytics › Sources and Audience to check that the periods have similar traffic mixes; real people are counted apart from bots.
  5. Run the test elsewhere, then measure here. Use your experiment tool for the random split and the significance test; use TapCub for the goal numbers, and divide α by the number of comparisons you made, as in the table above.

Sources

Was this explanation clear?

We use your answers to fix and expand this page.

Chen Yu

Technical Engineer

Focuses on tracking code, SDKs, and API integrations, with practical guidance on implementation checks and troubleshooting.

FAQ

Questions people ask

Short answers to the follow-up questions that come up most.

Still have a question?

Chat with the team behind TapCub — we usually reply within a few hours on business days.

What is a Type 1 error in simple terms?

It is a false alarm: the test says there is an effect, but there is none. A new landing page that changes nothing is declared a winner because random variation made it look better. The chance of this on a single test is set by the significance level, α.

Is a Type 1 error the same as a false positive?

Yes. In hypothesis testing, a false positive and a Type 1 error are the same event: rejecting a null hypothesis that is true. The probability is α. Note that this is the error rate among true nulls, not the share of your announced wins that are false.

Is the p-value the probability of a Type 1 error?

No. α is the Type 1 error rate you accept, set in advance. The p-value is computed from your data: how likely results this extreme are if there were no effect. You reject the null when p ≤ α, so α is the threshold, p is the evidence.

How do you reduce Type 1 errors?

Lower α, fix the sample size and the primary metric before the test, avoid stopping when p first dips below the threshold, and correct for the number of comparisons, for example with Bonferroni (α ÷ number of tests). Lowering α alone raises the Type 2 error risk unless you add data.

See this number on your own site

Add one script tag for web and apps, and the first reports fill in within minutes.

See your data clearly. Find your growth.

Every click, backed by data. Install one line of code and see your first numbers in a minute.

No credit card · Free plans for analytics and chat · Cookieless analytics