What is a p-value? Meaning, the 0.05 rule and an A/B test example

What is a p-value?

A p-value answers one narrow question: if there were really no difference, how often would I see a result at least as extreme as this one? The "no difference" assumption is called the null hypothesis.

• A small p-value (say 0.01) means your result would be rare if nothing were going on, so you have evidence against the null.
• A large p-value (say 0.40) means a result like yours happens often by chance alone, so the data do not separate a real effect from noise.

Before the test you choose a threshold, usually 0.05. If p falls below it, the result is called statistically significant. That is a convention, not a law of nature; the American Statistical Association warns against treating 0.05 as a bright line.

An A/B test example

A sign-up page test (sample numbers). The original converts 4.0%, the new version 4.8%: a 20% relative lift. Is it real?

Same rates, same lift, different verdicts. With 5,000 visitors each, a gap this size would turn up about 5 times in 100 even with no real difference, just above the 0.05 line. With twice the traffic the same gap would be rare by chance (6 in 1,000). The p-value depends on the effect and on the sample size together.

What a p-value does not tell you

1. It is not the probability that the null is true. p = 0.03 does not mean "3% chance there is no difference".
2. It is not the probability your variant is better. That is a different (Bayesian) question.
3. It is not the size of the effect. Always report the lift and a confidence interval.
4. It is not a guarantee. At 0.05, about 1 in 20 tests of changes that do nothing will still come out "significant": a Type I error.
5. It is not valid if you peek and stop. Checking daily and stopping the first time p < 0.05 inflates false positives well above 5%.

How to use p-values in product decisions

1. Fix the threshold and the sample size before you start, from your baseline rate and the smallest lift worth shipping. The method is in how to calculate statistical significance.
2. Run to the planned sample, covering full weeks so weekday patterns even out.
3. Read p together with the lift and its interval. A significant +0.1% lift may not be worth shipping; a non-significant +15% may be worth a longer test.
4. Count tests honestly. If you compare ten metrics, one will look significant by chance; pick one primary metric up front.
5. Remember the design. A p-value says the difference is unlikely to be noise; only a randomised test lets you say the change caused it (correlation vs causation).