TapCub is live — analytics, insights and live chat, free on one platform
Analytics

P-value meaning: how to interpret a p-value, with a worked example

A p-value is how often random variation alone would produce a gap as large as yours if there were no real difference. A 4.0% vs 4.8% worked example shows p = 0.168, why it does not mean 17% chance of no effect, and what to read next to it.

Xu An Published Nov 18, 2026 12 min read Analytics
Sample data: 4.0% vs 4.8% conversion at each sample size, two-sided two-proportion z-test. Bars scale to p = 0.168; highlighted rows are below 0.05. The gap never changes, only the amount of data does.Sample data

A p-value is the chance of seeing a gap at least as large as yours if there were really no difference, not the chance your result is true. In the 4.0% vs 4.8% example below, p = 0.168 and the interval runs from −0.34 to +1.94 points. Read the p-value with the gap and the interval, never alone.

What you will learn

  • The p-value assumes there is no difference; it cannot tell you how likely it is that there is none
  • The same 0.8-point gap goes from p = 0.168 to p = 0.0001 as visitors per group grow from 2,500 to 20,000, so p is not effect size
  • Read p together with the confidence interval and a smallest worthwhile gap you set before the test

What does a p-value actually mean?

A p-value is a probability about your data, not about your hypothesis. It answers one question: if there were really no difference (the null hypothesis), how often would random variation alone produce a gap at least as large as the one you saw? A p-value of 0.03 means that with no real difference, about 3 experiments in 100 would show a gap this big or bigger. It does not mean there is a 97% chance the difference is real, and it does not mean there is a 3% chance you are wrong.

The American Statistical Association put this in writing in 2016: p-values can indicate how incompatible the data are with a specified statistical model; they do not measure the probability that the hypothesis is true or that the data were produced by chance alone; and they say nothing about the size or importance of an effect. Two words in the definition do most of the work. "At least as large" means the p-value counts the observed result and everything more extreme, not just the exact result. "Assuming no difference" means the whole calculation is done in a world where your idea is false, so the p-value cannot tell you how likely that world is. The short glossary definition is in p-value; this guide is about reading one in a real result.

How do you read a p-value, step by step?

Use the same five steps every time:

  1. Write down the null hypothesis in words. "The two versions convert at the same rate." If you cannot say what "no difference" is, you cannot read the p-value.
  2. Note the test and the threshold you committed to before looking. Two-sided or one-sided, and the cut-off, usually 0.05. Choosing the cut-off after seeing the p-value is a form of p-hacking.
  3. Find the p-value and the estimate it belongs to. Always read the two together: the observed gap, and the p-value for that gap.
  4. Say it as a sentence. "If the two versions really convert the same, a gap of 0.8 points or more in either direction would show up in about 17 experiments in 100." If the sentence sounds strange, the reading is probably wrong.
  5. Look at the range of plausible gaps before deciding. The confidence interval and the size of the gap tell you whether the result is worth acting on; the p-value only tells you how surprising it is under "no difference".

The next section applies the five steps to real numbers.

Worked example: 4.0% vs 4.8% with 2,500 visitors per group

Sample data: version A converts 100 of 2,500 visitors (4.0%), version B converts 120 of 2,500 (4.8%). The gap is 0.8 points, a 20% relative lift. Test: two-sided two-proportion z-test.

  • Null hypothesis: the true conversion rate is the same for A and B.
  • Pooled rate under the null: (100 + 120) ÷ 5,000 = 4.4%.
  • Standard error of the gap under the null: √(0.044 × 0.956 × (1/2,500 + 1/2,500)) = 0.0058 (0.58 points).
  • Test statistic: z = 0.008 ÷ 0.0058 = 1.38.
  • p-value (two-sided): 2 × P(Z > 1.38) = 0.168. One-sided it would be 0.084.

The sentence: if A and B really convert the same, random variation alone would produce a gap of 0.8 points or more in either direction about 17% of the time. That is not unusual, so the result is not statistically significant at 0.05. For the gap to reach p = 0.05 with this sample, it would have to be about 1.1 points (for example 4.0% vs 5.1%).

Now the step that most write-ups skip. The 95% confidence interval for the gap is 0.8 ± 1.96 × 0.58 = −0.34 to +1.94 points, or from an 8% relative drop to a 48% relative lift. The data are compatible with B being slightly worse, no different or a lot better. "Not significant" here means "we cannot tell yet", and the test only had about 28% power to detect a true lift this size, so a miss is likely even if B really is better (see Type II error).

Sample size alone changes the p-value for the same observed rates:

Visitors per groupConversions A / BzTwo-sided p
2,500100 / 1201.380.168
5,000200 / 2401.950.051
7,500300 / 3602.390.017
10,000400 / 4802.760.0058
20,000800 / 9603.900.0001

The gap is 0.8 points in every row. Only the amount of data differs, which is why the p-value alone should never be treated as the size of an effect.

app.tapcub.com

Same 0.8-point gap, different sample

two-sided p-value
2,500 per group0.168
5,000 per group0.051
7,500 per group0.017
10,000 per group0.0058
20,000 per group0.0001
Sample data
Sample data: 4.0% vs 4.8% conversion at each sample size, two-sided two-proportion z-test. Bars scale to p = 0.168; highlighted rows are below 0.05. The gap never changes, only the amount of data does.

What are the three most common misreadings?

MisreadingWhy it is wrongCorrect reading, using the example
"p = 0.168 means there is a 16.8% chance the two versions are the same"The p-value is computed assuming they are the same; it cannot be the probability that assumption is true"If they were the same, I would see a gap this large about 17% of the time"
"p < 0.05 means the effect is large or important"A tiny gap with a huge sample is significantWith 500,000 visitors per group, 10.0% vs 10.2% gives p = 0.0009, but the gap is 0.2 points (interval 0.08 to 0.32)
"p > 0.05 means there is no difference"A small sample cannot separate a real gap from noiseAt p = 0.168 the interval still includes a 1.9-point lift; the test was underpowered, not conclusive

A fourth, related error is believing that p < 0.05 means a 95% chance the result will replicate. It does not; the chance of a repeat depends on the true effect and the sample size, which the p-value alone does not reveal.

One more fact helps calibrate. With no real difference at all, about 5% of tests come out significant at 0.05. In a simulation of 3,000 experiments where both versions truly convert at 4.4%, 4.6% reached p < 0.05. So a single significant result against a background of many tests is exactly what chance produces. Running 20 independent tests gives a 1 − 0.95²⁰ = 64% chance of at least one false alarm. The remedy is to decide the number of comparisons in advance and correct the threshold for them, or to confirm any surprise with a second test.

p-valuePlain readingWhat it does not mean
0.168With no real difference, a gap this big or bigger shows up in about 17 of 100 experimentsA 16.8% chance the two versions are the same
0.05The usual cut-off: about 5 in 100 experiments under no differenceA 95% chance the effect is real or will repeat
0.0009Very surprising under no differenceThe effect is large; with 500,000 visitors per group a 0.2-point gap gives this
0.30 with a wide intervalThe data cannot separate a real gap from noiseThere is no effect

Sample data from the worked example; two-sided two-proportion z-test.

What should you look at besides the p-value?

The p-value answers one narrow question, so pair it with three other numbers:

QuestionNumber to readIn the example
How surprising is this gap if there is no difference?p-value0.168
What range of true gaps does the data allow?95% confidence interval−0.34 to +1.94 points
How big is the gap in terms that matter?Effect size (absolute and relative)+0.8 points, +20% relative
Was the test able to see a gap worth acting on?Power or minimum detectable effectAbout 28% power at 2,500 per group; about 10,300 per group for 80%

The ASA statement says the same in its own words: a conclusion should not rest only on whether a p-value passes a threshold, and a p-value by itself is not a good measure of evidence. For reading the interval, see how to calculate a confidence interval; for the gap in practical terms, see effect size. The broader question of what "significant" should mean is in statistical significance.

How do you decide what to do with a result?

Match what you see to one row; do not use the p-value alone:

What you seeWhat it meansWhat to do
p below the threshold, the interval's low end is above your smallest worthwhile gapEvidence of a gap that mattersAct; confirm after rollout
p below the threshold, the interval sits mostly below the smallest worthwhile gapA real but too-small gapUsually do not act unless the change is free
p above the threshold, narrow interval around zeroA gap bigger than the interval's edge is unlikelyTreat as no meaningful difference
p above the threshold, wide interval spanning zero and a worthwhile gapInconclusiveCollect more data or test a bigger change
p just under 0.05 after many looks or comparisonsPossibly noiseTreat as a lead, and re-test

"Smallest worthwhile gap" is a business number you set before the test: what lift would pay for the change?

What are the common mistakes?

  • Peeking and stopping when p dips under 0.05. Checking every day and stopping at the first significant reading raises the false-alarm rate well above 5%. Fix the sample size or run length first.
  • Reporting only the p-value. "p = 0.03" without the gap or the interval cannot be used to decide anything.
  • Treating 0.049 and 0.051 as different worlds. They are nearly the same evidence. The 0.05 line is a convention, not a law of nature.
  • Testing many segments and reporting the one that worked. One in twenty comparisons will be "significant" by chance.
  • Applying a significance test to a non-random comparison. When people chose which version to use themselves, a small p-value tells you the gap is unlikely to be noise, not that the version caused it (see correlation vs causation).

How do you use TapCub to read a p-value?

TapCub does not run A/B experiments or compute p-values, so it has no significance test of its own. It supplies the inputs for the calculation above and keeps the data clean. Here is how the 4.0% vs 4.8% example is set up.

  1. Pull the conversions and visitors for each version. In Web analytics › Goals, read each version's goal conversions and visitors for the same dates. If you send the version name as an event property, you can compare the two groups' counts directly.
  2. Count real people only. TapCub counts bots and AI crawlers separately, so the 2,500 per group are people, not crawlers diluting the conversion rate.
  3. Run the test outside TapCub. Put the four numbers (conversions and visitors per version) into your own test or a calculator, and note the interval as well as the p-value. TapCub only measures; it does not do the test.
  4. Compare the same period in the trend. In All-platform analytics › Behavior › Event analysis, use the compare option to line a period up against the previous one, and mark the change in Configuration › Annotations so later readers know why it moved.
  5. Set a monitor for the metric you care about. In All-platform analytics › Operations › Monitors, a monitor on a built-in metric such as bounce_rate notifies you when it crosses a line. It only notifies; deciding whether the change is real stays with you.

FAQ

What does a p-value of 0.05 mean?

If there were no real difference, you would see a gap at least as large as yours in about 5 of 100 repeated experiments. It is a conventional cut-off for calling a result statistically significant, not a measure of how likely the finding is to be true.

What does a high p-value mean?

A high p-value means the gap you saw is the kind of gap random variation often produces when nothing is going on. It does not prove there is no effect. With a small sample, the interval around the gap can still include large real effects.

Is a p-value the probability that the null hypothesis is true?

No. The p-value is calculated assuming the null hypothesis is true, so it cannot give the probability that it is true. Getting that probability needs prior information and a different method, such as a Bayesian analysis.

Can a p-value be significant but unimportant?

Yes. With enough visitors, a 0.2-point difference in conversion can reach p < 0.001. Read the gap and its interval alongside the p-value, and compare the lower end of the interval with the smallest gap that would pay for the change.

Should I always use 0.05?

No. It is a convention. Choose the threshold before the test based on the cost of a false alarm and of a miss, and say so in the report. If you run many comparisons, tighten it or correct for the number of tests.

Sources

Filed underAnalytics

Was this article helpful?

We use your answers to fix and expand this page.

Xu An

Customer Experience Consultant

Writes about live chat, visitor conversations, and self-service, sharing practical ways to make customer communication clearer and more effective.

Get clean counts for every comparison

Goal conversions and visitors per version, bots and AI crawlers counted separately, so the numbers you feed a test are people.

See your data clearly. Find your growth.

Every click, backed by data. Install one line of code and see your first numbers in a minute.

No credit card · Free plans for analytics and chat · Cookieless analytics