P-hacking: what it is, examples and how to avoid it

What is p-hacking?

P-hacking is analyzing data in many different ways and reporting the version that crosses the significance threshold (usually p < 0.05), without accounting for all the attempts. Each choice may look reasonable on its own: one more day of data, one more segment, dropping a few outliers. Together they make "significant" results far more likely than 5% even when nothing real is going on.

It is also called data dredging, data fishing or significance chasing. It is usually not fraud; it is the natural result of looking for a story in noisy data under pressure to find one.

Common forms of p-hacking

How much it inflates false positives

We simulated an A/A test: two identical variants, both converting at 5%, 500 visitors per variant per day for 20 days, repeated 4,000 times (sample simulation).

Peeking alone multiplies false wins by about five. Testing many outcomes does the same arithmetic in another dimension: with 20 independent metrics and no real effect, the chance that at least one is significant at 5% is 1 − 0.95²⁰ = 64%; with 10 segments it is 40%.

Those false wins then get shipped, and the expected lift never shows up in the following quarter.

Signs of p-hacking

• Many results with p just under 0.05 (0.041, 0.048) and very few just above.
• The "key metric" in the write-up is not the one named before the test.
• Effects found only in a narrow, oddly specific segment.
• Sample sizes that are not round or pre-planned ("we ran it for 11 days").
• Results that do not replicate when the same change is tested again.

How to avoid it

1. Decide before you look. Write down the primary metric, the segments, the sample size or duration, and the analysis method. This is pre-registration in a sentence or two.
2. Use a fixed horizon, or a method built for peeking. If you must monitor continuously, use a sequential test or alpha-spending approach designed for repeated looks.
3. Correct for multiple comparisons. Adjust thresholds when you test many metrics or segments, for example with a Bonferroni correction or a false discovery rate method.
4. Separate exploration from confirmation. Exploring dashboards is fine; label what you find as a hypothesis and confirm it on new data or a new test.
5. Report everything. Say how many metrics and segments you looked at, and report p-values together with the size of the difference and its interval.
6. Plan for enough power. Underpowered tests tempt people to keep going until something shows up; size tests with a minimum detectable effect.

P-hacking in everyday analytics

Most p-hacking in companies happens outside formal experiments: comparing this week to last week across ten channels, slicing a dip by every dimension until one looks guilty, or re-running a cohort report with different windows. A few habits help:

• Keep a short list of the metrics that matter and look at them first.
• When a slice looks dramatic, check its size; a 300-visitor segment swings by chance.
• Confirm a finding over a second period before acting.
• Pair every primary metric with guardrail metrics that you did not choose after the fact.

In TapCub you can save a dashboard or report view before a launch, so the comparison you planned is the one you read afterwards; annotations on the chart record when the change shipped.

See also: What is sequential testing? Peeking, methods and an example.

Sources

• Simmons, J. P., Nelson, L. D. and Simonsohn, U. (2011). False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11).
• Head, M. L. et al. (2015). The extent and consequences of p-hacking in science. PLoS Biology, 13(3).
• Wasserstein, R. L. and Lazar, N. A. (2016). The ASA statement on p-values. The American Statistician, 70(2).