Quick answer
A holdout group is a randomly chosen part of an audience that deliberately does not get a campaign, feature or change, so its results can be compared with everyone else's. The gap is what the change caused; without a holdout you cannot separate it from what would have happened anyway.
What is a holdout group?
A holdout group is a randomly chosen part of an audience that is deliberately kept out of a campaign, a feature or a change, so its results can be compared with those of everyone who received it. The gap between the two groups is what the change actually caused. Without a holdout you cannot tell that apart from what would have happened anyway: people who were about to buy, return or upgrade do so whether or not you contact them.
The phrase "hold out" on its own is an ordinary verb, meaning to last or to wait for a better deal; this page is about the measurement tool. Holdouts are used in two settings: campaign holdouts (ads, email, push messages) and universal or global holdouts in product work, where a small share of users stays on the old experience while new features ship. If you want the marketing framing of the same idea, see incrementality.
Holdout group vs control group vs A/B test
| Campaign holdout | A/B test control | Universal holdout | |
|---|---|---|---|
| What is withheld | One campaign or message | The variant being tested | Everything shipped over a period |
| How long it lasts | The campaign | One experiment | An extended period, across many releases |
| What it measures | The extra result the campaign caused | The difference between variants | The combined effect of all changes, including interactions |
| Typical size | Chosen from the detectable effect (see below) | Split of the test traffic | Small: Amplitude suggests 1% to 10% of traffic, Spotify describes 1% to 5% as typical |
The universal holdout answers a question that single tests cannot: ten features that each won their own test can interact badly, and only a group that saw none of them shows the net result. Spotify's glossary also lists its limits: a small holdout can miss modest effects, holdout users do not get improvements (so fixes and compliance changes are usually exempted), and over time the group drifts so far from the live product that it may need to be released and recreated. A holdout is a design choice, not an experimentation platform.
Worked example: a win-back email with a 10% holdout
Sample data: 20,000 users who have been inactive for 90 days. A random 10% is held out; the other 90% get a win-back email. Success means returning within 14 days.
| Group | People | Returned | Return rate |
|---|---|---|---|
| Sent the email | 18,000 | 1,080 | 6.0% |
| Holdout, no email | 2,000 | 90 | 4.5% |
| Difference | +1.5 points |
How to recompute: 1,080 ÷ 18,000 = 6.0%; 90 ÷ 2,000 = 4.5%; 6.0% − 4.5% = 1.5 points. Relative lift = 1.5 ÷ 4.5 = 33.3%. Incremental returns = 0.015 × 18,000 = 270. Share caused by the email = 270 ÷ 1,080 = 25.0%; the other 810 would have returned anyway.
A two-proportion test (the method in the NIST/SEMATECH handbook) puts numbers on the noise: the pooled rate is 1,170 ÷ 20,000 = 5.85%, the standard error is about 0.00553, so z = 0.015 ÷ 0.00553 ≈ 2.71 and p ≈ 0.007. The 95% confidence interval for the difference is about +0.5 to +2.5 points. A dashboard without a holdout would credit the email with all 1,080 returns, four times the 270 it caused.
The holdout has a price: 2,000 × 1.5% = 30 returns that did not happen because those people were not emailed. That is the cost of knowing.
How big should a holdout be?
Size it from the smallest difference you would act on. Sample data, same 20,000 users and the same 4.5% baseline, at 80% power and a 5% significance level:
| Holdout share | Sent | Holdout | Smallest difference detectable |
|---|---|---|---|
| 2% | 19,600 | 400 | 2.95 points |
| 5% | 19,000 | 1,000 | 1.90 points |
| 10% | 18,000 | 2,000 | 1.39 points |
| 20% | 16,000 | 4,000 | 1.05 points |
| 50% | 10,000 | 10,000 | 0.86 points |
How to recompute: the detectable difference d solves d = (1.96 + 0.84) × √(0.045 × 0.955 ÷ n_holdout + (0.045 + d)(0.955 − d) ÷ n_sent). With equal groups, detecting 4.5% against 6.0% needs about 3,463 people in each. In the table, a 2% holdout can only reveal a gap of almost 3 points, nearly double the 1.5-point gap in this example; the 10% holdout can see 1.4 points, which is why the example's 1.5-point gap is detectable (power about 86%).
Two rules follow. If the smallest effect you care about is below what your holdout can detect, a "no difference" result means nothing; enlarge the holdout, extend the audience or accept a coarser answer. And every extra point of holdout is people denied the treatment, so stop at the size that detects the effect you need. Measuring advertising returns is notoriously noisy: Lewis and Rao (2015) showed that sales are so variable relative to the effect of one campaign that experiments often need very large samples. See also minimum detectable effect.
How to set up a holdout that stays honest
- Randomise before delivery. Assign by a random draw or a hash of the ID, never by convenience: "people without an email address" or "the last 10% of the list" correlate with behaviour and bias the result.
- Fix the size and the assignment. Do not change the holdout share after launch; Amplitude's tool does not allow it. A person should stay in the same group for the whole period.
- Block every route to the holdout. If holdout members get the same offer by another channel, the gap shrinks.
- Decide the metric and the window first. For example: returned within 14 days of the send date.
- Check the split matches the plan. Planned 18,000 and 2,000; if you see 17,200 and 2,800, find out why before reading results.
- Exempt what people need. For universal holdouts, bug fixes and compliance changes usually go to everyone.
- Retire old holdouts. A holdout kept for a year can drift from the live product; plan when to release it.
Common mistakes with holdout groups
- Calling "people who did not see it" a holdout. If they were not targeted, or chose not to open the email, they differ from the exposed group before the test starts; that is selection bias, not a holdout.
- Reading a small holdout's "no difference" as "no effect". It may only mean the test could not see an effect of that size.
- Stopping when the gap looks good. Checking daily and stopping at the first significant day inflates false positives; see sequential testing.
- Trusting platform-reported conversions alone. Platforms credit every conversion from someone who saw the ad; a holdout shows how many were caused. The retargeting page works this through for ads.
- Using the holdout as a target. Its job is to measure, not to be a list you also mail later in the same test.
Holdout groups in TapCub
Suppose the 20,000 lapsed users are identified people in TapCub. TapCub does not send the email or the ads, and it does not split traffic or run A/B tests. It can hold the two groups and measure them.
- Build the audience. In All-platform analytics › Users › Segments & tags, create a rule segment for the lapsed users, such as "did
purchaseat least once" and "did notpurchasein the last 90 days". Preview the count. - Split off the holdout. Add the random bucket condition. The console takes two numbers from 0 to 99 and treats them as a percentage range, and the bucket is worked out from the person's ID, so the same person lands in the same bucket on every refresh. Make two rule segments, each with the audience segment as a condition: bucket 0–9 for the holdout (about 10%) and bucket 10–99 for the send group.
- Use the lists outside TapCub. View users opens the members, which you take to your email or ad tool; the holdout segment is the one you leave out.
- Measure. Open each segment with Analyze to compare the same event, such as returning in the 14 days after the send, for the two groups. The z-test in this page is done outside TapCub.
- Know the limits. Anonymous visitors have no profile and never join a segment, and person-scoped segments have no members unless identified mode is on.
See also: How to calculate incrementality with a holdout group.
See also: Feature Flagging: What Feature Flags Are and How to Roll Out.
Sources
- Spotify Confidence, What is a holdout group?: holdouts vs A/B test controls, cumulative effect and limitations.
- Amplitude Docs, Holdout groups: purpose, 1% to 10% size guidance, and rules for adding experiments.
- NIST/SEMATECH e-Handbook of Statistical Methods, 7.3.3 How can we determine whether two processes produce the same proportion of defectives?: the two-proportion test used in the example.
- Lewis, R. A., and Rao, J. M. (2015). The Unfavorable Economics of Measuring the Returns to Advertising. Quarterly Journal of Economics, 130(4).
Was this explanation clear?
We use your answers to fix and expand this page.
Couldn’t send. Please try again in a minute.
Thanks for the feedback. We’ll use it to improve this page.You’ve already sent feedback on this page. Thanks.

Product Analyst
Focuses on event analysis, conversion funnels, and retention, using clear metric definitions and structured analysis to explain product performance.