TapCub is live — analytics, insights and live chat, free on one platform
Glossary

Holdout group: definition, how big it should be and a worked example

A holdout group is a random part of the audience that deliberately does not get a campaign or change, so you can see what it caused. A worked email example, how holdout size sets the detectable effect, and set-up rules.

Emma Collins Reviewed by Su He Also called holdout Last updated Nov 17, 2026 Glossary

Quick answer

A holdout group is a randomly chosen part of an audience that deliberately does not get a campaign, feature or change, so its results can be compared with everyone else's. The gap is what the change caused; without a holdout you cannot separate it from what would have happened anyway.

What is a holdout group?

A holdout group is a randomly chosen part of an audience that is deliberately kept out of a campaign, a feature or a change, so its results can be compared with those of everyone who received it. The gap between the two groups is what the change actually caused. Without a holdout you cannot tell that apart from what would have happened anyway: people who were about to buy, return or upgrade do so whether or not you contact them.

The phrase "hold out" on its own is an ordinary verb, meaning to last or to wait for a better deal; this page is about the measurement tool. Holdouts are used in two settings: campaign holdouts (ads, email, push messages) and universal or global holdouts in product work, where a small share of users stays on the old experience while new features ship. If you want the marketing framing of the same idea, see incrementality.

Holdout group vs control group vs A/B test

Campaign holdoutA/B test controlUniversal holdout
What is withheldOne campaign or messageThe variant being testedEverything shipped over a period
How long it lastsThe campaignOne experimentAn extended period, across many releases
What it measuresThe extra result the campaign causedThe difference between variantsThe combined effect of all changes, including interactions
Typical sizeChosen from the detectable effect (see below)Split of the test trafficSmall: Amplitude suggests 1% to 10% of traffic, Spotify describes 1% to 5% as typical

The universal holdout answers a question that single tests cannot: ten features that each won their own test can interact badly, and only a group that saw none of them shows the net result. Spotify's glossary also lists its limits: a small holdout can miss modest effects, holdout users do not get improvements (so fixes and compliance changes are usually exempted), and over time the group drifts so far from the live product that it may need to be released and recreated. A holdout is a design choice, not an experimentation platform.

Worked example: a win-back email with a 10% holdout

Sample data: 20,000 users who have been inactive for 90 days. A random 10% is held out; the other 90% get a win-back email. Success means returning within 14 days.

GroupPeopleReturnedReturn rate
Sent the email18,0001,0806.0%
Holdout, no email2,000904.5%
Difference+1.5 points

How to recompute: 1,080 ÷ 18,000 = 6.0%; 90 ÷ 2,000 = 4.5%; 6.0% − 4.5% = 1.5 points. Relative lift = 1.5 ÷ 4.5 = 33.3%. Incremental returns = 0.015 × 18,000 = 270. Share caused by the email = 270 ÷ 1,080 = 25.0%; the other 810 would have returned anyway.

A two-proportion test (the method in the NIST/SEMATECH handbook) puts numbers on the noise: the pooled rate is 1,170 ÷ 20,000 = 5.85%, the standard error is about 0.00553, so z = 0.015 ÷ 0.00553 ≈ 2.71 and p ≈ 0.007. The 95% confidence interval for the difference is about +0.5 to +2.5 points. A dashboard without a holdout would credit the email with all 1,080 returns, four times the 270 it caused.

The holdout has a price: 2,000 × 1.5% = 30 returns that did not happen because those people were not emailed. That is the cost of knowing.

How big should a holdout be?

Size it from the smallest difference you would act on. Sample data, same 20,000 users and the same 4.5% baseline, at 80% power and a 5% significance level:

Holdout shareSentHoldoutSmallest difference detectable
2%19,6004002.95 points
5%19,0001,0001.90 points
10%18,0002,0001.39 points
20%16,0004,0001.05 points
50%10,00010,0000.86 points

How to recompute: the detectable difference d solves d = (1.96 + 0.84) × √(0.045 × 0.955 ÷ n_holdout + (0.045 + d)(0.955 − d) ÷ n_sent). With equal groups, detecting 4.5% against 6.0% needs about 3,463 people in each. In the table, a 2% holdout can only reveal a gap of almost 3 points, nearly double the 1.5-point gap in this example; the 10% holdout can see 1.4 points, which is why the example's 1.5-point gap is detectable (power about 86%).

Two rules follow. If the smallest effect you care about is below what your holdout can detect, a "no difference" result means nothing; enlarge the holdout, extend the audience or accept a coarser answer. And every extra point of holdout is people denied the treatment, so stop at the size that detects the effect you need. Measuring advertising returns is notoriously noisy: Lewis and Rao (2015) showed that sales are so variable relative to the effect of one campaign that experiments often need very large samples. See also minimum detectable effect.

How to set up a holdout that stays honest

  1. Randomise before delivery. Assign by a random draw or a hash of the ID, never by convenience: "people without an email address" or "the last 10% of the list" correlate with behaviour and bias the result.
  2. Fix the size and the assignment. Do not change the holdout share after launch; Amplitude's tool does not allow it. A person should stay in the same group for the whole period.
  3. Block every route to the holdout. If holdout members get the same offer by another channel, the gap shrinks.
  4. Decide the metric and the window first. For example: returned within 14 days of the send date.
  5. Check the split matches the plan. Planned 18,000 and 2,000; if you see 17,200 and 2,800, find out why before reading results.
  6. Exempt what people need. For universal holdouts, bug fixes and compliance changes usually go to everyone.
  7. Retire old holdouts. A holdout kept for a year can drift from the live product; plan when to release it.

Common mistakes with holdout groups

  • Calling "people who did not see it" a holdout. If they were not targeted, or chose not to open the email, they differ from the exposed group before the test starts; that is selection bias, not a holdout.
  • Reading a small holdout's "no difference" as "no effect". It may only mean the test could not see an effect of that size.
  • Stopping when the gap looks good. Checking daily and stopping at the first significant day inflates false positives; see sequential testing.
  • Trusting platform-reported conversions alone. Platforms credit every conversion from someone who saw the ad; a holdout shows how many were caused. The retargeting page works this through for ads.
  • Using the holdout as a target. Its job is to measure, not to be a list you also mail later in the same test.

Holdout groups in TapCub

Suppose the 20,000 lapsed users are identified people in TapCub. TapCub does not send the email or the ads, and it does not split traffic or run A/B tests. It can hold the two groups and measure them.

  1. Build the audience. In All-platform analytics › Users › Segments & tags, create a rule segment for the lapsed users, such as "did purchase at least once" and "did not purchase in the last 90 days". Preview the count.
  2. Split off the holdout. Add the random bucket condition. The console takes two numbers from 0 to 99 and treats them as a percentage range, and the bucket is worked out from the person's ID, so the same person lands in the same bucket on every refresh. Make two rule segments, each with the audience segment as a condition: bucket 0–9 for the holdout (about 10%) and bucket 10–99 for the send group.
  3. Use the lists outside TapCub. View users opens the members, which you take to your email or ad tool; the holdout segment is the one you leave out.
  4. Measure. Open each segment with Analyze to compare the same event, such as returning in the 14 days after the send, for the two groups. The z-test in this page is done outside TapCub.
  5. Know the limits. Anonymous visitors have no profile and never join a segment, and person-scoped segments have no members unless identified mode is on.

See also: How to calculate incrementality with a holdout group.

See also: Feature Flagging: What Feature Flags Are and How to Roll Out.

Sources

Was this explanation clear?

We use your answers to fix and expand this page.

Emma Collins

Product Analyst

Focuses on event analysis, conversion funnels, and retention, using clear metric definitions and structured analysis to explain product performance.

FAQ

Questions people ask

Short answers to the follow-up questions that come up most.

Still have a question?

Chat with the team behind TapCub — we usually reply within a few hours on business days.

What is a holdout group?

A holdout group is a random share of an audience kept out of a campaign, feature or change. Comparing it with the group that received the change shows the extra result the change caused.

How big should a holdout group be?

Size it from the smallest difference you would act on. Amplitude suggests 1% to 10% of traffic for universal holdouts. In the sample here, a 10% holdout of 20,000 users could detect a 1.4-point gap, while 2% could only detect almost 3 points.

What is the difference between a holdout group and a control group?

A control group lasts as long as one A/B test and is compared with one variant. A holdout, especially a universal one, can stay out of many releases and show the combined effect of everything shipped.

Why not just compare people who did and did not see the message?

They differ before the test starts: people who open or are targeted are often already more likely to act. Only a randomly assigned holdout removes that selection bias.

See this number on your own site

Add one script tag for web and apps, and the first reports fill in within minutes.

See your data clearly. Find your growth.

Every click, backed by data. Install one line of code and see your first numbers in a minute.

No credit card · Free plans for analytics and chat · Cookieless analytics