What Is a Holdout Group?

A holdout group is a small segment of users intentionally excluded from marketing campaigns, experiments, or feature tests. This group receives no exposure to the change being tested and serves as a control group for comparison. By measuring performance differences between the holdout group and the exposed group, marketers can isolate the true impact of their campaigns or product updates.

How Are Holdout Groups Used in Testing or Marketing?

Holdout groups are used to evaluate whether changes in metrics—like conversions, retention, or revenue—are actually caused by the campaign or just natural fluctuations. Common use cases include A/B testing, long-term incrementality studies, and assessing the combined lift of multiple experiments. Typically, only 1% to 10% of users are held out to preserve sample size without disrupting performance.