Holdout tests in advertising

A holdout test withholds a defined treatment from a randomly assigned comparison group. Comparing outcomes by assignment can estimate the treatment’s effect when the design and analysis are valid. It does not reveal a certain result simply because one group did not see an advertisement.

Hold out a defined treatment

Specify exactly what is withheld: one campaign, a placement policy or another intervention. Define the control experience too. A campaign holdout may still encounter organic products or unrelated advertising; that does not invalidate the design when it is part of the stated counterfactual.

The IAB measurement guidance treats credible counterfactuals and bias control as central to evaluating incremental effects. A randomized holdout is one way to establish that comparison, subject to correct implementation.

Preserve the comparison created by assignment

Assign eligible units before the treatment influences their behaviour. Keep the unit stable across the relevant period and analyse it at the appropriate level. A store-randomized test needs a method that accounts for stores; a shopper-randomized test should not later retain only people who clicked.

In a synthetic allocation, 900 shoppers are assigned treatment and 100 control. Comparing total sales would favour the larger group even with identical purchasing. Compare a suitable normalized outcome and use the planned estimator. This illustrative allocation is not a recommendation for sample size or an assurance of statistical power.

Interpret an estimate, not certainty

Check delivery, group balance, event loss and overlapping campaigns. Shared budgets or stock limits can cause interference between groups. Report those limitations together with the outcome estimate and uncertainty.

A small holdout is not automatically cheap or sufficient. Its information value and commercial trade-off depend on the question. Use the incrementality guide to plan the sample, stopping rule, outcome definitions and responsibilities before launch.

Illustrative example

SYNTHETIC allocation:900 treatment+100 control=1000 assigned shoppers. This shows unequal groups and the need for a suitable normalized estimator, not a sample recommendation.

Sources

• Guidelines for Incremental Measurement in Commerce Media, checked 2026-10-25

How Particular Audience does it

• Measurement

Frequently asked questions

Does the holdout have to receive no advertising at all?

No. It must receive the control experience defined for the question. A campaign-level holdout can still see other campaigns or organic products. Document those conditions and any contamination so the estimated effect is interpreted as the difference actually tested.

How large should the holdout be?

Choose the allocation and sample using baseline behaviour, variability, the smallest useful effect and the analysis design. There is no universal percentage in this guide. An analyst should review power, trading cycles and operational constraints before the business commits to a test.

Why analyse assigned shoppers rather than only exposed shoppers?

Assignment establishes the comparison before the treatment can change behaviour. Exposure or clicks can select people with different purchasing intent or activity. Restricting analysis to those groups can introduce bias unless a justified method explicitly accounts for the selection.

Related guides

• Retail media measurement: incrementality, attribution and the metrics that matter: How to measure retail media: closed-loop attribution, holdout incrementality, halo, new-to-brand and the effect of ads on the retailer's own conversion.
• Incrementality in retail media: Incrementality estimates what advertising changes beyond the counterfactual. Learn why exposure comparisons, attributed ROAS and sequential periods need care.
• How to measure retail media incrementality: Plan a retail media holdout with stable assignment, comparable outcomes and visible uncertainty. Separate incremental sales from attributed ROAS and brand halo.

Glossary · All guides