Measure Retail Media Incrementality with Holdouts
Plan a retail media holdout with stable assignment, comparable outcomes and visible uncertainty. Separate incremental sales from attributed ROAS and brand halo.
How to measure retail media incrementality
Retail media incrementality asks how outcomes differ because a campaign was available. A well-designed randomized holdout can estimate that difference by comparing eligible groups assigned before exposure. Define the treatment, outcome and analysis first, then interpret the estimate with its uncertainty and the limits of the implementation.
Specify the change and the counterfactual
Name the campaign, audience eligibility, shopping surfaces and baseline experience. Decide whether the question concerns one campaign, a placement policy or an entire programme. The control should represent the experience that would otherwise occur, including any permitted organic discovery or other advertising.
The IAB incremental-measurement guidance centres credible counterfactuals, bias control and separating signal from noise. Use those principles to choose the method; a dashboard labelled “incremental” is not a substitute for a defensible design.
Assign eligible units before the campaign can affect them
Choose a randomisation unit that fits the journey, such as a shopper or a store, and keep assignment stable where the design requires it. Analyse according to assignment rather than selecting only treatment shoppers who clicked or bought. That selection can destroy the comparability created by randomisation.
Agree how shared campaign budgets, cross-device identities, stock constraints and overlapping campaigns could affect the groups. Record contamination and delivery failures. If store-level assignment is used, the analysis must account for that clustering rather than treating every shopper as an independently randomized unit.
Compare rates or value on a consistent basis
A synthetic arithmetic example has 10,000 assigned eligible shoppers in each group. Treatment net sales are £115,000 and control net sales are £100,000: £11.50 versus £10 per assigned shopper. The difference is £1.50, or £15,000 when applied to the treatment group. This demonstrates an estimator's arithmetic; it is not a measured case or an uncertainty calculation.
Keep returns, cancellations, observation windows and product scope consistent. Report product, agreed brand scope and category outcomes separately. The advertised item may gain sales partly by displacing another item, so the product result alone cannot explain retailer value.
Report uncertainty and operational checks
Plan sample requirements around the baseline, variability and smallest useful effect. State the estimator and stopping rule. Inspect assignment ratios and event quality before discussing uplift. Report an interval or other appropriate uncertainty assessment alongside the estimate, and distinguish an inconclusive result from evidence of no commercially relevant effect.
Do not use attributed ROAS as holdout proof
The Intersport case explicitly describes sequential periods rather than a randomized holdout. It belongs in attribution and rollout context, not as proof that this experimental method was deployed. A brand-halo ROAS figure also remains an attribution measure unless the underlying calculation uses a causal estimate.
Bring the agreed design to a measurement scoping discussion. The useful output is a decision to expand, change or stop, supported by the treatment definition, data checks, uncertainty and commercial trade-offs.
Illustrative example
SYNTHETIC: treatment 115000 GBP/10000 shoppers=11.50; control100000/10000=10; difference1.50; effect on treatment group15000 GBP. No interval, p-value or real-world causal result is asserted.
Sources
- Guidelines for Incremental Measurement in Commerce Media, checked 2026-10-25
- Intersport Austria retail-media case, checked 2026-10-25
- Measurement and Incrementality, checked 2026-10-25
How Particular Audience does it
In production
- Intersport Austria: Intersport Austria: five-week onsite-media launch and attributed brand-halo reporting (17.3× Reported attributed brand-halo ROAS (sequential periods, not a holdout))
Frequently asked questions
Should we compare people who saw an ad with people who did not?
Not automatically. Exposure can depend on shopping activity, relevance, budgets and other factors related to purchasing. A randomized eligibility or assignment design preserves a more credible comparison. Analyse the assigned groups according to the agreed method rather than filtering to exposed or clicking shoppers.
Does a holdout sacrifice a known amount of revenue?
The effect of withholding exposure is the question being measured, so its revenue cost is not known in advance. Discuss the opportunity cost, sample requirements and duration with commercial and analytics owners. A tiny holdout can also produce evidence too uncertain to support the intended decision.
Can brand-halo ROAS prove incrementality?
No. It may attribute sales across an agreed set of related products, but that does not establish which sales the campaign caused. Estimate incremental outcomes with a suitable comparison and define the halo product scope separately so the two ideas are not confused.
Related guides
- Retail media measurement: incrementality, attribution and the metrics that matter: How to measure retail media: closed-loop attribution, holdout incrementality, halo, new-to-brand and the effect of ads on the retailer's own conversion.
- Incrementality in retail media: Incrementality estimates what advertising changes beyond the counterfactual. Learn why exposure comparisons, attributed ROAS and sequential periods need care.
- Holdout tests in advertising: A holdout assigns an eligible group to a comparison experience. Learn how stable assignment, valid outcomes and uncertainty support advertising measurement.
- Brand halo in retail media: Define brand halo by product and category scope. Distinguish attributed halo ROAS from incremental halo sales and avoid double counting parent-product sales.