Particular Audience Particular Audience
Talk to us

AI for search, personalisation and retail media

A different store for every customer.

Bring what each shopper wants together with what your business needs.

Search, recommendations, retail media and promotions across your digital store and AI shopping experiences.

Measure Retail Media Incrementality with Holdouts

Plan a retail media holdout with stable assignment, comparable outcomes and visible uncertainty. Separate incremental sales from attributed ROAS and brand halo.

How to measure retail media incrementality

Retail media incrementality asks how outcomes differ because a campaign was available. A well-designed randomized holdout can estimate that difference by comparing eligible groups assigned before exposure. Define the treatment, outcome and analysis first, then interpret the estimate with its uncertainty and the limits of the implementation.

Specify the change and the counterfactual

Name the campaign, audience eligibility, shopping surfaces and baseline experience. Decide whether the question concerns one campaign, a placement policy or an entire programme. The control should represent the experience that would otherwise occur, including any permitted organic discovery or other advertising.

The IAB incremental-measurement guidance centres credible counterfactuals, bias control and separating signal from noise. Use those principles to choose the method; a dashboard labelled “incremental” is not a substitute for a defensible design.

Assign eligible units before the campaign can affect them

Choose a randomisation unit that fits the journey, such as a shopper or a store, and keep assignment stable where the design requires it. Analyse according to assignment rather than selecting only treatment shoppers who clicked or bought. That selection can destroy the comparability created by randomisation.

Agree how shared campaign budgets, cross-device identities, stock constraints and overlapping campaigns could affect the groups. Record contamination and delivery failures. If store-level assignment is used, the analysis must account for that clustering rather than treating every shopper as an independently randomized unit.

Compare rates or value on a consistent basis

A synthetic arithmetic example has 10,000 assigned eligible shoppers in each group. Treatment net sales are £115,000 and control net sales are £100,000: £11.50 versus £10 per assigned shopper. The difference is £1.50, or £15,000 when applied to the treatment group. This demonstrates an estimator's arithmetic; it is not a measured case or an uncertainty calculation.

Keep returns, cancellations, observation windows and product scope consistent. Report product, agreed brand scope and category outcomes separately. The advertised item may gain sales partly by displacing another item, so the product result alone cannot explain retailer value.

Report uncertainty and operational checks

Plan sample requirements around the baseline, variability and smallest useful effect. State the estimator and stopping rule. Inspect assignment ratios and event quality before discussing uplift. Report an interval or other appropriate uncertainty assessment alongside the estimate, and distinguish an inconclusive result from evidence of no commercially relevant effect.

Do not use attributed ROAS as holdout proof

The Intersport case explicitly describes sequential periods rather than a randomized holdout. It belongs in attribution and rollout context, not as proof that this experimental method was deployed. A brand-halo ROAS figure also remains an attribution measure unless the underlying calculation uses a causal estimate.

Bring the agreed design to a measurement scoping discussion. The useful output is a decision to expand, change or stop, supported by the treatment definition, data checks, uncertainty and commercial trade-offs.

Illustrative example

SYNTHETIC: treatment 115000 GBP/10000 shoppers=11.50; control100000/10000=10; difference1.50; effect on treatment group15000 GBP. No interval, p-value or real-world causal result is asserted.

Sources

How Particular Audience does it

In production

Frequently asked questions

Should we compare people who saw an ad with people who did not?

Not automatically. Exposure can depend on shopping activity, relevance, budgets and other factors related to purchasing. A randomized eligibility or assignment design preserves a more credible comparison. Analyse the assigned groups according to the agreed method rather than filtering to exposed or clicking shoppers.

Does a holdout sacrifice a known amount of revenue?

The effect of withholding exposure is the question being measured, so its revenue cost is not known in advance. Discuss the opportunity cost, sample requirements and duration with commercial and analytics owners. A tiny holdout can also produce evidence too uncertain to support the intended decision.

Can brand-halo ROAS prove incrementality?

No. It may attribute sales across an agreed set of related products, but that does not establish which sales the campaign caused. Estimate incremental outcomes with a suitable comparison and define the halo product scope separately so the two ideas are not confused.

Related guides

How-to guides · All guides