A/B Test Ecommerce Search and Recommendations
Design credible tests for search and recommendations: randomisation, sample size, exposure, uncertainty and commercial guardrails, with clear decision rules.
How to A/B test ecommerce search and recommendations
A test should resolve a decision, not manufacture a winning percentage. Define the change, eligible population, primary outcome and acceptable risks before looking at results. Then check the data, estimate the effect and report the uncertainty alongside what the experience actually changed.
Write the decision before running the test
State the hypothesis and the configuration change. Select one primary business outcome and the guardrails that could stop the release. For search, relevance and latency may be necessary guardrails alongside purchase outcomes. For recommendations, consider whether an apparent basket gain comes with lower completion or poorer contribution.
Choose the randomisation unit to fit the experience. A returning visitor should not be unintentionally switched between variants when the design requires stable assignment. Record inclusion rules before treatment can influence them. Comparing only people who clicked the new module can select a different population from the control.
Plan the sample and the stopping rule
Sample requirements depend on baseline behaviour, variability, the smallest commercially meaningful effect, allocation and analysis method. Use those inputs to plan the test. Set a duration that covers relevant trading patterns and an analysis rule appropriate to how often results will be examined. Repeatedly checking a conventional fixed-horizon test and stopping at the first favourable result can undermine the intended error control.
Check the data before interpreting the result
Confirm that assignment, exposure and outcomes are logged consistently. Investigate sample ratio mismatch rather than dismissing it as random noise. Microsoft’s experimentation guidance describes it as a symptom of possible data-quality problems that requires diagnosis before trusting the result. Primary guidance.
Report effect and uncertainty
Show absolute change and relative change with their denominators. In a synthetic example, 300 purchases from 10,000 visitors is a 3.0% conversion rate; 330 from 10,000 is 3.3%. The difference is 0.3 percentage points and 10% relative. Those counts alone do not establish a decision without the agreed analysis and uncertainty assessment.
A confidence interval is a statement about an estimation procedure's long-run coverage under its assumptions. It is not a certificate that the observed result is certain. NIST explanation.
Make a decision that matches the evidence
A positive point estimate with wide uncertainty may be inconclusive. A statistically detectable effect can still be commercially trivial. Review the primary metric, guardrails and implementation failures together. Record whether the result supports release, further testing or rejection, then preserve the experiment configuration and analysis record.
See the evidence categories used in the measurement standard.
Illustrative example
Synthetic conversion arithmetic: control 300/10000=.03=3.0%; treatment 330/10000=.033=3.3%; absolute difference=.003=0.3 percentage points; relative difference=.003/.03=.10=10%. No confidence interval, power result or statistical significance is asserted.
Downloads
- experiment-plan.csv (illustrative template)
- evidence-record.csv (illustrative template)
Sources
- An Introduction to A/B Testing, checked 2026-10-25
- The Measurement Standard, checked 2026-10-25
- Microsoft: Diagnosing Sample Ratio Mismatch in A/B Testing, checked 2026-10-25
- NIST: What are confidence intervals?, checked 2026-10-25
How Particular Audience does it
Frequently asked questions
Does statistical significance mean the result is certain?
No. Statistical procedures quantify evidence and uncertainty under assumptions. They do not remove logging errors, biased populations or changing conditions. Report the effect, interval or other uncertainty measure and practical importance, then check the design and data before deciding.
Can attributed revenue be called incremental revenue?
Not on attribution evidence alone. Attribution assigns credit using a rule; incrementality asks what changed because of the intervention. Use a valid comparison or other defensible causal method and describe its assumptions before claiming an incremental effect.
How long should an A/B test run?
Determine that from the baseline rate or variance, minimum effect worth detecting, allocation, chosen analysis and relevant trading cycles. Avoid a universal duration. Set the stopping rule in advance and use a method suited to repeated monitoring if that is required.