How to Evaluate Ecommerce Search with Reliable Evidence
Evaluate ecommerce search with representative queries, calibrated relevance labels and clearly defined metrics. Validate commercial impact with a planned live test.
How to evaluate ecommerce site search
Evaluate search by comparing engines on the same representative queries, catalogue context and eligibility rules. Use an explicit relevance rubric, calibrate human and model judgments, and report several metrics by query group. Follow offline diagnosis with a well-designed live experiment to estimate the commercial effect for your retailer.
Define the decision before collecting scores
Decide whether you are comparing retrieval, ranking, merchandising configuration or the complete search experience. Record which components change between variants. A test that changes several components can support a decision about the bundle, but cannot isolate every component’s contribution.
Sample real head, mid-frequency and rare queries across categories, languages and devices. Include exact identifiers, modifiers, misspellings, negative requirements and no-match cases. Freeze the catalogue context, filters, availability rules and result depth for the offline comparison. Document frequency weighting rather than letting the easiest queries dominate.
Make relevance judgments reproducible
Write what each grade means, including how brand, size, compatibility and missing evidence affect the decision. Hide engine identity from graders. Build an adjudicated human reference subset and compare automated labels against it by query type.
An LLM can assist at scale, but its reliability must be checked. DoorDash’s evaluation account illustrates how explicit intent facets and resolved boundary cases improve the evaluation process. Save model, prompt and rubric versions; repeat a subset and inspect disagreements instead of assuming consistency.
Use metrics that answer distinct questions
Precision@k describes relevant items among the evaluated top positions. Recall@k depends on a defined set of relevant candidates; when judgments cover a pool, label it pooled recall rather than complete-catalogue recall. Reciprocal rank focuses on the first relevant result. nDCG measures graded, position-sensitive ranking quality relative to an ideal ordering; it is not the percentage of products judged relevant. Metric documentation also makes unlabeled-result handling explicit.
Show both weighted and per-query views, with severe failures and constraint violations alongside averages. Retain returned IDs and judgments so another reviewer can inspect a score.
Confirm the commercial decision live
Predefine the assignment unit, primary outcome, guardrails, sample plan and stopping rule. Check event integrity and allocation before interpreting results. Report effects and uncertainty rather than a single winning percentage.
PA’s Shopify bikes case reports AOV of A$241.82 versus A$223.89 in a 25-day split test covering 590,878 visitors. That is an approximately 8.0% relative difference in one retailer’s test; the published AOV win chance was 92.8%. It does not establish the result a different retailer will achieve.
Use the offline findings to explain failures and the live findings to decide whether the tested experience earns a wider rollout.
Illustrative example
{"label":"Synthetic relevance-metric fixture","grades":[4,2,0,0,0,0,0,0,0,0],"relevant_threshold":2,"judged_relevant_universe":2,"precision_at_10":0.2,"pooled_recall_at_10":1,"mrr":1,"ndcg_at_10":1,"gain":"2^grade - 1","discount":"log2(rank + 1)","no_relevant_items_policy":"nDCG not applicable"}
Evidence
- PA reports A$241.82 AOV versus A$223.89 for the incumbent, approximately 8.0% higher, in one 25-day Shopify bikes test. (Vendor-published single-retailer test; AOV win chance 92.8%; no universal guarantee or same-confidence claim for all KPIs.) Source
Sources
- LLM-as-a-Judge: Evaluating natural language search, checked 2026-10-25
- Evaluate ranked search results, checked 2026-10-25
How Particular Audience does it
Frequently asked questions
How many queries are enough?
Choose coverage based on the decision, traffic mix and important failure classes. A convenient total is not a validity guarantee. Record sampling and weighting, then inspect whether material query groups have enough evidence.
Can an LLM provide objective relevance grades automatically?
It can produce useful judgments, but needs an explicit rubric, evidence, calibration and repeatability checks. Human reviewers also need calibration; unresolved disagreement should remain visible rather than being forced into a confident score.
Does better nDCG prove higher revenue?
No. nDCG describes ordering against relevance judgments. Revenue also depends on availability, price, presentation and shopper behaviour. Estimate commercial effects with a separately designed live comparison.
Related guides
- How to use an LLM as a judge for ecommerce search: Build a calibrated LLM judge for ecommerce search. Define intent facets, create reference labels, test agreement and preserve evidence before trusting scores.
- How to reduce zero-result searches: Diagnose zero-result searches using genuine query events, catalogue checks and constraint-aware recovery. Measure useful results without hiding failed searches.
- Vector search vs keyword search for ecommerce: Compare vector, keyword and hybrid ecommerce search. Understand exact matches, semantic recall, filters and ranking, then evaluate them on your own catalogue.