Particular Audience Particular Audience
Talk to us

AI for search, personalisation and retail media

A different store for every customer.

Bring what each shopper wants together with what your business needs.

Search, recommendations, retail media and promotions across your digital store and AI shopping experiences.

How to Evaluate Ecommerce Search with Reliable Evidence

Evaluate ecommerce search with representative queries, calibrated relevance labels and clearly defined metrics. Validate commercial impact with a planned live test.

How to evaluate ecommerce site search

Evaluate search by comparing engines on the same representative queries, catalogue context and eligibility rules. Use an explicit relevance rubric, calibrate human and model judgments, and report several metrics by query group. Follow offline diagnosis with a well-designed live experiment to estimate the commercial effect for your retailer.

Define the decision before collecting scores

Decide whether you are comparing retrieval, ranking, merchandising configuration or the complete search experience. Record which components change between variants. A test that changes several components can support a decision about the bundle, but cannot isolate every component’s contribution.

Sample real head, mid-frequency and rare queries across categories, languages and devices. Include exact identifiers, modifiers, misspellings, negative requirements and no-match cases. Freeze the catalogue context, filters, availability rules and result depth for the offline comparison. Document frequency weighting rather than letting the easiest queries dominate.

Make relevance judgments reproducible

Write what each grade means, including how brand, size, compatibility and missing evidence affect the decision. Hide engine identity from graders. Build an adjudicated human reference subset and compare automated labels against it by query type.

An LLM can assist at scale, but its reliability must be checked. DoorDash’s evaluation account illustrates how explicit intent facets and resolved boundary cases improve the evaluation process. Save model, prompt and rubric versions; repeat a subset and inspect disagreements instead of assuming consistency.

Use metrics that answer distinct questions

Precision@k describes relevant items among the evaluated top positions. Recall@k depends on a defined set of relevant candidates; when judgments cover a pool, label it pooled recall rather than complete-catalogue recall. Reciprocal rank focuses on the first relevant result. nDCG measures graded, position-sensitive ranking quality relative to an ideal ordering; it is not the percentage of products judged relevant. Metric documentation also makes unlabeled-result handling explicit.

Show both weighted and per-query views, with severe failures and constraint violations alongside averages. Retain returned IDs and judgments so another reviewer can inspect a score.

Confirm the commercial decision live

Predefine the assignment unit, primary outcome, guardrails, sample plan and stopping rule. Check event integrity and allocation before interpreting results. Report effects and uncertainty rather than a single winning percentage.

PA’s Shopify bikes case reports AOV of A$241.82 versus A$223.89 in a 25-day split test covering 590,878 visitors. That is an approximately 8.0% relative difference in one retailer’s test; the published AOV win chance was 92.8%. It does not establish the result a different retailer will achieve.

Use the offline findings to explain failures and the live findings to decide whether the tested experience earns a wider rollout.

Illustrative example

{"label":"Synthetic relevance-metric fixture","grades":[4,2,0,0,0,0,0,0,0,0],"relevant_threshold":2,"judged_relevant_universe":2,"precision_at_10":0.2,"pooled_recall_at_10":1,"mrr":1,"ndcg_at_10":1,"gain":"2^grade - 1","discount":"log2(rank + 1)","no_relevant_items_policy":"nDCG not applicable"}

Evidence

  • PA reports A$241.82 AOV versus A$223.89 for the incumbent, approximately 8.0% higher, in one 25-day Shopify bikes test. (Vendor-published single-retailer test; AOV win chance 92.8%; no universal guarantee or same-confidence claim for all KPIs.) Source

Sources

How Particular Audience does it

Frequently asked questions

How many queries are enough?

Choose coverage based on the decision, traffic mix and important failure classes. A convenient total is not a validity guarantee. Record sampling and weighting, then inspect whether material query groups have enough evidence.

Can an LLM provide objective relevance grades automatically?

It can produce useful judgments, but needs an explicit rubric, evidence, calibration and repeatability checks. Human reviewers also need calibration; unresolved disagreement should remain visible rather than being forced into a confident score.

Does better nDCG prove higher revenue?

No. nDCG describes ordering against relevance judgments. Revenue also depends on availability, price, presentation and shopper behaviour. Estimate commercial effects with a separately designed live comparison.

Related guides

How-to guides · All guides