How to Evaluate Ecommerce Site Search Objectively | Particular Audience
Test site search relevance with a fixed query set, graded results and head-to-head comparisons, including LLM-as-a-judge grading.
How to evaluate your ecommerce site search
Evaluate ecommerce site search with a fixed set of real queries covering head, mid and long-tail searches, grade the top results for relevance against your own catalogue, and compare engines on the same queries. Large language models can grade results consistently at scale. Confirm the findings with an A/B test on live traffic before switching engines.
Build the query set
Sample real queries from your logs across short, descriptive and conversational searches.
Grade relevance
Score the top results per query. A consistent grader, human or model, matters more than the scale used.
Compare head to head
Run the same queries through each engine and compare scores by query type.
Confirm live
An A/B test on real traffic shows the commercial effect.
How Particular Audience does it
In production
- Shopify bikes retailer (head-to-head A/B): One Shopify bikes retailer, 590,878 visitors: PA won every KPI head-to-head (+8.0% Average order value (A$241.82 vs A$223.89))
Frequently asked questions
How many queries do we need?
Enough to cover each query type with several examples. A few hundred well-chosen queries is a practical start.
Related guides
- How to reduce zero-result searches on your ecommerce site: Five steps to find, fix and prevent zero-result and near-zero-result searches, from query logs to meaning-based search.
- Vector search vs keyword search: what's the difference for ecommerce?: Keyword search matches words; vector search matches meaning. Why that matters for long, descriptive and conversational shopper queries.