Ecommerce Search Evaluation: Compare Engines Fairly
Compare site-search engines on real queries and your catalogue. Review relevance grades, ranking metrics, failure examples and a clearly scoped evaluation offer.
An objective, catalogue-grounded evaluation of your site search: real shopper queries, every engine graded by an LLM judge on a published 0–4 rubric. USD $2,000 plus applicable taxes.
Site search quality is high-dimensional and often hidden: query types, modifiers, brand versus type, category mix, the store's own catalogue, page depth, merchandising rules and evolving intent all interact.
Method: Real shopper queries — Your top search terms, plus targeted modifier queries (dietary, size, format). Every engine's results — The full first page from each engine — including your incumbent — captured for every term. Catalogue answer pool — Every engine's results plus catalogue recall candidates, scoped to what the store stocks. The catalogue is the ground truth, not any engine. LLM judge grades 0–4 — Each product graded from the shopper's perspective against a published rubric. Qualified queries are graded on the modifier. Standard metrics — nDCG, precision, exact-match share, recall and MRR — at the top of the page (@10) and the full page (@60).
Rubric: 4 Exact match; 3 Right brand and type; 2 Right type, other brand; 1 Related; 0 Irrelevant. Relevant means grade 2 or higher.
Anonymised evaluation: A top-tier US grocery retailer, one store's live catalogue; 667 real search terms, 19,000+ products graded, 7 engines. +0.057 nDCG@10 vs incumbent (best first impression); +0.103 nDCG@60 vs incumbent (best full first page); ~59 / 60 products filled per page (vs ~44 for the incumbent); 9 severe misses in 667 (vs 43 for a global-merchandising setup).
Merchandising applied globally helped about 125 terms and hurt about 272; it should be applied per term.
Deliverables: Executive summary and a clear engine recommendation; Full leaderboard at @10 and @60; Per-term commentary: what happened, for every query; Every engine's full page, per term; Per-term merchandising guidance; Modifier-query deep dive; One-page summary and slide outline for your team. Price: USD $2,000 plus applicable taxes.
Adaptive Transformer Search · Universal Search · Book an evaluation
Evaluate ecommerce search on the queries your customers use
Compare your incumbent and candidate engines using the same shopping tasks, catalogue context and relevance criteria. A search evaluation should reveal where an engine succeeds, where it misses and which configuration deserves a live test. It should also make the evidence inspectable by your search and merchandising teams.
What the evaluation covers
PA's published offer uses shopper queries, a catalogue-grounded candidate pool and a relevance rubric scored from zero to four. It reports ranking and retrieval metrics at the top of the page and across a fuller result page, with per-query evidence and merchandising commentary. The published price is US$2,000 plus applicable taxes; confirm the scope when booking. Current offer.
Agree the comparison before grading it
Document which engines and configurations are being tested, the catalogue snapshot, store and availability filters, query sample and result depth. Keep the same conditions across candidates. Record rule layers separately so a strong retrieval system is not confused with a poor merchandising configuration.
Include frequent terms and specific shopping tasks: brands, sizes, pack formats, identifiers and other modifiers. Report how the sample was selected. An equally weighted query panel answers a different question from a traffic-weighted panel, and both can be useful when their purpose is clear.
Read each metric for what it measures
Precision describes the share of returned results meeting the chosen relevance threshold. Recall describes the share of relevant items retrieved from the defined reference set. When that reference is a pooled candidate set, say so; items never included in the pool remain a limitation.
nDCG measures the quality of ordering under a defined grading and discount rule, relative to an ideal ordering. It is not the percentage of products that are relevant. MRR focuses on the position of the first qualifying result. Read these measures together with the underlying products rather than choosing an engine from a single headline.
Inspect disagreements and severe misses
Use a human review sample to check whether the grading rubric matches the retailer's interpretation of the query. Decide how modifiers, exact brands, substitutes and genuinely unavailable products should be treated. Keep disputed judgements visible and record whether resolving them changes the conclusion. Agree this calibration process and responsibility as part of the evaluation scope.
The published example on this page covers 667 queries across seven engines for one store's catalogue. It is an example study, not a promise that every engagement has that sample or the same result. Example and deliverables.
Turn a leaderboard into a decision
Ask which configuration improves the queries that matter without creating unacceptable failures. Carry the selected configuration into an operational check for latency, data freshness and fallback behaviour. Use a controlled live test to estimate commercial impact. Offline relevance grades do not establish a conversion or revenue increase.
Bring a representative query export, catalogue context and the incumbent configuration to the scoping discussion. The useful outcome is a decision your team can explain and test, supported by examples it can inspect.
Frequently asked questions
Does nDCG tell us the percentage of relevant products?
No. nDCG is a normalised measure of ranked quality under a specified gain and position discount. Precision is the relevant share of the returned set under a defined threshold. A high nDCG score must not be restated as a percentage of results judged relevant.
Will the evaluation prove a revenue uplift?
No. It compares offline search quality under the study conditions. Use the findings to choose a configuration and investigate failures, then run an appropriate live test for commercial outcomes. Operational performance and data quality also need separate acceptance checks.
What does the published price cover?
The current page lists an executive recommendation, leaderboard, query-level commentary and result pages, merchandising guidance, a modifier review and summary material. It lists US$2,000 plus applicable taxes. Confirm the query sample, engines, data preparation and any additional work before commissioning.