LLM as a Judge for Ecommerce Search: A Practical Guide
Build a calibrated LLM judge for ecommerce search. Define intent facets, create reference labels, test agreement and preserve evidence before trusting scores.
How to use an LLM as a judge for ecommerce search
An LLM relevance judge is useful when its decisions can be checked against an explicit rubric and reliable product evidence. Start with adjudicated reference examples, test disagreement and repeatability, and version the model and prompt. Keep uncertain cases visible and validate business impact separately from automated relevance scores.
Define the judgment precisely
Give the judge a query, relevant shopping context and the evidence available for one product. Specify which requirements are mandatory and which are preferences. Ask it to identify supporting attributes and unresolved evidence before assigning a grade.
For a hypothetical query for a blue waterproof jacket, colour and waterproofing need separate checks. A product photo may support blue; it should not establish waterproof performance. An unknown required attribute needs an unknown-evidence outcome, not a confident guess.
Build a reference set with adjudication
Have trained reviewers label a varied subset independently, then resolve disagreements against the product data and rubric. Include straightforward matches, clear mismatches, partial matches, unfamiliar brands and missing attributes. Keep calibration examples separate from the held-out set used to assess the finished judge.
Record the reason for each reference decision. Human agreement is useful evidence, but agreement around an underspecified rule can still produce the wrong criterion. DoorDash’s primary engineering account explains how decomposing intent into explicit facets addressed ambiguous search judgments.
Test the judge before scaling it
Measure exact grade agreement, agreement on the relevant/irrelevant boundary and error rates on mandatory constraints. Review a confusion matrix by query family; an average can hide systematic failures on sizes, negation or brands.
Repeat a sample with the same configuration and compare results. If you use pairwise judgments, reverse candidate order. Hide engine names and commercial claims from the judge. Research on LLM judging has identified order, verbosity and self-preference effects in assistant evaluation; your product-judging process should test for analogous sensitivity rather than assume it away.
Preserve the evidence trail
Store the catalogue snapshot, product evidence, model version, prompt version, rubric version and raw structured response. Product descriptions are evidence to assess, not instructions for the judge to follow. Validate the response schema and separate failed requests from valid low scores.
Route unsupported or ambiguous cases to review. Recheck calibration when the model, prompt, catalogue domain or grading rules change. Do not quietly compare scores produced under materially different rubrics.
Use judgments in an evaluation, not as a sales forecast
Aggregate labels into clearly defined ranking metrics and inspect examples behind changes. A reliable judge does not by itself establish complete-catalogue recall, unbiased query sampling or increased revenue.
Use Particular Audience’s Search Evaluation as a starting point for a scoped comparison. Agree the calibration evidence and adjudication process as part of that scope, then use a live experiment for the commercial decision.
Illustrative example
No numerical illustration needed; qualitative examples in the body are hypothetical unless explicitly attributed.
Sources
- LLM-as-a-Judge: Evaluating natural language search, checked 2026-10-25
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, checked 2026-10-25
- Search Evaluation, checked 2026-10-25
How Particular Audience does it
Frequently asked questions
Which grading scale should we use?
Use a scale whose boundaries reviewers can apply to your products. A binary eligibility decision and a separate graded relevance score can be easier to diagnose than one opaque score. Document the chosen thresholds before comparing engines.
Is a second LLM enough to validate the first?
A second model can expose disagreements, but both may share errors or missing evidence. Compare against an adjudicated reference set and inspect material errors rather than treating model agreement as ground truth.
What should happen when product evidence is missing?
Return an explicit unknown or review-required state. Do not infer safety, compatibility or performance claims from branding, images or a plausible description. Define how unknowns affect each aggregate metric.
Related guides
- How to evaluate ecommerce site search: Evaluate ecommerce search with representative queries, calibrated relevance labels and clearly defined metrics. Validate commercial impact with a planned live test.
- How to enrich product attributes for ecommerce search: Improve product attributes for search with a controlled AI enrichment workflow. Define evidence, review uncertain values and map approved data into your catalogue.