Fixing grocery site search

An anonymized top-10 grocer improved catalog coverage by 65%, cut severe misses by 40% and recorded a 67% better dissatisfied-searcher rate in a live rollout.

• +65% More relevant catalog surfaced (Recall@120 0.485 → 0.800)
• −40% Fewer severe misses (2.1% → 1.3% of queries)
• 67% better Dissatisfied-searcher rate vs incumbent
• +19% Better result quality (nDCG@120 0.737 → 0.880)
• 9 in 10 Top results judged relevant (nDCG@10 0.922 vs 0.868)
• 705 Query panel, graded two pages deep
• 275 → 75 Merchandising rules after rationalization

The subject is a top-10 North American grocery retailer — roughly $13B in eCommerce revenue across 550+ stores — running a legacy keyword search engine that got shoppers half-way to the right answer. Head terms mostly worked. Anything with a modifier, a dietary constraint, a pack size, a typo or a natural-language phrasing quietly failed, and every failure trained shoppers to stop searching.

We evaluated the incumbent against Particular Audience's transformer (vector) search on a blinded 705-term panel spanning basic staples, brand-vs-generic, compound and multi-attribute queries, produce specificity, misspellings, natural language, disambiguation, dietary and lifestyle, pack size, zero-result edge cases, department-only and seasonal terms. Both arms ran the same filter to depth 120 — two full pages of results. An LLM judge graded every returned product 0–4 for shopper intent: roughly 169,000 product grades rolled up into 12 metrics per term, per arm.

The headline result: coverage — the share of genuinely relevant catalog the engine can even reach — rose 65% at depth 120 (Recall@120 0.485 → 0.800). Overall rank-aware quality improved 19% (nDCG@120 0.737 → 0.880), and severe failures (queries scoring below 0.25 at nDCG@10) fell 40%, from 2.1% to 1.3% of the panel. At the top of the page, better than nine in ten results were judged relevant.

A subsequent one-week live rollout across web and app added a direct customer signal. After normalizing for the roughly three-quarter PA and one-quarter incumbent traffic split, PA recorded a 67% better dissatisfied-searcher rate, measured as negative survey responses per session. The calculation assumes comparable survey exposure and response behaviour across both arms.

Getting there meant fixing four failure modes, not just swapping engines. Semantic collisions: vector search reads descriptive meaning, so 'craft mayonnaise' drifted toward Kraft and 'kettle chips' straddled a cooking style and a brand — brand and category signals had to stay strong. Popularity distortion: popularity measured in raw units sold let staples like bananas crowd out precise matches, so it was normalized within category demand. Personalization conflict: predictive personalization lifted general relevance but suppressed the repeat-purchase items shoppers expected, so repeat-purchase ranking was rebuilt as five evidence tiers, from personal exact match down to pure content relevance. And over-rule-ing: roughly 275 live merchandising rules were fighting each other.

That last one was the quiet story. At depth 60 the raw model beat the incumbent (nDCG@60 0.854 vs 0.790) — but with the legacy rule layer applied on top, the same model scored 0.769, below the keyword engine it was meant to replace. The model was never the blocker; the rule layer was. Rationalizing searchandising from 275 rules to 75 was the prerequisite to final state, and it is why the production configuration now matches the raw model's quality while keeping the merchandising control the category teams need.

What a relevance gain is actually worth: more of the catalog becomes buyable, because products previously invisible to a query become reachable inventory — the same catalog, monetized harder. Fewer sessions end in distrust, because every severe miss teaches a shopper that search does not work and pushes them to browse or to a competitor. And every ad slot inherits the lift: sponsored placements sit inside these result pages, so result-page quality is the multiplier on CTR and ROAS across the entire retail media network, priced per impression on inventory the retailer already owns.