A visual-search pilot almost always passes acceptance. It is validated against a catalogue snapshot that was frozen the week the embeddings were built, and against that snapshot the top-k accuracy numbers look good. Three months later the same system is still returning confident matches — to products that are out of stock, discontinued, or re-shot with photography the index has never seen. Nothing alarms. The similarity scores are as high as they were on day one.
That is the failure mode worth naming: when catalogue churn is high, a snapshot-built image index decays silently, because the metric that guarded the pilot is insensitive to the thing that broke.
What “high catalogue churn” actually looks like
Churn is not one number. For product discovery it has three distinct components, and they degrade the index in different ways:
- Additions — new SKUs published to the catalogue. If they are not embedded, they are invisible to image search. The shopper photographs the new product and gets last season’s near-neighbour.
- Retirements — SKUs discontinued or out of stock. Their vectors stay in the index and keep winning matches. The match is correct and useless.
- Re-imaging — the same SKU re-shot with new lighting, new background, new crop, or a new model. The index still holds the old vector; a shopper photographing the product as it now appears is matching against a photograph that no longer exists.
Fashion, seasonal home goods, and grocery private-label ranges turn over fast enough that a snapshot taken at pilot time describes a materially different assortment a quarter later. Consumer electronics churn slower but re-image aggressively at every packaging revision. The point is that churn is a measurable property of the buyer’s own catalogue, and it should be measured before the pipeline is designed — not discovered afterwards.
Why do similarity scores stay high while match quality degrades?
Because cosine similarity in an embedding space answers a narrower question than the business one. It answers “which vector in this index is nearest to the query?” It does not answer “is that SKU still sellable, still photographed this way, still the best available answer?”
An embedding model — CLIP-style dual encoders and their successors are the common choice here — will always return a nearest neighbour, and it will always return one with a plausible score, because the index is dense enough that something looks close. Retire 15% of the catalogue and the query does not fail; it silently reroutes to the 85% that remains. Top-1 accuracy against the original test set is unchanged, because the original test set was drawn from the original snapshot. Conversion leaks instead, and conversion is noisy enough that a slow leak hides inside seasonality for a full quarter.
This is the structural reason the failure is silent: the pilot’s acceptance metric and the production failure live in different coordinate systems. Accuracy is measured against the index; the loss is realised against the live assortment.
Early warning signs
These are the signals we look for when a retail visual-search surface is described as “working but underperforming”:
| Signal | What it usually means |
|---|---|
| Image-search results include out-of-stock or delisted SKUs | Retired vectors are never evicted; no availability filter at retrieval time |
| Match rate is flat but image-search-to-cart is declining month over month | Silent staleness — the index answers, the answer is unbuyable |
| New-arrival categories underperform in image search but fine in text search | Re-embedding cadence is slower than SKU publication |
| Match quality varies sharply by catalogue cohort (launch-era vs recent) | Index built once; nobody instrumented per-cohort quality |
| Confidence distribution unchanged since launch | Score-based monitoring is blind to this failure; it needs availability-joined metrics |
| Re-shoot campaigns coincide with complaint spikes | Old vectors surviving a photography refresh |
The fifth row is the one that catches teams out. Monitoring dashboards built around confidence thresholds cannot detect this failure at all, by construction.
What to instrument instead
Four measurements replace the single global accuracy number, and each maps to a decision someone actually owns:
- Catalogue-freshness latency — hours between SKU publication (or re-shoot) and that SKU being queryable in the index. This is the pipeline’s real SLA. Pick a target, then cost it.
- Share of image-search results pointing to unavailable or retired SKUs — the direct measure of stale-vector leakage. It should trend to near-zero and stay there; if it drifts, eviction is broken.
- Fallback rate — how often confidence drops below threshold and the surface hands off to text, facets, or a “closest available” result set. A fallback path that never fires is not a safety net; it is an untested branch.
- Match quality per catalogue cohort — segment by publication generation, not globally. A pipeline that holds match rate on launch-era SKUs while collapsing on recent ones looks healthy in aggregate.
Add one cost measurement to make the trade-off legible: re-embedding and re-index cost per thousand SKUs at the buyer’s actual churn rate. That converts freshness from an engineering preference into a line item, which is where the conversation with a platform lead usually becomes productive. The economics of that recurring line are worked through separately in our breakdown of what it costs to keep a product-image index fresh; this article is about why the decay goes unnoticed in the first place.
Which parts are fixable in the index layer
Not all of this needs the pilot re-scoped. Splitting the two is the first useful triage:
Fixable in the index and retrieval layer:
- Retired-SKU leakage — join availability at retrieval time and evict on delist.
- Add latency — move from periodic full rebuilds to incremental upsert as SKUs publish.
- Re-shoot staleness — key vectors by image asset ID, not SKU ID, so a replaced asset invalidates its own vector rather than waiting for a rebuild.
- Fallback behaviour — an explicit low-confidence branch that returns available near-matches with honest framing instead of one confident wrong answer.
Requires re-scoping the pilot:
- Churn rate exceeding what the embedding and index budget can sustain. If the assortment turns over faster than the pipeline can re-embed at an acceptable cost, the honest answer is to narrow the categories in scope rather than accept permanent staleness.
- Acceptance criteria defined on a frozen snapshot. Those criteria have to be rewritten around freshness latency and cohort-segmented quality, or the same pilot will pass again and decay again.
- Categories where near-duplicate density means even a fresh index cannot separate variants. That is a different failure with different mitigations — we cover it in where CV product-matching fails in retail visual search.
We treat the re-embedding loop as ordinary computer-vision pipeline engineering rather than a retail special case: throughput sizing, batch scheduling, and GPU utilisation under a known arrival rate. The computer vision engineering practice is where that sizing work sits, and the surrounding product-discovery architecture — image to catalogue match to result, with no shopper profiling in the loop — is developed in our wider work on AI in retail and in the parent discussion of how AI visual search changes product discovery for retailers.
If you ignore the signs
The pilot does not break. It keeps its budget, keeps its dashboard, and keeps returning answers. What erodes is trust: merchandisers notice delisted products surfacing, product leads see image-search conversion drift below the text baseline, and the eventual conclusion is that “visual search didn’t work for our catalogue.” The model was rarely the problem. The refresh loop was never sized, and the metric chosen to protect the launch was structurally incapable of noticing.
The question we would put to any team scoping this next: what is your catalogue’s measured add, retire and re-shoot rate this quarter — and what freshness latency have you agreed to pay for?
Frequently Asked Questions
What does “visual search pilots fail when catalogue churn is high” mean in practice? Stripped down, Visual Search Pilots Fail is the following. It means the pilot passes acceptance against a frozen catalogue snapshot and then degrades as the live assortment diverges from the index. In practice shoppers get confident matches to retired, out-of-stock or re-photographed products, while accuracy dashboards stay flat.
How do we measure our actual catalogue churn rate before scoping a visual-search pilot? Pull SKU publication, delist and image-asset replacement events from the catalogue system over a representative period, and report them as three separate rates rather than one blended figure. Additions drive re-embedding volume, retirements drive eviction, and re-shoots drive invalidation — each sizes a different part of the loop.
Why do similarity scores stay high while match quality degrades? Because nearest-neighbour retrieval always returns a plausible neighbour from whatever is in the index; it has no notion of availability or of whether the stored photograph still represents the product. The acceptance metric is measured against the index, while the loss is realised against the live assortment.
What catalogue-freshness latency is acceptable, and what does hitting it cost? There is no universal figure — it depends on how fast the assortment turns over and how much of image-search traffic lands on new arrivals. Set the target explicitly, then cost re-embedding and re-index compute per thousand SKUs at your measured churn rate, so freshness becomes a priced line item instead of an assumption.
How should the fallback path behave when confidence drops or the matched SKU is unavailable? It should return available near-matches with honest framing, or hand off to text and facet search, rather than surfacing one confident wrong answer. Track the fallback rate: a branch that never fires has not been tested.
What changes when product photography is re-shot rather than when SKUs are added or retired? Re-shoots leave the SKU identifier intact, so SKU-keyed pipelines see no change and never invalidate the stale vector. Keying vectors by image asset ID instead makes a replaced asset invalidate its own embedding.
Which failure modes can be fixed in the index layer, and which require re-scoping the pilot? Retired-SKU leakage, add latency, re-shoot staleness and fallback behaviour are index and retrieval fixes. Churn that outruns the affordable re-embedding budget, snapshot-based acceptance criteria, and categories with near-duplicate density that no refresh can resolve all require re-scoping.
Catalogue churn thresholds that predict failure
When more than 15% of SKUs turn over per quarter, static embedding indexes drift fast enough that re-indexing overhead exceeds the labour cost of manual search—making the business case collapse. Everything else is detail.
