A visual-search pilot that shows lift on the image-upload surface has told you one thing: some shoppers had an intent that words handled badly. It has not told you that image queries are better, and it certainly has not told you to put image matching in front of the catalogue and demote keyword and attribute search behind it. That substitution is the failure this article is about, and it fails quietly — which is what makes it expensive.
Visual search is a query mode, not a discovery surface. Text search and structured facets are other query modes. A retail discovery surface that keeps all of them instrumented can measure which intents each mode serves and route accordingly. A surface that replaces one with another loses the queries the retired mode was carrying, and the loss shows up in aggregate as “search seems to be converting worse lately” — a reading that sends the team off to tune the wrong thing.
What does “visual search does not replace text search” mean in practice?
It means the image is a poor container for most of the constraints a shopper actually holds. An uploaded photo carries shape, colour, pattern, silhouette, finish, and visual similarity. It does not carry size, price band, compatibility, stock status, material, or the shopper’s own modification of what they are looking at — “same shirt but in linen”, “this lamp but with an E27 fitting”, “these trainers in a 44”. Those intents live in words and in structured attributes, and no embedding of a photograph recovers them, because the information was never in the pixels.
So the practical meaning is narrow and testable: for any query mode you deploy, you should be able to name the intents it can represent and the intents it cannot. Then you route.
| Intent the shopper holds | Expressible in an image query | Needs text / structured attributes |
|---|---|---|
| “This pattern, something like it” | Yes | No |
| “This silhouette in a different colour” | Partially — colour is a filter, not the image | Yes, for the colour constraint |
| “Size 44, in stock” | No | Yes |
| “Under £60” | No | Yes |
| “Compatible with model X” | No | Yes |
| “Same item, different material” | No — the delta is verbal | Yes |
| “I cannot name this thing at all” | Yes | No |
| “SKU 4417-B” | No | Yes |
The row that matters most is the second-to-last one. “Same but different” queries look like image queries to a product manager and behave like text queries to a retrieval system, because the useful part of the query is the modification, not the picture. Substitution decisions strand exactly this class of shopper.
What breaks first when image search becomes the default entry point
The first breakage is not accuracy. It is coverage. Attribute-driven queries that previously resolved through the facet stack now arrive at an image-first surface with nothing to upload, or upload a photo whose match confidence is fine but whose result set ignores the constraint that made the query worth issuing. The shopper sees plausible products that are the wrong size or the wrong price, and leaves. Zero-result rate barely moves — the index returned something — so the dashboards look healthy while the abandoned-session volume climbs.
The second breakage is that the honesty floor of visual search now applies to the whole surface instead of one lane of it. Catalogue churn and uneven product imagery degrade visual matching before they degrade anything else: a re-shot hero image, a newly added SKU that has not been embedded yet, a supplier photo taken on a different background. Text search keeps discovery working while the index catches up. Remove text search from the primary path and every index-freshness gap becomes a discovery outage rather than a quality dip in one mode. We see this pattern regularly in retail deployments where the catalogue turns over faster than the re-embedding cadence — the text path is doing more load-bearing work than anyone had credited it with.
The third breakage is measurement, and it is the one that hides the first two. Lift on the image surface and cannibalised text-search conversion are the same number if you only track site-wide search conversion.
Routing and fallback: the engineering shape
The correct architecture is a router in front of several retrieval paths, not a hierarchy with one mode on top. In practice that means:
- Classify the query, not the input surface. An image plus the text “in linen” is a hybrid query. Route the image to the visual index for candidate generation and the text fragment to attribute filtering — do not let the upload determine the whole retrieval path.
- Set an explicit match-confidence threshold below which the surface stops presenting visual results as answers and offers the text/facet path instead. The threshold is a product decision informed by measured top-k behaviour on your own catalogue, not a vendor default.
- Make the fallback a visible, first-class path. “We could not confidently match this image — try describing it, or browse this category” converts better than a page of near-misses, because it hands the shopper back a mode that can carry their constraint.
- Check index freshness at query time. If the SKU cohort a query falls into has not been re-embedded since the last catalogue update, the visual path is answering from a stale index and should be weighted down.
The thresholds and embedding-similarity limits that decide when fallback fires are general computer-vision engineering, not retail logic — they come from the same practice as any other production computer vision retrieval system, and they need to be measured against your catalogue’s near-duplicate density before they mean anything. Index performance and match-confidence measurement are what make the routing rule enforceable rather than aspirational; a GPU performance audit of the image index is usually where those numbers first become trustworthy.
Instrumenting both modes so lift is not an illusion
Discovery coverage, not a single-surface win, is the measurable outcome. That requires per-mode metrics reported side by side:
- Image-search-to-cart and image-search-to-purchase rate — the surface-level win, tracked on its own.
- Text-search conversion, tracked over the same window. If it drops while image conversion rises, you have moved traffic, not created value.
- Fallback rate — share of image queries that end up issuing a text or facet query. A healthy number is not zero; zero usually means the fallback path is invisible.
- Zero-result rate per query mode, never blended.
- Abandonment after a served result set — the metric that catches the “plausible but wrong constraint” failure that zero-result rate misses.
A correctly scoped split shows visual search absorbing visually-expressible intent without depressing the text path. Once a deployment is stable, image queries typically settle as a minority share of discovery traffic concentrated in visually-driven categories — an observed pattern across the retail engagements we have worked on rather than a benchmarked industry rate, and one that varies enormously with catalogue type. Treat any projection that has image search carrying the majority of discovery as a claim requiring evidence from your own traffic.
Our broader framing of what a visual-search programme should own — and what it should refuse — sits in AI visual search for retail product discovery, and the per-category question of where image input genuinely wins is worked through separately in the visual-vs-text category rubric. Both assume the routing discipline described here rather than replacing it. For the commercial framing of discovery work across the wider retail practice, the same rule holds: coverage first, then surface-level lift.
The uncertainty worth naming is the hybrid query. Multimodal retrieval is closing the gap between “this picture” and “this picture, but in linen”, and as it does, the boundary between the two modes moves. What does not move is the requirement that a shopper holding a constraint an image cannot express still has a path that carries it.
Frequently Asked Questions
What does “visual search does not replace text search” mean in practice for a retail discovery surface?
Visual search augments rather than supplants traditional text-based retrieval systems. It means visual search is one query mode inside the surface, scoped to intents images actually carry — shape, colour, pattern, visual similarity — with text and structured attributes retained as peer paths rather than legacy fallbacks. In practice, the discovery surface routes a query to the mode that can represent it, and every mode stays instrumented so you can see which intents each one is serving.
Which shopper intents can an image query actually express, and which ones only text or structured attributes can carry?
Images express visual properties: silhouette, pattern, finish, colourway, and general similarity — including the case where the shopper cannot name the item at all. Size, price band, compatibility, stock status, material, SKU identifiers, and any “same but different” modification are verbal or structured constraints that were never encoded in the pixels, so no embedding recovers them.
How should a discovery surface route between image, text and facet queries, and what fallback path fires when match confidence is low?
Classify the query rather than the input surface, so an image plus a text fragment is routed as a hybrid — image to candidate generation, text to attribute filtering. Below an explicit match-confidence threshold measured against your own catalogue, the surface should stop presenting visual results as answers and hand the shopper a visible text/facet path instead of a page of near-misses.
What breaks first when visual search is made the default entry point instead of a complementary mode?
Coverage breaks before accuracy does: attribute-driven queries arrive at a surface that cannot represent their constraint, get served plausible-but-wrong products, and abandon — without moving the zero-result rate. Because the index returned something, dashboards look healthy while abandoned-session volume rises, which is why the failure is usually diagnosed late and misread as generic search decay.
How do catalogue churn and uneven product-image quality change the balance between image matching and text search?
Churn and inconsistent imagery degrade visual matching first — unembedded new SKUs, re-shot hero images, supplier photos on different backgrounds — while text and attribute search keep working from catalogue data. That makes the text path the thing holding discovery together during index-freshness gaps, so the higher your churn rate, the more load-bearing text search is and the worse a substitution decision performs.
How do we instrument both query modes so lift on the image surface is not confused with cannibalised text-search conversion?
Report per-mode metrics side by side rather than blended: image-search-to-cart and image-search-to-purchase alongside text-search conversion over the same window, plus fallback rate from image to text, zero-result rate per mode, and abandonment after a served result set. If image conversion rises while text conversion falls, traffic moved rather than value being created.
Complementary retrieval: when images beat keywords and vice versa
Users type queries when they know the category; they upload photos when language fails—support both or lose half your intent signal. If Visual Search Replace Text is on your roadmap, the next step is to map it onto your own constraints rather than copy a reference architecture.
