Type “waterproof running jacket” into a modern search box and a keyword index is no longer what decides the ranking. A learned relevance model is: the query becomes a vector, that vector is compared against a store of document vectors, and a re-ranking model often re-scores the top candidates before anything reaches the user. Machine learning for search replaces literal keyword matching with learned relevance — and the moment it does, the economics of the feature change in a way most teams notice only when the cloud bill arrives. The mistake we see most often is treating “the search model” as the whole system. It isn’t. The model is trained once. The feature runs on every keystroke, every query, every user, forever. That distinction is the entire story of what ML search costs to operate, and it is why relevance improvements have to be weighed against a recurring serving cost, not just an offline accuracy score. What does machine learning for search work like in practice? Classic search is lexical. An inverted index maps terms to documents; a query is a set of terms; the engine returns documents containing those terms, ranked by something like BM25. It is fast, cheap, and blind to meaning. Search for “car” and it will not surface a document that only says “automobile.” Machine learning for search works differently. Instead of matching strings, it matches meaning, represented numerically. Text — both the query and the documents — is passed through a model that produces an embedding: a dense vector, typically a few hundred to a couple thousand dimensions, where semantically similar text lands close together in vector space. Relevance becomes a geometry problem. The documents whose vectors sit nearest the query vector are the candidates, and a similarity metric (cosine or dot product) scores them. That is the mechanism people mean when they say “semantic search.” The corpus is embedded ahead of time and stored in a vector index. At query time, only the query is embedded live, then compared against the pre-computed store. Many production systems go one step further and add a re-ranker — a heavier model that takes the query and each top candidate together and produces a sharper relevance score, correcting the coarse geometry of the first pass. Keyword search versus semantic search: what actually differs The temptation is to frame this as “old versus new” and move on. That misses where the cost lives. Both approaches answer the same question — which documents are relevant? — but they spend compute in very different places. Dimension Keyword (lexical) search Machine-learning (semantic) search Relevance signal Term overlap, TF-IDF / BM25 Learned vector similarity, optional re-rank score Query-time compute Index lookup — cheap, CPU-bound Embedding pass + vector search + optional re-rank — model inference Handles synonyms / intent Poorly Well Dominant recurring cost Index storage, some CPU GPU/accelerator inference per query Cost scaling driver Corpus size Query volume The last row is the one that surprises people. A lexical index costs roughly what your corpus costs to store; query traffic barely moves the needle. Semantic search inverts that: the corpus is embedded once, but every incoming query pays for at least one embedding inference, one vector similarity pass, and — if you re-rank — another model call over the candidate set. Cost scales with how many people search, not with how much you have indexed. This is exactly the per-request lens that governs any production AI feature. If you have read our breakdown of spec-ing the compute behind a production AI feature and its cost-per-request reality, the pattern here is the same primitive applied to a search path: the unit of cost is the request, and the request fires an inference chain. What happens on each query: embed, retrieve, re-rank Walking a single query through a typical serving path makes the cost structure concrete. Assume a semantic search feature with a re-ranking stage — a common shape for e-commerce and support-content search. Embed the query. The raw query string is tokenised (WordPiece or a byte-pair variant, depending on the encoder) and passed through an embedding model — often a bi-encoder such as a sentence-transformer. This is a forward pass. On a modern GPU it is small but not free, and at high query rates it is a steady inference load. Our note on what a bi-encoder is and when to use it in retrieval explains why this architecture is chosen precisely so the corpus side can be pre-computed. Retrieve candidates. The query vector is searched against the pre-built vector index (FAISS, Milvus, or a managed equivalent) using approximate nearest-neighbour search. This returns the top-k candidates — say the nearest 100. ANN search is compute-bound but tunable; recall-versus-latency is a dial, not a fixed cost. Re-rank. The top candidates are passed, query-plus-document at a time, through a cross-encoder re-ranker. This is the expensive hop: a cross-encoder runs a full forward pass per candidate, so re-ranking 100 documents is roughly 100 model calls, not one. Most teams cap this at the top 20–50 to control the blast radius. Each of those three hops is an inference call. The retrieval-plus-generation cousin of this pipeline is covered in our RAG AI example showing how a retrieval-augmented pipeline works and gets optimized; ML search is the retrieval half of that pattern without the generation step, and it shares the same multi-hop cost profile. Why is every search query an inference call with a recurring cost? Because the model does not answer the question at training time — it answers it at query time. Training produces the weights. Serving spends compute to apply those weights to a specific query, and that spending recurs for the life of the feature. This is the divergence point between teams. One team ships a semantic search feature, watches offline relevance metrics climb, and treats the job as done. Another team ships the same feature but attaches a cost-per-query target to it from the start. When traffic triples — a good problem, more users searching more — the first team discovers that relevance has quietly become one of their largest recurring cloud line items, because they never modelled the per-query inference. The second team saw it coming, because they sized the serving path against projected query volume rather than against a one-time training run. We see this pattern regularly in cost reviews: the offline accuracy story is excellent, the online margin story was never written down. Framing search as per-query inference is what makes the second story visible before it becomes a surprise. How do you measure cost-per-query and search latency? Three metrics carry the feature, and they should be measured together because trading one off silently moves the others. Quick answer — the metrics that decide a search serving config Cost-per-query. Total GPU/accelerator-seconds (or vendor token charges) consumed by the embed → retrieve → re-rank chain for one query, divided by nothing — it is the per-query figure. Include every hop, not just the embedding. p95 search latency. The 95th-percentile end-to-end time from query received to results returned, across all three hops. Averages hide the re-rank tail; p95 exposes it. GPU-seconds (or tokens) per query. The resource unit that ties directly to the invoice. This is what you multiply by projected query volume to forecast spend. Measuring these per hop — not just for the feature as a whole — is where the actionable signal lives, and it is inherently a GPU-level exercise. Profiling the embed-retrieve-rerank path at the accelerator level is how per-query cost is actually attributed to each stage; that measurement discipline is the subject of our work on machine learning compilers and how they cut cost-per-request in production AI, and it is the same profiling posture a GPU-level cost audit brings to any serving path. Once you have per-hop numbers, the optimisation choices become obvious rather than guessed: shrink the re-rank candidate set, quantise the embedding model, batch queries, or move the re-ranker to a cheaper accelerator. Each of those is a lever on a metric you can now see. How does query volume drive inference cost as the feature scales? Linearly, in the naive case — and that is the trap. A semantic search feature at 10 queries per second and one at 1,000 queries per second run the same model, but the second one runs it 100 times more often. Cost tracks the query curve, not the model. Corpus growth barely matters to per-query cost (it is a one-time re-embed plus a slightly larger index); traffic growth matters enormously. This is why the cost-per-query baseline is worth establishing early. With it, a relevance improvement that adds a re-rank hop can be evaluated honestly: “+3% relevance, +40% cost-per-query at current traffic, projected to be $X/month at target volume.” Without it, that trade-off is invisible until an invoice makes it visible. The vector-store side of this — inspecting where per-query cost accumulates in the retrieval layer — is walked through in our look at Attu for Milvus and inspecting a vector store’s cost-per-query in practice. For teams building search and discovery features on cloud infrastructure, this per-query framing is the difference between a feature with a defensible gross margin and one that erodes it as it succeeds. Our AI infrastructure for SaaS work starts from exactly this lens: the recurring serving cost, measured per request, is the number that decides whether a feature is worth shipping at scale. FAQ What should you know about machine learning for search in practice? ML search replaces literal keyword matching with learned relevance. Text is converted into embeddings — dense vectors where semantically similar content lands close together — and relevance becomes a matter of vector similarity rather than term overlap. In practice, the corpus is embedded ahead of time and stored in a vector index, while each query is embedded live, compared against that store, and often re-ranked before results are returned. What is the difference between keyword search and machine-learning (semantic) search? Keyword search matches strings using an inverted index and a scoring function like BM25 — cheap, CPU-bound, and blind to meaning. Semantic search matches meaning via learned vector similarity and optional re-ranking, which handles synonyms and intent far better. The critical difference is cost structure: lexical cost scales with corpus size, while semantic cost scales with query volume because every query fires model inference. What happens on each query in an ML search path — embedding, retrieval, and re-ranking? Three hops run per query. The query is embedded into a vector by an encoder model; that vector is searched against a pre-built vector index to retrieve the top-k candidates via approximate nearest-neighbour search; and the top candidates are optionally passed through a cross-encoder re-ranker that scores query-plus-document pairs one at a time. The re-rank stage is usually the most expensive because it runs a forward pass per candidate. Why is every search query an inference call with a recurring cost? Because the model is applied at query time, not at training time. Training produces the weights once; serving spends compute to apply those weights to each specific query, and that spending recurs for the life of the feature. A search feature’s cost therefore scales with how often people search, not with the one-time cost of training the ranker. How do you measure cost-per-query and search latency for an ML-powered search feature? Track three metrics together: cost-per-query (total accelerator-seconds or token charges across every hop), p95 search latency (end-to-end 95th-percentile time, which exposes the re-rank tail averages hide), and GPU-seconds or tokens per query (the resource unit that ties to the invoice). Measuring these per hop — embedding, retrieval, re-rank — rather than only for the feature as a whole is where the actionable optimisation signal lives. How does search query volume drive inference cost as the feature scales? In the naive case, linearly with query volume — the same model simply runs more often. Corpus growth barely affects per-query cost (it is a one-time re-embed and a slightly larger index), but traffic growth drives spend directly. This is why a cost-per-query baseline set early lets teams weigh relevance improvements against their recurring serving cost rather than discovering the trade-off through a surprise invoice. Where do embedding and re-ranking models run in a deployed search serving path? The embedding model runs live on the query at request time, typically on a GPU or other accelerator, producing the query vector. Vector similarity search runs against the pre-computed index in a vector store such as FAISS or Milvus. The re-ranking cross-encoder, when present, runs on an accelerator over the retrieved candidate set — usually the top 20–50 — which is why it is the hop most teams cap to control per-query cost. If your search feature’s offline relevance story is strong but no one has written down its per-query serving cost, that gap is the failure class to close first: an embed → retrieve → re-rank path is a concrete multi-hop inference target, and profiling it per hop is what turns “a search model” back into a feature with a knowable gross margin.
