Vector Databases for LLMs: How Retrieval Latency Becomes a GPU Bottleneck

A retrieval-augmented generation deployment misses its p95 latency target. The team opens nvidia-smi, sees the GPU pinned above 90 percent during token generation, and concludes the model is the bottleneck. So they add another GPU. Latency barely moves. Cost goes up.

This is one of the most common misreads we encounter in RAG systems, and it comes from a reasonable-looking instinct: the GPU is where the expensive math happens, so the GPU must be where the time goes. It isn’t. In a RAG request, a large fraction of wall-clock time is frequently spent before the LLM kernel ever runs — in the vector-database query, the embedding step, and the host-side plumbing that moves candidates back to the serving process. High GPU utilisation during generation tells you nothing about the retrieval stage that precedes it.

What does a vector database for an LLM actually do?

Strip away the branding and a vector database is an index over high-dimensional embeddings plus an approximate-nearest-neighbour (ANN) search on top. When a query arrives, the system embeds it into the same vector space as the stored documents, then asks the index for the k closest vectors. Those become the retrieved context that gets stitched into the prompt the LLM sees.

The important part for performance reasoning is that this whole path runs largely off the GPU on most stacks. The ANN search in engines like FAISS, Milvus, Qdrant, or pgvector is typically a CPU-bound graph or quantisation traversal. Query embedding may run on the GPU, but it is a small, latency-sensitive forward pass, not a sustained throughput workload. And the results have to travel back across process and memory boundaries before generation can start. Each of those hops is wall-clock time the GPU spends idle.

So the mental model that matters is not “the LLM is slow.” It is that a RAG request is a pipeline of heterogeneous stages, and the constraint can live in any of them. Naming the stage where time actually accumulates is the first engineering move, not the last.

Where does the bottleneck really sit in a RAG pipeline?

The honest answer is: measure it, because it varies with corpus size, index type, and how well-tuned each stage is. But there are recurring patterns worth stating plainly.

When the vector store is large and the index is unbuilt, misconfigured, or forcing an exact brute-force scan, retrieval can dominate the request. We have seen request paths where the ANN search alone consumes more wall-clock time than the entire generation step, while the GPU sits at single-digit utilisation until the context is assembled (observed across engagements; not a published benchmark). In that regime, buying GPU capacity is engineering by superstition — the added silicon has nothing to do because the request is stalled upstream.

The inverse also happens. With long output sequences and a small, well-indexed corpus, generation genuinely dominates, and retrieval is a rounding error. This is the case people implicitly assume, and it does exist. The failure is not assuming it — it is never checking whether you are in it.

The reason the misread persists is a measurement artefact: the GPU utilisation number is real, but it describes only the window during which the kernel runs. If generation takes 200 ms at 90 percent utilisation and retrieval took 600 ms before it, the utilisation graph shows a busy GPU and hides the fact that three-quarters of the request happened somewhere else. This is the same category of error we describe in ML model metrics that explain GPU bottlenecks versus metrics that mislead: a locally-true number that answers a different question than the one being asked.

How do you profile the full request path?

You separate the stages and time each one, end to end, under realistic load. The goal is a breakdown of where a request’s milliseconds go — not a single aggregate latency, and not GPU utilisation.

A workable decomposition for most RAG serving paths:

Stage	What it covers	Typical device	What makes it slow
Query embedding	Encode the query into a vector	GPU or CPU	Cold model load, tiny-batch inefficiency
Vector search (ANN)	Find top-k candidates	Usually CPU	Unbuilt index, high dimensionality, exact scan
Metadata filtering	Post-filter by attributes	CPU / DB	Filter applied after search, not fused
Result transfer	Move candidates to serving process	Host / network	Serialisation, network hop, oversized payloads
Prompt assembly	Build the context window	CPU	Large k, token-count blowup
LLM generation	Autoregressive decode	GPU	Sequence length, batch, model size

To get these numbers, wrap each stage in explicit timing spans and export them as traces. Distributed-tracing tooling — OpenTelemetry spans surfaced in a system you can query — makes the breakdown visible per request rather than averaged into oblivion. For the GPU stages specifically, layer in the kernel-level view from PyTorch’s profiler or Nsight Systems so you can confirm generation time is what you think it is. The point is a percentage split: what fraction of p95 latency is retrieval versus generation? Everything downstream follows from that one measurement. The broader discipline of instrumenting a serving path this way is what we mean by applying the three pillars of observability to GPU utilisation.

Which vector-database factors move end-to-end latency most?

Once you know retrieval is the constraint, a small set of factors dominate what you can do about it.

Index type. An HNSW or IVF-PQ index answers approximate queries in sub-linear time; a flat exact index scans everything. On a corpus of any real size, the difference between an ANN index and a brute-force scan is the difference between milliseconds and seconds. An unbuilt or default index is the single most common cause of retrieval-dominated latency we see.
Dimensionality. Higher embedding dimensions raise both search cost and transfer size. The choice of embedding model quietly sets a floor on retrieval latency before any tuning.
Filtering strategy. Metadata filters applied after an unrestricted search waste the search; pre-filtering or filtered-ANN keeps the candidate set small. This is a design decision, not a knob.
Transfer and payload. Returning full documents when you need IDs, serialising across a network hop, or over-large k values inflate the host-side portion of the request. Data movement is the stage teams profile last and pay for first.

The recurring lesson across these is that the vector store is a data-feed problem, and data-feed problems have the same shape whether the consumer is an LLM or a recommender. The same reasoning shows up in where DLRM inference latency actually lives and in Cassandra database performance for GPU-fed AI pipelines — the accelerator is only as fast as the thing feeding it. If you are weighing a specific engine, the licensing-and-footprint trade-offs we cover in is Milvus open-source are worth reading alongside the latency picture, because the cheapest-to-run index and the fastest index are not always the same choice.

Why does adding GPUs sometimes do nothing?

Because a GPU can only accelerate the stage that runs on it. If retrieval owns 70 percent of your p95 latency and lives on the CPU-bound ANN search, then doubling GPU capacity affects, at best, the 30 percent that generation owns — and only if generation was throughput-limited in the first place. Amdahl’s law is doing the talking here: the speedup ceiling of any optimisation is bounded by the fraction of time it touches.

The ROI framing follows directly. The right anchor for a RAG system is end-to-end request latency and cost-per-query, not raw kernel throughput. Fixing an unbuilt index or trimming an oversized transfer payload frequently delivers a larger end-to-end speedup than any kernel tuning, at a fraction of the cost — no new hardware, one configuration change. Scaling GPUs when the constraint is the vector store spends money to leave the bottleneck exactly where it was.

This is not an argument against GPU capacity. It is an argument for measuring first. When generation genuinely dominates — long outputs, large model, tight index — more or faster GPUs is the correct move, and the tuning work on the serving side is real. The failure is deciding without the breakdown. Our broader treatment of when accelerator capacity helps and when it doesn’t lives on the GPU engineering practice page.

FAQ

What does working with a vector database for an LLM involve in practice?

A vector database stores document embeddings and runs approximate-nearest-neighbour search to find the k closest vectors to a query embedding. In practice it is a data-feed stage that runs largely off the GPU — the ANN search is typically CPU-bound and its results have to travel back to the serving process before generation begins.

In a RAG pipeline, when is the bottleneck the vector database retrieval rather than the LLM inference kernel?

Retrieval tends to dominate when the corpus is large and the index is unbuilt, misconfigured, or forcing an exact scan, or when transfer payloads are oversized. Generation dominates when outputs are long and the corpus is small and well-indexed. Which regime you are in is an empirical question you resolve by timing each stage, not by watching GPU utilisation.

How do I profile the full request path to separate vector search time from GPU generation time?

Wrap each stage — query embedding, ANN search, filtering, result transfer, prompt assembly, generation — in explicit timing spans and export them as traces with tooling like OpenTelemetry, then confirm the GPU stages with the PyTorch profiler or Nsight Systems. The deliverable is a percentage split of p95 latency across stages, not a single aggregate number.

Why can GPU utilisation look high during generation while end-to-end query latency is dominated by retrieval?

GPU utilisation only describes the window during which the kernel runs. If generation takes 200 ms at 90 percent utilisation but retrieval took 600 ms before it, the utilisation graph shows a busy GPU while hiding that most of the request happened upstream on the CPU or the network.

What vector-database factors most affect end-to-end latency?

Index type has the largest effect — an ANN index like HNSW or IVF-PQ versus a brute-force scan is the difference between milliseconds and seconds. Dimensionality, filtering strategy (pre- versus post-search), and transfer payload size are the other main levers, all of which are design decisions rather than runtime knobs.

When does adding GPU capacity fail to fix RAG latency?

Whenever the constraint lives outside the GPU. By Amdahl’s law, adding GPU capacity can only speed up the fraction of the request that runs on the GPU; if retrieval owns most of the latency on a CPU-bound index, more silicon does nothing but raise cost.

How do embedding generation and host-side retrieval interact with the GPU workload in a serving path?

Query embedding may run on the GPU as a small latency-sensitive forward pass, but the ANN search and result transfer are host-side and serialise ahead of generation. Because they gate when the generation kernel can start, their latency adds directly to the request even though the GPU is idle during it.

The cleanest way to settle the argument is to stop debating the utilisation graph and produce the stage breakdown — the single number for what fraction of p95 latency is retrieval versus generation. A GPU performance audit that profiles the full RAG request path is built to surface exactly that split, so the next capacity or index decision is made against the constraint that actually exists rather than the one the GPU meter implies.