Your production LLM endpoint prepends the same 800-token system prompt and five few-shot examples to every request. On a cold pass, the model recomputes the attention state for all of those tokens each time a user hits the endpoint. If the shared prefix is 90% of the input and only the user’s final question changes, you are paying — in GPU-seconds and in latency — to compute the same thing thousands of times a day. Radix cache is the serving-layer answer to that waste: it organises the KV cache as a radix tree so overlapping prompt prefixes are matched once and reused across every request that shares them. That single structural decision is what turns a prefill-bound workload into a decode-bound one. It is also the reason prefix caching sits on the boundary between a pure performance optimisation and a data-handling control — because the thing you are now retaining and sharing across requests is prompt content, and prompt content carries PII and system-prompt secrets. How does radix cache actually work? Start with what an LLM does on every request. A transformer processes the input tokens (the prefill stage), building a key-value cache — the attention state for each token at each layer. Then it generates output tokens one at a time (the decode stage), each new token attending back over the accumulated KV cache. Prefill is compute-heavy and parallel; decode is memory-bandwidth-bound and sequential. The naive serving loop treats every request as cold. It runs the full prefill over the entire input — system prompt, few-shot examples, retrieved context, and the user’s actual query — then decodes. Two requests that share the first 800 tokens still each pay for 800 tokens of prefill. Multiply that by concurrent traffic and the prefill stage dominates your GPU budget while doing redundant work. Radix cache breaks that assumption. Instead of discarding the KV cache after a request finishes, the server keeps it and indexes it by the token sequence that produced it. When a new request arrives, the server walks its token prefix against a tree of already-computed prefixes. Any matched prefix is reused directly — its KV state is already resident — and only the divergent tail (typically the user’s unique query and the generated output) needs fresh prefill. In practice this means the shared system prompt gets computed once and served from cache to every subsequent request that begins with it. The mechanism is the same idea that underpins SGLang’s RadixAttention. If you want the attention-kernel-level view of how the reuse is wired into the forward pass, our companion explainer on KV-cache reuse in RadixAttention covers the runtime side; this article stays at the serving-architecture level. What is a radix tree, and why is it the right structure for this? A radix tree (a compressed prefix trie) stores strings by their shared prefixes, collapsing any chain of single-child nodes into one edge labelled with the whole shared substring. For KV-cache reuse the “strings” are token sequences, and each edge carries the KV state for its span of tokens. Why this structure and not a flat hash of full prompts? Because prompt sharing is rarely all-or-nothing. Two requests might share a 600-token system prompt but differ in their few-shot examples; three requests might share the system prompt and the examples but differ only in the final question. A flat cache keyed on the whole prompt would miss every partial overlap — a one-token difference at the end invalidates the entire entry. A radix tree naturally captures nested sharing: the common system prompt is one path near the root, the branches diverge where the prompts diverge, and every request reuses the longest matching prefix available. This is a structural property, not a tuning trick. The tree matches the actual shape of production prompt traffic, where a small number of templates fan out into many concrete requests. That fan-out is exactly the pattern a radix tree indexes efficiently, which is why serving frameworks converged on it rather than on simpler caches. Which workloads benefit most from prefix caching? Prefix reuse pays off in proportion to how much of the token budget is shared and stable. The workloads where it helps most are the ones where a large, fixed context precedes a small, variable query. Workload Prefix sharing Typical benefit Why Shared system prompt (agents, RAG, tool use) High — same instructions on every call Large The system prompt is identical across all traffic; it is computed once Few-shot prompting High — same exemplars reused Large Few-shot examples are a fixed block that dominates input length Multi-turn chat Growing per session Moderate to large Each turn extends the previous turn’s prefix; the conversation history is reused turn over turn Long shared document context High if the document is reused Large A long retrieved or pasted document shared across follow-up questions Fully unique prompts (no shared structure) None Negligible Nothing to match; every request is genuinely cold The decision rule is simple: estimate the ratio of shared-and-stable tokens to unique tokens across your real traffic. If shared tokens dominate, prefix caching is one of the highest-leverage serving optimisations available. If every request is genuinely unique, the tree overhead buys you nothing and you should invest elsewhere — for example in decode-stage techniques like the ones we cover in hierarchical KV-cache and tiered reuse. To size the shared-vs-unique split before you commit, our token estimation guide is a useful starting point. How much throughput and latency improvement is realistic — and what limits it? Radix caching commonly removes the redundant prefill on shared prefixes, which can raise serving throughput several-fold and cut time-to-first-token substantially on prompt-heavy workloads (observed pattern across prompt-sharing deployments; the exact multiple depends entirely on your sharing ratio, not a benchmarked constant). The intuition is direct: if 90% of your input tokens are shared and cached, you eliminate roughly 90% of the prefill compute, and prefill is where a prompt-heavy endpoint spends most of its time. The gain is real but it is bounded by four things, and none of them are optional to understand before you rely on the speedup: Sharing ratio. No sharing, no benefit. The improvement scales with how much of the token budget is actually reused. A workload with fully unique prompts sees essentially nothing. Cache capacity and eviction. The KV cache lives in GPU HBM, which is finite. When it fills, the server evicts entries — usually least-recently-used. A prefix that gets evicted before the next matching request arrives has to be recomputed. Under high prefix diversity, your effective hit rate is set by eviction policy, not by the tree. Decode-bound shift. Once prefill stops being the bottleneck, the workload becomes decode-bound. Your ceiling moves to memory-bandwidth and decode throughput, so further gains come from batching and decode-stage optimisation, not from more prefix caching. Match granularity. Reuse only kicks in on exact token-prefix matches. A change near the start of the prompt — even a timestamp or a per-user ID injected before the shared instructions — breaks the match for everything after it. Prompt construction order matters: put the stable, shared content first and the volatile content last. That last point is the one teams miss most often. Prefix caching rewards prompt templates that are deliberately structured for reuse, and it silently earns you nothing if a per-request token sneaks in ahead of the shared block. What data-protection risk does caching prompt prefixes introduce? Here is where the performance story becomes a governance story. A cold serving loop holds prompt content only for the life of one request. A radix cache, by design, retains prompt content and shares it across requests to make reuse possible. If a system prompt contains credentials, or if user-supplied context in a shared prefix contains PII, that content now persists in GPU memory and is reachable by the serving path across request boundaries. This is not a reason to avoid prefix caching — it is a reason to treat it as a control that must stay inside the residual risk leadership has already accepted. In our experience reviewing production GenAI serving stacks, the recurring failure is that prefix caching is enabled as a pure latency win without anyone updating the data-handling model to reflect that prompt content now has a lifetime longer than a single request. Concretely, the questions that decide whether a cached prefix is safe are: does the shared prefix ever contain PII, and if so is that within the boundary the risk owner signed off on; is the cache scoped so that one tenant’s cached prefix can never be matched and served into another tenant’s request; and how long does cached content persist before eviction, and is that lifetime documented. This is exactly the kind of serving-layer control our GenAI feasibility audit checks — that cached prompt and PII content stays inside the accepted residual-risk boundary rather than quietly expanding it. Prefix caching is a prototype-to-production serving optimisation, which means it has to pass the same operational governance gate as any other production change. It is not a config flag you flip in isolation. How does radix caching relate to PagedAttention and commercial prompt caching? These are complementary, not competing, techniques, and conflating them causes real confusion in architecture reviews. PagedAttention solves memory management for the KV cache. It stores the cache in fixed-size, non-contiguous blocks — like virtual memory paging — so the server avoids fragmentation and can pack more concurrent sequences into HBM. It does not, by itself, decide which prefixes to reuse. Radix caching solves reuse: which prefixes match and can be served from existing state. A well-built serving stack uses both — paged block management underneath, radix indexing on top. Commercial prompt caching (the “cached tokens” line item on hosted LLM APIs) is the same underlying idea — prefix reuse — exposed as a pricing feature. The provider caches your repeated prompt prefixes and bills them at a lower rate. The mechanism is essentially radix-style prefix reuse operated on your behalf; the difference is that you control neither the eviction policy nor the data-handling boundary, which is precisely the trade-off a governance review has to weigh when choosing between a hosted API and self-hosted serving. The practical takeaway: radix caching is the policy layer for prefix reuse, PagedAttention is the memory layer beneath it, and commercial prompt caching is the same policy sold as a managed feature. FAQ How should you think about radix cache in practice? Radix cache keeps the KV-cache state of already-computed token prefixes instead of discarding it after each request. When a new request arrives, the server matches its token prefix against a tree of cached prefixes, reuses the longest match directly, and runs fresh prefill only on the divergent tail. In practice a shared system prompt is computed once and served from cache to every request that begins with it. What is a radix tree, and why is it the right structure for reusing KV-cache prefixes? A radix tree is a compressed prefix trie that stores sequences by their shared prefixes, collapsing single-child chains into one edge. It fits KV-cache reuse because prompt sharing is usually partial and nested — requests may share a system prompt but differ in few-shot examples or final query. The tree captures the longest matching prefix for each request, where a flat full-prompt cache would miss every partial overlap. Which workloads benefit most from prefix caching? Workloads where a large, stable context precedes a small variable query: shared system prompts in agents and RAG, few-shot prompting, multi-turn chat (each turn extends the prior prefix), and long shared document contexts. Fully unique prompts with no shared structure gain essentially nothing. The benefit scales with the ratio of shared-and-stable tokens to unique tokens in real traffic. How much throughput and latency improvement can radix caching realistically deliver, and what limits it? On prompt-heavy workloads with high prefix sharing it commonly raises throughput several-fold and cuts time-to-first-token, because it eliminates redundant prefill — the dominant cost on such endpoints. The gain is bounded by sharing ratio, cache capacity and eviction policy, the shift to a decode-bound ceiling, and exact-match granularity: a volatile token placed before the shared block breaks reuse for everything after it. What data-protection risk does caching prompt prefixes introduce when PII lands in shared context? Prefix caching retains and shares prompt content across requests, so PII or system-prompt secrets in a shared prefix now persist in GPU memory beyond a single request. The controls that keep it inside accepted residual risk are: confirming whether shared prefixes can contain PII and whether that was signed off, scoping the cache so one tenant’s prefix cannot be served into another’s request, and documenting cache lifetime and eviction. How does radix caching relate to other KV-cache techniques like PagedAttention and prompt caching in commercial APIs? They are complementary. PagedAttention manages KV-cache memory in fixed-size blocks to avoid fragmentation; radix caching decides which prefixes to reuse. A strong stack uses both. Commercial prompt caching is the same prefix-reuse idea sold as a managed pricing feature, with the trade-off that you control neither its eviction policy nor its data-handling boundary. What should a team check before enabling prefix caching in a production GenAI deployment? Estimate the shared-vs-unique token ratio to confirm the benefit is real; structure prompts so stable content comes first and volatile tokens last; size cache capacity and choose an eviction policy against expected prefix diversity; and treat it as a data-handling change — confirm PII in shared prefixes stays inside accepted residual risk, cache scoping isolates tenants, and content lifetime is documented and reviewed. Where the speedup stops being free The reuse is real, and on a prompt-heavy endpoint it is one of the largest serving wins available. But the moment a cached prefix can carry PII or a system-prompt secret across request boundaries, the optimisation has changed the data-handling model — and that is a decision a risk owner has to make, not a flag an engineer flips. The question worth carrying into your next serving review is not “how much faster does radix cache make us,” but “what content now has a lifetime longer than a single request, and is that lifetime inside the boundary we already accepted?” That is the same gate every prototype-to-production serving change has to clear, and it is exactly what a GenAI feasibility audit is for.
