“We turned on the radix cache and latency barely moved.” We hear a version of this often enough that it is worth naming the misconception directly: a radix cache is not a global speed dial you toggle for a flat win. It is a prefix tree over KV-cache state, and its entire value is a function of how much prefix your traffic actually shares. If your requests rarely overlap at the start, the honest measured gain is close to zero — and knowing that before you build around it is worth more than the optimism you started with. That is the core claim of this article. Everything below explains the mechanism well enough that you can predict, before enabling anything, whether a radix cache touches your bottleneck or leaves it exactly where it was. How does a radix cache work? Start with what actually happens when an LLM serves a request. The engine splits work into two phases: prefill, where it processes the entire input prompt in one pass and builds the key-value (KV) cache for every token, and decode, where it generates output tokens one at a time, reading from and appending to that KV cache. Prefill is compute-heavy and roughly proportional to prompt length. Decode is memory-bandwidth-bound and roughly proportional to output length. The insight a radix cache exploits is that prefill for a given token sequence produces exactly the same KV-cache state every time, given the same model weights. If two requests begin with the same 800-token system prompt, the engine is doing the same prefill arithmetic for those 800 tokens twice. That is pure redundancy. A radix cache stores computed KV-cache blocks in a radix tree (a compressed prefix tree) keyed by the token sequence that produced them. When a new request arrives, the engine walks the tree to find the longest already-computed prefix, reuses those KV blocks directly, and only runs prefill on the novel suffix. The redundant recomputation becomes a tree lookup. This is the mechanism popularised by RadixAttention in the SGLang runtime; we cover the attention-kernel side of it in RadixAttention and KV-cache reuse on GPU clusters. In practice, the meaning is narrow and specific: a radix cache reduces prefill compute and time-to-first-token (TTFT) for requests that share a prefix with something the cache has already seen. It does nothing for the decode phase, and nothing for prefixes it has never seen. That narrowness is the whole point. Where does a radix cache sit — the runtime layer, the model compute, or the host glue? This question matters more than it looks, because it determines what a radix cache can and cannot fix. A radix cache lives in the runtime layer — the serving engine that schedules requests, manages the KV-cache memory pool, and decides what to compute. It is not part of the model’s forward pass (the weights and kernels are untouched), and it is not host-side glue (tokenisation, request queuing, HTTP handling). It sits between them, deciding which forward-pass work is actually necessary. The distinction is the reason “just turn it on” fails as a strategy. If your latency is dominated by decode — long generations, short prompts — a runtime-layer prefix cache has almost nothing to grip. If your latency is dominated by host overhead — slow tokenisation, serialization, network round-trips — the radix cache is optimising a phase that was never your problem. It only bites when a meaningful share of your wall-clock latency lives in prefill for repeated prefixes. This is why we treat a radix cache as one attribution target among several rather than a default. Before enabling it, you need to know where the latency lives, which is the same discipline we describe in what “computationally expensive” really means in an inference path. If you have not located the bottleneck, you are guessing about whether prefix reuse is even relevant. What is the relationship between a radix cache and the KV cache it indexes? They are easy to conflate, and the conflation causes real confusion. The KV cache is the raw storage of attention keys and values for tokens already processed — every serving engine has one, because decode requires it. The radix cache is an index and reuse policy over KV-cache state across requests. Put differently: the KV cache answers “what are the keys and values for the tokens in this request so I can keep generating?” The radix cache answers “have I already computed the KV state for this token prefix in some request, and can I hand it to a new request instead of recomputing?” KV cache — per-sequence, mandatory for decode, lives for the duration of a request (or a session). Radix cache — cross-request, optional, keyed by token prefix, lives as long as the blocks stay resident in the memory pool. The radix cache does not replace the KV cache; it decides which KV-cache blocks can be shared and keeps them addressable by prefix. That also means it competes for the same GPU memory. A large radix cache holds more reusable prefixes but leaves less room for concurrent sequences’ KV state — a trade-off between hit rate and batch capacity that only shows up under real concurrency, not in a single-request test. Which workloads see a real latency gain — and which see almost none? This is the decision the whole article builds toward. Prefix reuse is workload physics, not a setting. The table below is the honest version. Workload pattern Prefix overlap Expected radix-cache effect Shared long system prompt across all requests (agents, assistants) High Large TTFT reduction; prefill largely eliminated on hits Few-shot prompting with a fixed preamble High Strong gain; the preamble is computed once Multi-turn chat, same conversation High (within session) Each turn reuses the growing shared prefix RAG with variable retrieved context prepended Low–moderate Gain only on the static instruction header, not the retrieved chunks Fully distinct user prompts, no shared preamble Near zero Negligible; measured gain close to zero Long generations, short unique prompts Near zero No effect — the bottleneck is decode, not prefill The pattern is consistent: the gain tracks the length of the shared prefix multiplied by the frequency it recurs. A 2,000-token system prompt reused across thousands of requests is close to the ideal case. A batch of unrelated one-off queries is close to the null case — and reporting that null result is a legitimate, useful outcome, because it tells the team to stop chasing this lever and look at decode, batching, or quantization instead. Speculative decoding, which we cover in SpecForge and speculative decoding for lower latency, attacks the decode phase that a radix cache cannot touch. How do you measure whether prefix reuse is available before enabling it? You do not need to enable the cache to estimate its ceiling. You need your traffic, a tokenizer, and a little arithmetic. A worked estimate (explicit assumptions). Suppose you sample 10,000 recent production prompts and tokenise them. You find that 70% of requests begin with an identical 1,500-token system prompt, and the average total prompt is 1,800 tokens. Shared prefix fraction of prefill work: on those 70% of requests, roughly 1,500 / 1,800 ≈ 83% of prefill tokens are reusable. Weighted across all traffic: 0.70 × 0.83 ≈ 58% of total prefill compute is potentially cacheable. If prefill accounts for, say, half of your end-to-end latency on those requests, the theoretical ceiling on latency reduction is roughly 0.5 × 0.58 ≈ 29% — before accounting for eviction, memory pressure, and cold-start misses, which pull the realised number down. This is an illustrative calculation to show the method, not a benchmarked result; your fractions come from your own traffic. The measurable signals once you do enable it are three, and they should be tracked as operational measurements from the deployed engine, not inferred: Prefix cache hit rate — the share of incoming prefill tokens served from the tree rather than recomputed. Share of prefill latency eliminated on hits — how much TTFT drops on cache-hit requests versus cold requests. Throughput under high-overlap load — requests/second at fixed latency, which reveals the memory-contention trade-off. If the hit rate is near zero after warm-up, the diagnosis is complete: your traffic does not share prefixes, and no tuning changes that. If it is high but TTFT barely improves, prefill was not your bottleneck. Either result is decision-grade. Why treating a radix cache as a flat on/off speedup misleads a cost decision The failure mode we see in inference-cost work is a team that quotes “radix caching gives an X% speedup” as if X were a property of the cache. X is a property of their traffic. Import the number from a benchmark run on agent workloads with 2,000-token shared system prompts, apply it to a RAG service with mostly-unique retrieved context, and the projected savings evaporate on contact with production. The correct framing is attribution first, lever second. Locate where latency lives — prefill, decode, or host — then ask which levers touch that phase. A radix cache is a prefill-reuse lever; it belongs in the conversation only after profiling shows prefill-for-repeated-prefixes is a real cost. This is the runtime-layer attribution step in our inference cost-cut sprint, where the profiling baseline measures whether prefix reuse is even available before anything is enabled. The broader engineering context lives on our GPU acceleration practice page. There is one more trap worth naming: a radix cache that improves an isolated benchmark can regress a production path under concurrency, because the memory it holds for reusable prefixes reduces the batch capacity available for live sequences. Whether the deployed path actually improves — not just the microbenchmark — is exactly the kind of behaviour release-readiness validation exists to confirm before it is claimed as a production win. FAQ What matters most about radix cache in practice? A radix cache stores computed KV-cache blocks in a compressed prefix tree keyed by token sequence. When a new request shares a prefix with something already computed, the engine reuses those blocks and only runs prefill on the novel suffix. In practice it reduces prefill compute and time-to-first-token for requests that share a prefix — and does nothing for prefixes it has never seen. Where does a radix cache sit in the inference engine — the runtime layer, the model compute, or the host glue? It sits in the runtime layer: the serving engine that schedules requests and manages the KV-cache memory pool. It does not modify the model’s forward pass or the host-side glue like tokenisation and networking. It decides which forward-pass work is necessary, which is why it only helps when latency actually lives in repeated-prefix prefill. What is the relationship between a radix cache and the KV cache it indexes? The KV cache is the raw per-sequence storage of attention keys and values, mandatory for decode. The radix cache is a cross-request index and reuse policy over that KV-cache state, keyed by token prefix. It does not replace the KV cache; it decides which blocks can be shared — and competes for the same GPU memory, trading hit rate against batch capacity. Which workloads see a real latency gain from radix caching, and which see almost none? Workloads with long shared prefixes that recur often — agents with fixed system prompts, few-shot preambles, multi-turn chat within a session — see large TTFT reductions. Workloads with distinct one-off prompts, or with variable retrieved context, or with long generations and short prompts, see close to zero. The gain tracks shared-prefix length times recurrence frequency. How do you measure whether prefix reuse is available in your traffic before enabling it? Sample recent production prompts, tokenise them, and measure what fraction of prefill tokens belong to shared prefixes — this gives a theoretical ceiling before you enable anything. Once enabled, track three operational signals: prefix cache hit rate, share of prefill latency eliminated on hits, and throughput under high-overlap load. A near-zero hit rate after warm-up is a complete diagnosis. Why can treating a radix cache as a flat on/off speedup mislead an inference-cost decision? Because the speedup is a property of your traffic’s prefix overlap, not of the cache. Importing an “X% speedup” figure from a benchmark run on high-overlap agent workloads and applying it to a low-overlap RAG service produces projected savings that vanish in production. The correct order is attribution first — locate where latency lives — then apply the lever that touches that phase. How does prefix caching relate to attributing latency to the runtime layer versus host overhead? Prefix caching is a runtime-layer lever, so it only pays off when profiling shows prefill-for-repeated-prefixes is a meaningful share of wall-clock latency. If the latency is host overhead — tokenisation, queuing, serialization — the radix cache optimises a phase that was never the bottleneck. Correct attribution across host, runtime, and model compute is the prerequisite for knowing whether the cache is relevant at all. The useful question is not “should we enable the radix cache?” but “how much of our prefill is redundant prefix recomputation, and where does the rest of our latency actually live?” Answer that with a tokenised sample of real traffic first. If prefix reuse is available, the cache converts recomputation into a lookup; if it is not, you have saved yourself the trouble of tuning a lever that was never connected to your bottleneck — which is the datapoint worth having before you commit.
