Every LLM serving request that carries a shared system prompt, a fixed set of few-shot examples, or a growing conversation history recomputes tokens that were already computed on the last request. RadixAttention is the observation that this recomputation is avoidable — and the mechanism that makes avoiding it cheap enough to be worth doing in a production serving loop. The naive serving path treats each inference request as isolated. The runtime receives a prompt, computes the key/value tensors for every token through every attention layer, generates the completion, then discards the KV-cache the moment the request finishes. The next request — even one that begins with the exact same 400-token system prompt — starts from zero and pays the full prefill cost again. On a busy endpoint serving a single application, a large fraction of prefill FLOPs go into recomputing token prefixes that were identical to the last request, and the one before that. RadixAttention diverges at exactly that point. Instead of discarding KV-cache entries, it stores them in a radix tree keyed on token prefixes. When a new request arrives, the runtime walks the tree to find the longest cached prefix that matches the incoming tokens, reuses those KV entries directly, and only computes attention for the genuinely new suffix. The shared head — the part every request has in common — is computed once and served from cache thereafter. That is the whole idea, and it is why this matters for anyone paying per GPU-hour. How does RadixAttention work in practice? Attention needs, for every token, a key vector and a value vector at each layer. During prefill, the model computes these for the whole prompt before it can generate the first output token. This is the expensive part of serving a long prompt: prefill cost scales with prompt length, and for a 2,000-token prompt the KV computation dominates time-to-first-token. The insight behind RadixAttention is that KV vectors depend only on the tokens before and including a given position — they are a pure function of the prefix. Two requests that share the first 400 tokens produce byte-identical KV entries for those 400 positions. There is no reason to compute them twice. A radix tree (a compressed prefix trie) is the data structure that makes this lookup fast. Each edge represents a run of tokens; each node holds the KV-cache block for the path from the root to that node. A new request’s tokens are matched against the tree edge by edge. The matched prefix is loaded from GPU memory; the unmatched suffix goes through normal prefill. This is the same structure explained in our walkthrough of what a radix cache is and where it cuts LLM inference latency, applied specifically to the attention KV-cache rather than to a generic response cache. The runtime that popularised this approach is SGLang, which builds RadixAttention into its scheduler and pairs it with prefix-aware batching so that requests sharing a prefix are grouped and served against the same cached blocks. Under the hood the KV blocks live in HBM alongside the model weights, and eviction is driven by an LRU policy over tree nodes — least-recently-matched prefixes are dropped first when memory pressure rises. The mechanics of that prefix trie in isolation are covered in Radix Cache for LLM Inference: how prefix reuse cuts GPU work. What is the difference between RadixAttention and a standard per-request KV-cache? Every modern LLM runtime already keeps a KV-cache within a single generation — that is what lets it generate token N+1 without recomputing tokens 1..N. The difference is scope and lifetime. Dimension Per-request KV-cache RadixAttention Cache lifetime Duration of one request Across many requests, until eviction Shared across requests? No Yes, on matching prefixes Data structure Flat contiguous buffer Radix tree keyed on token prefixes Eviction Freed at request end LRU over tree nodes under memory pressure Wins when Always (intra-request) Prefixes repeat across requests Extra cost None Tree lookup + memory held longer The standard per-request cache eliminates recomputation inside a generation. RadixAttention extends that same principle between generations. Both are the same idea — do not recompute what you already have — applied at two different time scales. Framed that way, RadixAttention is less a new algorithm than the natural completion of an optimisation the field already accepted for the single-request case. Which serving workloads benefit most — and which see little gain? The benefit is entirely a function of how much prefix repeats across your traffic. This is the variable to reason about before adopting anything. Three workload shapes see large gains: Shared system prompts. A production assistant with a 500-token system prompt prepended to every request reuses that entire prefix on every call. The system prompt prefill is computed once and served from cache thereafter. Multi-turn chat. Each turn in a conversation extends the previous turn’s tokens. The growing history is a strict prefix of the next request, so a well-behaved cache serves the whole accumulated context and only prefills the new user message. Batched few-shot calls. Classification or extraction jobs that share a long few-shot preamble across thousands of items reuse the preamble prefix on every item. Two shapes see little to no gain: High-entropy, single-turn traffic where every prompt is unique from token one — ad-hoc search queries, one-off document summarisation with no shared template. There is no common prefix to cache. Workloads where the shared part is small relative to the unique part. If the system prompt is 20 tokens and each user prompt is 3,000 unique tokens, the reusable fraction is tiny and the tree bookkeeping is not worth it. A useful heuristic: estimate the average shared-prefix fraction — the mean number of leading tokens a request shares with something already in cache, divided by total prompt length. If that fraction is high (system-prompt-heavy or chat-heavy traffic), prefix reuse is a strong lever. If it is near zero, RadixAttention is inert and you should look elsewhere, such as SGLang PD disaggregation, which separates prefill and decode for GPU throughput regardless of prefix sharing. How does RadixAttention affect GPU utilisation and cost per useful FLOP? This is where the concept connects to the number an infrastructure lead actually cares about. On the naive path, a GPU-second spent recomputing a shared prefix is billed the same as a GPU-second spent generating a new token — but only the second one produces useful output. Prefix recomputation is idle-but-billed compute: the hardware is busy, utilisation graphs look healthy, yet a slice of that busyness is redundant. RadixAttention converts that redundant slice into headroom. When a prefix is served from cache, the prefill FLOPs it would have consumed are simply not spent, so more of each GPU-second goes to new tokens. In prefix-heavy serving, higher cache hit rates translate into higher sustained tokens-per-second at the same fleet size — a throughput gain and a reduced compute-per-request without additional GPU procurement. The framing that matters is cost per useful FLOP, not raw utilisation. A cluster can run at 95% utilisation and still be wasteful if a third of that compute is recomputing prefixes. Distinguishing useful from redundant work is exactly the lens applied in our comparison of AWS vs Azure for GPU workloads on a cost-per-useful-FLOP basis, and prefix reuse is one of the clearest cases where headline utilisation hides the real story. Note the discipline on numbers here: the magnitude of the gain is workload-dependent and not a fixed multiplier. In configurations with heavy prefix sharing, the reduction in prefill compute can be substantial; in low-sharing traffic it is negligible. Any specific figure quoted without naming the traffic profile that produced it (an observed-pattern claim at best, never a portable benchmark) should be treated with suspicion. What are the limits and overheads of RadixAttention? Prefix reuse is not free, and treating it as a universal win is the failure mode worth naming. First, memory pressure. Holding KV-cache across requests means those blocks stay resident in HBM longer than a per-request cache would keep them. On a memory-constrained deployment, an aggressive prefix cache competes with the model weights and the active-request KV for the same HBM budget. When the cache grows past what fits, eviction kicks in — and if your traffic thrashes the LRU (many distinct prefixes cycling faster than they are reused), you pay the tree bookkeeping cost without the reuse benefit. Second, lookup and management overhead. Walking the radix tree, matching prefixes, and maintaining eviction state costs cycles. For workloads with negligible prefix sharing, this overhead is pure loss. The overhead is small relative to a genuine cache hit, but it is not zero relative to no cache at all. Third, it does nothing for the decode phase. RadixAttention accelerates prefill by reusing prompt KV. It does not speed up the autoregressive generation of new tokens, which for long completions is often the dominant cost. If your latency budget is blown by decode rather than prefill, prefix caching is aimed at the wrong bottleneck — a distinction that only a per-workload measurement will reveal. This is why we treat prefix reuse as one lever among several rather than a headline fix. How do I estimate the gain before adopting it? Before changing a serving stack, quantify the opportunity rather than assuming it. A GPU Performance Audit measures actual utilisation per workload and, for inference serving specifically, quantifies how much compute is being spent recomputing shared prefixes that KV-cache reuse could eliminate. Concretely, that means instrumenting your live traffic to measure the shared-prefix fraction, then modelling the prefill compute that fraction represents — the ceiling on what RadixAttention can recover. You can approximate this yourself: log a representative traffic sample, compute how many leading tokens each request shares with a recently seen request, and multiply the reusable-token count by the per-token prefill cost. That product is the redundant compute you are billing for today. If it is a meaningful share of your inference bill, prefix reuse is worth the engineering; if it is a rounding error, the honest answer is that RadixAttention will not move your cost curve and you should look at decode-side optimisations or model routing instead. FAQ How does RadixAttention actually work? RadixAttention stores KV-cache entries in a radix tree keyed on token prefixes, so that shared leading tokens — system prompts, few-shot examples, conversation history — are computed once and reused across requests. In practice it means the expensive prefill step for a shared prompt head runs once, and subsequent requests matching that prefix skip straight to computing their unique suffix. What is the difference between RadixAttention and a standard per-request KV-cache? A standard per-request KV-cache eliminates recomputation within a single generation and is freed the moment the request ends. RadixAttention extends the same principle across requests: cached prefixes persist in a radix tree and are reused whenever a new request’s leading tokens match, evicted only under memory pressure via LRU. It is the same “don’t recompute what you already have” idea applied at a longer time scale. Which serving workloads benefit most from RadixAttention prefix reuse — and which see little gain? Workloads with heavy prefix sharing benefit most: shared system prompts, multi-turn chat where history accumulates, and batched few-shot calls with a common preamble. High-entropy single-turn traffic where every prompt is unique from token one sees little to no gain, and neither do workloads where the shared prefix is tiny relative to the unique portion of each prompt. How does RadixAttention affect GPU utilisation and cost per useful FLOP during inference? By serving shared prefixes from cache, RadixAttention stops spending prefill FLOPs on recomputation, so more of each GPU-second produces new tokens. In prefix-heavy serving this raises sustained tokens-per-second at the same fleet size, improving cost per useful FLOP even when headline utilisation was already high — because that utilisation included redundant recomputation. What is the KV-cache radix tree, and how does it decide what to keep or evict? The KV-cache radix tree is a compressed prefix trie whose edges represent runs of tokens and whose nodes hold the KV blocks for the path from the root. Incoming requests are matched edge by edge to find the longest cached prefix. Under memory pressure the runtime evicts least-recently-matched tree nodes first via an LRU policy, dropping the prefixes least likely to be reused. How do I estimate the throughput gain from prefix caching before adopting it in my serving stack? Log a representative traffic sample and measure the shared-prefix fraction — the mean number of leading tokens each request shares with a recently seen request, over total prompt length. Multiply the reusable-token count by the per-token prefill cost to size the redundant compute you bill today; that figure is the ceiling on what prefix reuse can recover. A GPU Performance Audit performs this measurement against your live workload. What are the limits and overheads of RadixAttention, and when does it not help? It holds KV blocks in HBM longer, competing with model weights and active-request KV for memory; it adds tree-lookup and eviction overhead that is pure loss when prefixes rarely repeat; and it accelerates only prefill, not the autoregressive decode phase. It does not help high-entropy single-turn traffic or workloads where decode, not prefill, is the dominant cost. The number that actually decides this RadixAttention is best understood as a specific answer to a general question: how much of your GPU is recomputing the same tokens? For a serving stack dominated by shared system prompts or long chat histories, the answer is often “more than you think,” and prefix reuse turns that redundant compute into throughput headroom. For high-entropy traffic it is inert. The honest move is not to adopt it on faith but to measure the shared-prefix fraction in your real traffic first — the same /gpu performance-audit discipline we apply to any utilisation claim, because a mechanism that helps one workload’s cost curve can be dead weight on another’s.
