A team hits a throughput ceiling on their LLM serving cluster and reaches for the obvious lever: more GPUs. Before the purchase order goes out, it is worth asking whether the cluster is actually compute-bound — or whether it is paying, over and over, to recompute the same prompt prefixes. That distinction is the whole story of RadixAttention, and it is the difference between buying capacity you do not need and reclaiming throughput you already own.

RadixAttention is a KV-cache management strategy that organizes cached attention state in a radix tree so that shared prompt prefixes — system prompts, few-shot examples, common conversation heads — are computed once and reused across requests. The naive serving model treats every request as an independent forward pass with its own ephemeral cache. Where requests overlap, that model quietly pays for the same prefill compute again and again. RadixAttention removes that duplication automatically, and the savings land exactly where prefix overlap is highest.

What is the KV cache, and why does recomputing prefixes waste GPU time?

To generate the next token, a transformer needs the key and value tensors for every preceding token in the sequence. These are produced during the prefill phase, when the model processes the input prompt in one pass, and they are stored so the decode phase can attend to them without recomputing attention over the full history at every step. That stored state is the KV cache, and on any realistic model it is large: capacity scales with sequence length, layer count, and the number of attention heads, so a long shared prompt consumes real high-bandwidth memory (HBM) per request.

Here is where the naive model leaks. Consider a serving workload where every request carries the same 800-token system prompt plus a handful of few-shot examples. If the cache is treated as private per-request state, each incoming request runs prefill over that identical prefix from scratch. The GPU does the same matrix multiplications thousands of times a minute and produces bit-identical KV blocks each time. That is not compute doing useful work — it is compute burning cycles to rediscover something the cluster already knew. In workloads with heavy prompt overlap, redundant prefill can dominate the serving cost, and no amount of extra hardware makes recomputation cheaper. It only makes it faster to keep wasting.

The point generalizes beyond system prompts. Multi-turn conversations share a growing common head as the dialogue extends. Batched evaluation runs share a fixed instruction template. Tree-of-thought and self-consistency sampling branch from a common context. Each of these is a case where the prefix is shared even when the completions diverge — and the prefix is exactly the part the cache can serve without recomputing.

How does the radix tree structure enable automatic prefix reuse?

A radix tree (a compressed trie) stores strings as paths, where each edge represents a run of tokens and shared prefixes converge on shared nodes. RadixAttention maps this structure onto KV-cache blocks: the tokens of each request form a path through the tree, and the KV state for those tokens is attached to the corresponding nodes. When a new request arrives, the serving runtime walks the tree to find the longest matching prefix that is already cached. Everything up to that match point is reused directly; only the divergent suffix needs prefill.

The value of this structure is that reuse becomes automatic and content-addressed. There is no need for the application to declare “these requests share a prefix” or to pin a specific prompt as a cache key by hand. The tree discovers overlap by matching token sequences, so a system prompt, a shared few-shot block, and an in-progress conversation head are all handled by the same mechanism without special-casing. This is the property that separates RadixAttention from ad-hoc prompt caching: it is a general prefix-sharing index, not a lookup table for a fixed set of known prompts.

RadixAttention became widely known through its role in the SGLang serving runtime, where it works alongside the runtime’s scheduler to keep high-hit-rate prefixes resident. If you want the narrower framing of the underlying data structure on its own, we cover what a radix cache is and where it cuts LLM inference latency as a companion piece; the mechanics there feed directly into how RadixAttention decides what to keep.

Which inference workloads benefit most — and which see little gain?

The honest answer is that RadixAttention is a conditional win, and the condition is prefix overlap. A workload where every request is a unique, low-overlap prompt with no shared instruction block and no multi-turn history will see almost nothing from prefix reuse, because there is no shared prefix to reuse. Selling RadixAttention as a universal throughput multiplier is the mistake; the discipline is to measure your actual overlap first.

The table below is a decision rubric for reasoning about fit before committing to a serving architecture change.

Where prefix reuse pays — a fit rubric

Workload pattern	Prefix overlap	Expected benefit	Why
Fixed system prompt + few-shot template across all requests	Very high	Large	Long shared prefix computed once, reused every request
Multi-turn chat with growing conversation history	High	Large	Each turn reuses the full prior turn’s KV state
Tree-of-thought / self-consistency sampling from one context	High	Large	Many branches share the root prompt
RAG with a stable instruction wrapper, variable retrieved context	Medium	Moderate	Instruction prefix reused; retrieved chunks diverge
Single-turn requests with unique prompts, no shared template	Low	Minimal	Little or no prefix to share
High prompt diversity, short prompts, cache churns constantly	Low	Minimal to negative	Eviction overhead can outweigh reuse

The rubric is a starting hypothesis, not a promise (this reflects the structure of the mechanism and patterns observed across serving workloads, not a benchmarked figure for your cluster). The operational number that settles it is the prefix cache hit rate: the fraction of incoming tokens served from cached KV state rather than recomputed. Measure that against a representative traffic sample before you conclude anything about throughput.

How does prefix reuse translate into throughput and inference cost?

The mechanism cuts redundant prefill compute. On a fixed GPU footprint, freeing prefill cycles does two things: it lowers time-to-first-token for requests that hit a warm prefix, and it frees compute that the scheduler can spend advancing decode for other requests in the batch. The net effect is higher effective tokens-per-second at a given batch size, concentrated in workloads with high overlap.

The framing that matters for procurement is GPU-hours saved per million requests, not raw hardware count. If a workload has a 70% prefix hit rate, roughly 70% of prefill token-processing is served from cache rather than recomputed — an illustrative figure, not a measured rate, and the realized saving depends on how much of your total compute is prefill versus decode. A decode-heavy workload with long generations gains less proportionally than a prefill-heavy workload with short completions and long shared prompts, because the reuse only touches the prefill side. This is why “we just need more GPUs” is so often the wrong reaction to a throughput ceiling. A cache-reuse problem does not get cheaper with more hardware; it gets cheaper with better cache management. Understanding where inference cost actually accrues is the same reasoning we apply across the GPU engineering and cluster-sizing work — you size capacity against useful work, not against the recomputation you could have eliminated.

Prefix reuse also interacts with other serving optimizations. Separating the two phases so prefill and decode run on differently-tuned resources — the pattern behind SGLang PD disaggregation for GPU throughput — compounds with prefix caching, because a warm cache reduces exactly the prefill work that disaggregation is trying to schedule efficiently. The two are complementary levers, not competing ones.

What are the memory and eviction trade-offs?

Nothing about RadixAttention is free. Cached KV blocks occupy HBM that would otherwise hold larger batches or longer contexts, so the runtime must decide what to keep and what to evict when memory pressure rises. The radix tree makes eviction tractable — least-recently-used or reference-count policies operate on tree nodes, and evicting a leaf reclaims memory without breaking the shared prefixes above it — but the policy still has to balance hit rate against the batch capacity the memory could otherwise serve.

Two failure modes are worth naming. First, a high-diversity workload can churn the cache so hard that eviction overhead and near-zero hit rates make the tree pure overhead; in that regime the naive per-request model is genuinely fine and RadixAttention buys nothing. Second, an over-aggressive cache that holds cold prefixes starves the batch of memory, cutting the concurrency that drives throughput — you can win on prefill and lose more on batch size. The right cache footprint is a tuning decision measured against your traffic, not a fixed setting. In our experience, the teams that get the most from prefix caching are the ones who instrument hit rate and memory pressure together and treat the cache size as a load-dependent parameter, not a constant.

FAQ

How does RadixAttention actually work?

RadixAttention manages the KV cache using a radix tree, where each request’s tokens form a path and shared prefixes converge on shared nodes. When a request arrives, the runtime finds the longest already-cached prefix and reuses its KV state, running prefill only on the divergent suffix. In practice this means shared system prompts, few-shot blocks, and conversation heads are computed once and reused across requests automatically, without the application declaring the overlap.

What is the KV cache, and why does recomputing shared prompt prefixes waste GPU time?

The KV cache stores the key and value tensors for every preceding token so decode can attend to history without recomputing attention each step. When each request is treated as independent, the GPU re-runs prefill over identical shared prefixes and produces the same KV blocks repeatedly. In high-overlap workloads that redundant prefill can dominate serving cost, and adding hardware only makes the waste faster, not smaller.

How does the radix tree structure enable automatic prefix reuse across requests?

A radix tree stores token sequences as paths where common prefixes share nodes, and RadixAttention attaches KV-cache blocks to those nodes. A new request is matched against the tree to find its longest cached prefix, which is reused directly. Because matching is content-addressed on the token sequence, reuse is automatic — no manual prompt keys or special-casing for known prompts.

Which inference workloads benefit most from RadixAttention, and which see little gain?

Workloads with high prefix overlap benefit most: fixed system-prompt-plus-few-shot templates, multi-turn chat, and branching sampling from a common context. RAG with a stable instruction wrapper sees moderate gains. Single-turn requests with unique, short prompts and no shared template see little or nothing, and high-diversity traffic can even make cache churn a net overhead. Measure your prefix cache hit rate before assuming a benefit.

How does prefix cache reuse translate into throughput and inference-cost improvements on a fixed GPU footprint?

Reuse eliminates redundant prefill compute, lowering time-to-first-token on warm prefixes and freeing compute the scheduler can spend on decode, which raises effective tokens-per-second at a given batch size. The procurement-relevant metric is GPU-hours saved per million requests rather than hardware count. The realized saving depends on how much of your total compute is prefill versus decode, so prefill-heavy, high-overlap workloads gain most.

What are the memory and eviction trade-offs when caching KV blocks for reuse?

Cached KV blocks occupy HBM that could otherwise hold larger batches or longer contexts, so the runtime must evict under memory pressure — typically with LRU or reference-count policies on tree nodes. Over-caching cold prefixes starves the batch and cuts the concurrency that drives throughput, while high-diversity traffic can churn the cache into pure overhead. The right cache footprint is a load-dependent tuning decision, measured against real traffic.

The question worth carrying out of this is not “is RadixAttention good” but “what is my prefix hit rate, and what fraction of my compute is prefill versus decode.” Answer those two before sizing a cluster, and the throughput ceiling usually turns out to be a cache-management decision rather than a hardware one.