Hierarchical Caching for Low-Latency LLM Inference: KV-Cache, Prefix, and Tiered Reuse

The second request is where naive LLM caching quietly fails. A user sends a follow-up in the same chat, or a new session arrives sharing the same 2,000-token system prompt and retrieved context, and a flat single-request KV-cache throws all of that away and recomputes it from scratch. The prefill work you already paid for on the first request is discarded the moment that request completes. Hierarchical caching is the answer to that specific waste: instead of one flat cache that lives and dies with a single request, you layer reuse across tiers — a prefix cache for shared prompts, a session cache that persists across turns, and a tiered eviction path that spills warm entries out of GPU memory rather than deleting them.

That difference lands directly on the number that decides whether a real-time GenAI product feels responsive: time-to-first-token. In a streaming interface, the user watches for the first token to appear. If every request pays full prefill cost on a long shared prefix, first-token latency climbs with prompt length and collapses further under concurrency, because prefill is compute-bound and competes with every other request for the same GPU. Reuse the prefill and the first token arrives after you compute only the genuinely new tokens.

What is a KV-cache, and why does recomputing shared prefixes dominate first-token latency?

Transformer inference has two phases with very different cost profiles. Prefill processes the entire prompt at once, computing attention over every token and producing a key/value tensor pair for each layer — this is the KV-cache. Decode then generates output tokens one at a time, each new token attending to the cached keys and values so it never re-reads the prompt from scratch. The KV-cache exists precisely so that decode stays cheap; without it, generating token n would require re-attending to all n-1 prior tokens every step.

The subtlety most teams miss: the KV-cache is scoped to one request. Once generation finishes, the tensors for that prompt are freed. So when the next request arrives sharing most of the same prefix — the same system prompt, the same retrieved documents, the same conversation history — the serving engine runs prefill again over content it computed minutes ago. In a retrieval-augmented setup where the shared context is long and the user’s new turn is short, prefill dominates first-token latency because it scales with the shared prefix, not the new tokens.

That is the divergence point. On the naive path, request two pays full prefill cost. On the hierarchical path, request two reloads a cached prefix and computes only the new tokens. Everything downstream — TTFT, GPU occupancy, cost per million tokens — follows from which path you took. The core claim is blunt: for real-time LLM inference with shared context, redundant prefill is usually the largest controllable latency cost, and reusing the KV-cache across requests is the direct fix.

How prefix caching reuses shared prompts across requests

Prefix caching keys the KV-cache by the token sequence rather than by the request. When a new prompt arrives, the engine matches its leading tokens against cached prefixes. If tokens 0–1,999 match an existing entry, those keys and values are reused verbatim and prefill begins at token 2,000. Because attention is causal — each token only attends to tokens before it — a shared prefix produces identical KV tensors regardless of what follows, which is what makes the reuse mathematically exact rather than an approximation.

The data structure matters. A radix tree lets many prompts share overlapping prefixes efficiently: a system prompt shared by every session is stored once and branches into per-session continuations. This is the mechanism behind prefix reuse in production LLM serving with a radix cache, and the attention-level variant RadixAttention that reuses KV-cache to speed up inference. Serving frameworks such as vLLM (via PagedAttention) and SGLang implement forms of this; the common thread is that KV memory is managed as reusable pages or tree nodes rather than per-request scratch space.

Three levels of reuse are worth naming distinctly, because they have different hit-rate behaviour:

Prompt/prefix cache — shared system prompts, few-shot examples, and retrieved context that recur across many independent requests. Highest overlap when a template dominates.
Session KV-cache — a single conversation’s growing history, persisted between turns so turn three does not re-prefill turns one and two.
Cross-session prefix sharing — distinct users sharing the same leading tokens (a common instruction preamble), served from one cached branch.

How tiered storage extends cache capacity without unbounded growth

The KV-cache is large. A rough sense of scale: for a mid-sized model, KV memory per token runs into the tens of kilobytes across all layers, so a few thousand concurrent long-context sessions can consume most of a GPU’s high-bandwidth memory (HBM) before you have loaded a single extra weight. HBM is finite and expensive. If you only cache what fits in HBM, eviction is aggressive and hit rates fall exactly when concurrency rises.

Tiered caching treats memory as a hierarchy rather than a single pool. Hot entries live in HBM. As pressure builds, warm entries spill to CPU RAM over PCIe or NVLink, and colder entries spill to local NVMe. A cache miss that hits CPU RAM costs a copy back into HBM — far cheaper than recomputing prefill — and a disk hit still beats a full recompute for a long prefix. The eviction policy (typically LRU or a frequency-weighted variant) decides what moves down the hierarchy, and the key design goal is that eviction means demotion, not deletion.

Tier	Access path	Relative latency	Best for
GPU HBM	on-device	fastest	active sessions, hottest prefixes
CPU RAM	PCIe/NVLink copy	moderate	warm sessions likely to return
Local NVMe / SSD	disk read + copy	slow but < recompute	cold-but-reusable long prefixes
Recompute (no cache)	full prefill	slowest under load	last resort; unavoidable on a true cold prefix

(Latency ordering is structural, not a benchmark; the exact ratios depend on model size, context length, and interconnect — observed pattern, not a published rate.)

The bound on growth comes from the eviction policy plus a capacity ceiling per tier. Without a ceiling, a session cache grows unbounded and eventually you are paging so much that the copies cost more than recompute. That crossover point is the thing to measure, not assume.

How cache hit rate maps to latency, throughput, and cost

The single number that ties this together is cache hit rate on shared prefixes. It is the lever that moves everything else, and it is the number a feasibility budget has to be built on.

Consider a worked example with explicit assumptions. Suppose a request carries a 2,000-token shared prefix and a 50-token new turn, and prefill throughput on your GPU is roughly proportional to token count. On a full recompute, prefill processes 2,050 tokens. On a prefix hit, it processes 50. The prefill work drops by around 97% for that request — first-token latency for the prefill portion falls proportionally, and the freed GPU cycles raise the concurrency the same hardware can sustain. The illustrative arithmetic here is meant to size the effect, not to promise it; your real gain scales with your actual prefix-to-new-token ratio and hit rate.

That is why the four metrics in the CCU’s ROI framing belong together:

Time-to-first-token (TTFT) — falls when prefill is reused; the user-visible responsiveness number.
Tokens/sec at target concurrency — rises because prefill cycles reclaimed from cache hits are available for other requests’ decode.
Cache hit rate on shared prefixes — the input variable; low hit rate means the other three barely move.
GPU-hours per million tokens — the cost number; reused prefill is compute you do not pay for again.

The trap in a demo is that you measure the warm-cache TTFT and quote it as the product’s latency. Under real traffic the first request of every new prefix is a cold miss, and cold-cache TTFT is a different, larger number. A GenAI feasibility audit for streaming inference has to state both, because the latency budget the demo showed is not the latency budget production will hit. Sizing that gap correctly is the same discipline we apply to reading DeepSeek inference cost in production — the headline number and the sustained-load number are rarely the same. This is where hierarchical caching sits inside a broader latency-optimisation practice: it is one concrete instance of trading memory tiering against recompute cost on the GPU, the same trade-off that governs query routing for lower-cost, low-latency LLM inference.

How does hierarchical caching relate to attention-sink and streaming-LLM techniques?

For very long-running streams — an assistant open for hours, a document that never resets — the KV-cache eventually exceeds any tier’s capacity no matter how you evict. Streaming-LLM techniques (attention sinks, sliding-window attention) address a different failure: keeping a bounded KV-cache stable as the stream grows past the model’s trained context, by retaining a few initial “sink” tokens plus a recent window. That is complementary to hierarchical caching, not a substitute. Hierarchical caching decides where cached tokens live and when they move; attention-sink methods decide which tokens stay in the attention computation at all once you can no longer keep them all. A long-lived real-time system often needs both: sliding-window attention to bound per-stream growth, and a tiered cache to keep many bounded streams warm across HBM, RAM, and disk.

When hierarchical caching is not worth it

The honest boundary condition. Hierarchical caching earns its complexity when prefixes overlap; it is dead weight, or a net loss, when they do not.

Low prefix overlap. If every prompt is unique — no shared system prompt, no reused retrieval, no multi-turn sessions — hit rate stays near zero and you pay cache-management overhead for nothing.
Tight memory budgets. On a GPU already saturated by model weights, stealing HBM for a KV-cache can hurt more than it helps; the spill machinery adds copies that, at high miss rates, cost more than the recompute they replace.
Highly personalised prompts. If personalisation is injected early in the prompt (per-user data before the shared instructions), it breaks prefix matching because the leading tokens differ per user. Moving invariant content to the front of the prompt is often what makes caching viable at all — a prompt-ordering decision, not an infrastructure one.

The disqualifier to watch for is a measured hit rate that stays low no matter how you tune eviction. That is not a caching problem; it is a signal that your workload does not have the shared structure caching exploits, and the engineering effort belongs elsewhere.

FAQ

What’s worth understanding about hierarchical caching first?

Instead of one flat KV-cache scoped to a single request, hierarchical caching layers reuse: a prefix cache for shared prompts, a session cache that persists across conversation turns, and a tiered eviction path that spills warm entries from GPU HBM to CPU RAM to disk. In practice it means the second request reloads a cached prefix and computes only the new tokens, rather than re-running prefill over shared context.

What is a KV-cache, and why does recomputing shared prefixes dominate first-token latency in real-time LLM inference?

A KV-cache stores the per-layer key/value tensors produced during prefill so that decode can generate tokens cheaply without re-reading the prompt. Because a flat cache is freed when the request ends, the next request re-runs prefill over the same shared prefix — and since prefill scales with the shared prefix length rather than the short new turn, that redundant work usually dominates first-token latency for retrieval-augmented or multi-turn workloads.

How does prefix caching reuse shared system prompts, retrieved context, and conversation history across requests?

Prefix caching keys the cache by token sequence rather than by request, so leading tokens that match an existing entry are reused verbatim and prefill starts after the match. Because attention is causal, a shared prefix produces identical KV tensors regardless of what follows, making the reuse exact. A radix tree lets many prompts share overlapping prefixes, storing a common system prompt once and branching per session.

How do tiered storage layers and eviction policies extend cache capacity without unbounded growth?

Hot entries stay in GPU HBM, warm entries spill to CPU RAM, and colder entries spill to local NVMe, so eviction means demotion rather than deletion — a cache miss served from RAM or disk still beats a full prefill recompute. Growth is bounded by a per-tier capacity ceiling plus an eviction policy such as LRU; the crossover point where paging costs exceed recompute is a number to measure, not assume.

How does cache hit rate map to time-to-first-token, throughput per GPU, and cost per million tokens?

Cache hit rate on shared prefixes is the input variable that moves everything else: higher hit rate reuses more prefill, lowering TTFT, raising tokens/sec at target concurrency because reclaimed cycles serve other requests, and cutting GPU-hours per million tokens. Low hit rate means the other three barely move, which is why a warm-cache TTFT from a demo overstates production latency, where cold-miss first requests are unavoidable.

How does hierarchical caching relate to attention-sink / streaming-LLM techniques for long-running streams?

They are complementary. Hierarchical caching decides where cached tokens live and when they move across HBM, RAM, and disk. Attention-sink and sliding-window methods decide which tokens remain in the attention computation once the stream outgrows the model’s trained context. A long-lived real-time system often needs both — sliding-window attention to bound per-stream growth and a tiered cache to keep many bounded streams warm.

When is hierarchical caching not worth it — low prefix overlap, tight memory budgets, or highly personalised prompts?

It stops paying off when prompts do not share structure. With low prefix overlap, hit rate stays near zero and you pay management overhead for nothing. On a GPU already saturated by weights, the spill machinery can cost more than the recompute it replaces. And when personalised content is injected before shared instructions, it breaks prefix matching — often fixable by moving invariant content to the front of the prompt, a prompt-ordering decision rather than an infrastructure one.

The question a latency budget really turns on is not “how fast is the cache?” but “what is the sustained cache hit rate under the concurrency this product will actually see, and what is the cold-miss TTFT when it isn’t hit?” Answer those two before you commit to a streaming latency budget — the warm number a demo shows is not the number production has to hold.