KV cache memory per request equals 2 × layers × heads × head dimension × bytes per element × tokens. For a 4,096-token Llama 2 7B request that product is about 2 GB, and it keeps growing while the request decodes. That growth is why admission in a multi-tenant LLM fleet has to account for each request’s future footprint, not only whether it fits right now. A check that only asks “does it fit today?” admits requests that collide later and evict each other. The eviction policy that decides which cached blocks to drop (LRU, priority-based or workload-aware) should be tailored to how the traffic reuses cache. Change the engine’s default policy only against a measured cache hit rate.
Calculating KV cache memory per request
During decoding, a transformer keeps one key vector and one value vector per layer for every token processed so far. That store is the KV cache. NVIDIA’s inference-optimisation guide gives the per-token size as 2 × num_layers × (num_heads × dim_head) × precision in bytes, where the 2 covers K and V. The total for a batch is that figure multiplied by batch size and sequence length. The model attributes you need are in the model card or config file.
The formula above counts every attention head as storing its own K and V. Multi-query and grouped-query attention (GQA) cut the number of stored key and value heads, so the term to use is KV heads × head dimension, not the full hidden size. Plugging hidden size into a GQA model overstates the cache by the GQA factor.
Worked example: Llama 2 7B, and what grouped-query attention changes
NVIDIA’s example: Llama 2 7B in 16-bit precision, batch size 1, a 4,096-token sequence. That is 1 × 4096 × 2 × 32 × 4096 × 2 bytes, roughly 2 GB of KV cache. The weights of the same model take about 14 GB in FP16.
| Quantity | Working | Result |
|---|---|---|
| Per-token KV, Llama 2 7B (32 layers, 32 heads × 128 dim, FP16) | 2 × 32 × (32 × 128) × 2 bytes | 524,288 bytes (512 KiB) |
| One 4,096-token request | 512 KiB × 4,096 | 2 GiB |
The table figures are arithmetic from the model’s configuration, not a measurement. Published sizes show the grouped-query effect on real models. In the CachedAttention paper (USENIX ATC 2024), LLaMA-65B needs 2.5 MB per token, while LLaMA-70B needs 0.31 MB and Falcon-40B 0.12 MB because they use GQA with factors of 8 and 16.
The per-token number is only half of the accounting. The other half is how many tokens a request will have when it peaks, which is not known at admission.
Does the KV cache live in GPU memory, and when does it spill to CPU or disk?
Yes, the working KV cache lives in GPU memory. It has to sit in high-bandwidth memory (HBM) for attention to run fast, and unlike disk-backed virtual memory, HBM cannot be overcommitted. Decode reads that cache on every step, which is what makes the decode phase memory-bandwidth-bound.
Fleets add capacity by moving cache down a tier hierarchy. In the systems described in the cloud provider’s trace study, new KV blocks land in HBM first. When HBM is full, an eviction policy pushes blocks to CPU memory, swapped asynchronously layer by layer, and blocks are dropped when CPU memory fills too. CachedAttention adds disks below host memory and uses hints from the job scheduler to prefetch the cache that waiting jobs will need.
How much extra tier you need depends on the workload. On the cloud provider’s chat trace, Llama3-70B needed about 4× the HBM reserved for KV cache to approach an ideal hit rate. The authors note that an 8×A100 node with 1 TB of CPU memory already offers roughly that per GPU. On their API trace the required capacity was smaller than the reserved HBM.
When an active request runs out of room, systems such as vLLM and Sarathi-Serve preempt it and recompute its cache later, while DistServe offloads the cache to host memory. In vLLM V1 the default preemption mode is RECOMPUTE rather than SWAP, because recomputation has lower overhead in that architecture. Recompute cost scales with sequence length: the Llumnix authors (OSDI 2024) measured 3.5 s to recompute an 8k-token sequence on LLaMA-30B, about 54 decode steps of stall.
Why KV cache memory caps concurrent long-context requests
The number of long-context requests you can run at once is bounded by the HBM left after weights, divided among their KV footprints. The footprint grows linearly with batch size and sequence length, which directly limits throughput for long-context work. Two published examples show the scale:
- Serving a 175B-parameter model at batch size 64 with 4,096-token contexts, the KV cache can occupy 1.2 TB against 350 GB of weights.
- On 4 A100 80 GB GPUs running LLaMA-65B, the 190 GB of HBM left after weights fills in about 14 seconds when KV is generated at the 2K-token prefill rate of about 13.9 GB/s.
How memory is allocated decides how much of that ceiling you actually get. When runtimes reserve the maximum sequence length up front, much of the reservation goes unused, which wastes memory and shrinks the batch. FasterTransformer, for example, reserves 2,048 tokens for GPT-NEOX even when a sequence ends after 50. PagedAttention splits the cache into fixed-size blocks allocated as tokens arrive, which brings KV waste close to zero. Across a cluster, though, external fragmentation still hurts prefill, because a prompt needs many blocks on one instance at once.
Shrinking bytes per token is the other lever. KV cache quantization affects accuracy differently by method and task, so it deserves its own evaluation. Whichever way the per-token figure moves, the admission logic below still applies.
How do I admit or reject a request before it starts decoding?
Admit a request only if its projected peak KV footprint (prompt tokens plus expected output tokens, times bytes per token) fits within the memory that will be free over its lifetime. Also cap the rate of admissions so that requests admitted together do not all peak together.
TechnoLynx can build this layer for AI infrastructure and production SaaS teams running multi-tenant serving fleets. A TechnoLynx engagement starts by characterising your tenant traffic, SLA tiers, context-length distribution and recent overcommit incidents. Then TechnoLynx calibrates a footprint model against your runtime and hardware, wires occupancy, projection and SLA-tier signals into the scheduling path, and replays captured traffic against candidate eviction policies before anything ships. After deployment, TechnoLynx monitors occupancy, rejection rates and SLA violations and tunes the thresholds.
Why a one-step fit check is not enough
At decode step j, a request holds its input context plus j generated tokens plus one slot reserved for the next step. A request that fits at admission can stop fitting simply by continuing to run. The 2026 paper on service-induced congestion models the greedy baseline common in serving systems, which admits whenever the next step fits and reserves nothing for growth. It shows that this policy creates synchronised memory peaks later, triggering eviction cascades that throw away computation and collapse throughput. Their fix is a per-iteration admission cap set at the eviction-free equilibrium rate. In their simulator and real-GPU experiments this cap reduced cascades while keeping the system close to full utilisation. Reserving for the worst case also fails, because it suppresses concurrency and leaves GPU capacity idle.
Projecting the footprint when output length is unknown
Output length is not known until a request finishes, and that is why this is hard. Prompt length is known at arrival, and how long a request has already waited affects how much delay it can still tolerate. Some systems predict the output length. S3 fine-tuned a DistilBERT classifier to pick one of ten output-length buckets, reaching 98.61% bucket accuracy on Alpaca. It then packed sequences against HBM with first-fit-decreasing. When a prediction was too short, a supervisor preempted the sequence once it used up its reservation and moved its state back to the request pool.
Predictions miss, so the reservation needs a margin, and the right margin depends on which failure costs you more. A framework for output-length uncertainty models under-reservation as preemption and recompute cost, and over-reservation as lost concurrency. The ratio of those two costs sets the best reservation quantile. Batch workloads tolerate preemption and real-time systems do not, so one global quantile will not suit chat, RAG and code-generation classes at the same time. Set reservations per request class rather than apply one global percentile.
Which signals should drive admission in real time?
Published systems admit on current KV occupancy, the request’s projected memory and its SLO, each in a different system. Queue depth and a per-tenant fairness counter refine them. The table maps each signal to a published mechanism.
| Signal | What it tells the controller | Published example |
|---|---|---|
| KV cache occupancy | Room left right now | The Kubernetes Gateway API Inference Extension (CNCF, April 2025) sends sheddable requests only to endpoints under 80% KV utilisation with fewer than 5 queued requests, and drops them otherwise |
| Projected request memory | Whether its peak fits over its lifetime | S3 packs predicted per-sequence KV sizes against HBM capacity; SLOs-Serve admits only while cumulative demand stays below the token budget |
| Virtual usage with priority headroom | Load including queued work and reserved space | Llumnix counts queued requests in instance load and adds headroom to high-priority requests so their decode speed holds |
| SLA tier and deadline | Which requests to protect or reject early | OmniServe admits a latency-sensitive request only if queuing plus prefill time fits its prefill SLO |
| Per-tenant service received | Whether one tenant is crowding out others | The Virtual Token Counter tracks weighted input and output tokens per client and serves the lowest counter first |
| Eviction rate | Early warning that admission is outrunning capacity | Persistent eviction signals operation beyond the eviction-free region |
In OmniServe’s ablation, latency-sensitive TTFT attainment fell sharply without admission control as load rose, and held at 94.1% with it enabled. SLOs-Serve reports a geometric-mean 2.2× gain in serving capacity at 90% SLO attainment over the better of Sarathi-Serve and vLLM. That gain comes from several mechanisms combined, not from admission control alone. Blunt per-tenant rate limits trade throughput for fairness. In the VTC evaluation, a requests-per-minute limit of 5 cut throughput to about 340 output tokens per second against about 779 for VTC.
The controller sits on the scheduling path, so its own overhead needs measuring. Both studies report small overheads: 79 µs per eviction decision for a workload-aware policy in vLLM, and 1.18 ms of predictor inference in PBKV, under 10% of its 12.34 ms mean decode step. A controller that runs heavier than that should be measured against decode time before it ships. Calibration matters in both directions. Reserve too much and GPU capacity sits idle; admit too loosely and eviction cascades discard computation and collapse throughput.
LRU, priority or workload-aware eviction: which one holds up under your traffic?
No single policy wins everywhere, and the published evidence says the answer depends on how your tenants reuse cache. LRU is the default in TensorRT-LLM and SGLang. Both engines also expose priority-based options. TensorRT-LLM lets callers set a retention priority and a duration per block. SGLang’s priority policy evicts the lowest-priority request’s prefix first, so cache retention follows the admission ranking. An SGLang policy only scores candidates; it cannot pin KV, and a protected node is still evicted if nothing else is left to free.
Where LRU falls short, the published reasons are specific:
- Mixed turn counts. On the cloud provider’s chat trace, a flood of single-turn requests can push out 10-turn conversations under LRU, even though those conversations are more likely to return. A workload-aware policy that ranks blocks by profiled reuse probability per request category gave 3.9% more cache hits and up to 41.4% better mean response time than LRU and LFU on production traces.
- Known future access. CachedAttention’s scheduler-aware eviction, which spares cache that queued jobs will need, beat LRU by 27% and FIFO by 31% in overall hit rate at 128 GB DRAM with 2 TB SSD.
- Agent workflows. On a multi-agent workload, LRU’s hit rate stayed below 30%. LRU also exhausted memory and crashed in 3 of 10 trials at concurrency 84 and in all 10 at 96.
When eviction happens inside a request’s own cache at decode time, a separate issue dominates. A study ran seven policies, LRU among them, under a globally capped decode-time harness. All seven collapsed to near-zero quality unless prompt-boundary positions were protected. Production patterns such as pinned prefix blocks provide similar protection implicitly. Token-level compression and eviction inside the attention engine are a GPU-kernel topic in their own right; for fleet admission, the point is that boundary protection should be in place before you compare policies.
Replaying tenant traffic before an eviction policy goes live
Benchmark admission and eviction policies on captured traces of your own traffic, under load, on the target hardware. Generic benchmark results do not say which policy suits your deployment, because the best choice depends on the workload. Several of the studies cited here state this caveat themselves. S3’s authors assumed public Q&A datasets stand in for real request traces because none were available. The provider-trace study covers one week of chat and API traffic and leaves newer reasoning workloads uncharacterised. The boundary-protection recovery is reported for the harness the authors tested.
A replay run looks like this:
- Capture traces with the fields the policies read: arrival time, tenant, SLA tier, prompt and output lengths, and prefix-sharing structure. Anonymised traces can be replayed by generating token lists that reproduce the original hit pattern and by forcing the recorded output lengths.
- Fix the baseline first. Run the engine’s default (LRU in SGLang) and enable its cache metrics, since SGLang’s documentation advises changing policy only against a measured hit rate.
- Sweep load past the knee. Below about 60 concurrent workflows, PBKV’s authors saw evictions too rare to tell policies apart. Push concurrency until evictions and preemptions occur.
- Use a simulator to screen, then real GPUs to confirm. Vidur reports latency and throughput errors under 5% against real systems, and the congestion study used it alongside real-GPU runs.
- Score against your SLAs, not hit rate alone. Report TTFT and inter-token latency attainment per tier, throughput, preemption count and rejection rate. A higher hit rate means little if a premium tier misses its TTFT target.
- Schedule re-validation. Traffic mix and prompt templates drift, and reservation policies fitted to old traces degrade. EVICPRESS re-profiles when predicted and observed quality diverge, and on an Azure trace that recovered about 11% quality over a stale profile.
Replay results are tied to the tested traffic mix, which drifts over time. A new tenant class, a longer system prompt or an agent framework rollout is enough reason to run the replay again.
When fixed concurrency is enough
Not every fleet needs this machinery. With a single tenant, low variance in request length and comfortable memory headroom, a fixed concurrency cap or a static KV budget is easier to reason about and fails more predictably. In vLLM, fixing the KV cache size by flag caps concurrency and throughput if conservative, and fails at allocation if optimistic. Sophisticated policies also matter less when memory is plentiful. The EVICPRESS authors report that their advantage diminishes when DRAM can hold most of the cache.
The case for full admission control rests on bursty arrivals, long and variable contexts, and tenants on different SLA tiers. This is also where serving-path tuning (admission, caching, scheduling) can relieve cost and latency pressure before anyone proposes a bigger GPU or a different model. The question to settle for your fleet is how your tenant mix distributes output lengths and reuse, because that decides the reservation margin, the eviction policy and how often both need revisiting.
FAQ
Is KV cache used in training or inference?
The KV cache is used during autoregressive decoding at inference. It stores the keys and values of earlier tokens so that autoregressive decoding does not reprocess the whole sequence for every new token.
What does the KV cache contain?
It holds a pair of key and value vectors per layer for every token processed so far, covering the prompt and all generated tokens.
What does hit ratio mean for a KV cache?
Hit ratio is the share of KV blocks a workload needs that can be reused from an earlier request instead of being recomputed. On the cloud provider’s production traces it was 62% for the consumer-facing trace and 54% for the business API trace.
