KV cache quantization (also written K V cache quantization) stores the keys and values that attention keeps for every earlier token in fewer bits and leaves the model weights alone, and it is a harder problem than weight quantization. Weight quantization is static and applied once after training. KV cache quantization has to run during inference on data that only exists at runtime, which means new rounding error enters at every decoding step and builds up as the output gets longer. A good weight-quantization result therefore tells you little about the cache. KV precision has to be measured per model, per precision and per context length, with checks that look past perplexity.
It is tempting to treat the cache as just one more tensor: if INT4 weights held up, an INT4 cache should too. That belief is understandable, because both are called “4-bit quantization”. It breaks on three points: key outliers sit in fixed channels, cache error compounds over the generation, and whether it saves time depends on the kernel and hardware. The wider argument that quantization is a controlled approximation rather than model damage still applies. The cache just needs different controls.
What the cache holds, and why it ends up dominating memory
During autoregressive decoding, every layer stores a key and a value vector for each earlier token and reads all of them again at each new step. The cache is append-only: keys and values for new tokens go on the end, and older entries stay as they are. The cache grows with every token and every concurrent request.
A 2024 survey reports KVQuant’s measurement on LLaMA-7B: the KV cache used 2% of inference memory at a 512-token sequence and 84% at 128k tokens. The KVQuant authors put it this way: weights are the main memory bottleneck at short sequence lengths, but at long sequences and large batch sizes the KV cache takes over, especially when the weights are already low precision. One practitioner benchmark by Inventive HQ estimates a 7B model at about 4 GB of weights plus another 2 to 4 GB or more of KV entries at 32K+ context. The weight side of that budget is covered in LLM quantization and where accuracy breaks. Architectures such as multi-head latent attention reduce the cache by design instead of compressing it afterwards.
Can a weight-quantization result be used to predict the KV-cache result on the same model?
No. The two operations quantize different tensors, and activation outliers are harder to extract than weight outliers. In the key cache, a few fixed channels have very large magnitudes, which the KIVI authors (2024) tie to earlier findings on activation outliers. The KVQuant authors (2024) note that, unlike with weights, pulling outlier values out of activations is not straightforward because activations are dynamic.
The NQKV authors (2025) describe KV-cache quantization as orthogonal to weight and activation quantization, so the two can be combined. Orthogonal here means two separate levers. It does not mean one lever predicts the other.
They also interact. In Anbeeld’s llama.cpp cache benchmark on Qwen 3.6 27B at 64k context, most symmetric cache presets scored 3 to 5% higher on 99.9% tail precision with IQ4_XS weights than with Q5_K_S weights. The author’s explanation is that higher-precision weights produce richer KV distributions, so quantizing the cache has more to lose. Speed is affected too: Hugging Face reports that combining weight quantization with KV cache quantization can cut generation speed threefold. On accuracy, KVQuant reports small perplexity costs of 0.02 and 0.1 when its cache runs alongside 4-bit and 3-bit weight-only quantization on LLaMA-7B and LLaMA-13B. Only the combination you will actually serve, measured together, settles the question.
Rounding error that compounds as generation runs
The PM-KVQ authors (2025) describe the mechanism. Quantizing keys and values adds error at each decoding step, and the accumulated error grows with the number of generated tokens, which hurts long chain-of-thought models badly. In their 70B test, DeepSeek-R1-Distill-LLaMA-70B scored 69.14% pass@1 on AIME-2024 at 16 bits and 51.88% with a 2-bit KIVI cache. Inventive HQ describes the same effect at the output level: small divergences compound across 32+ layers and hundreds of generated tokens until the response follows a different path.
One source seems to point the other way. The 2026 alignment-collapse study found that 256-token and 512-token outputs got identical safety classifications on 50 AdvBench prompts. The same study cites prior work showing that safety alignment is concentrated in early output tokens. We read the two findings as consistent. On that reading, a refusal is decided early. A long reasoning chain has thousands of steps in which to drift.
Several mitigations keep the first or most recent tokens at higher precision. KIVI keeps a residual window of at most 128 recent tokens at full precision. The Hugging Face Transformers implementation keeps a fixed-size residual cache for the same reason. KVQuant gets perplexity benefits, particularly at 2 bits, by keeping only the first token in fp16, because of the attention-sink effect.
Should keys and values be quantized to the same precision?
Not by default. Keys are the more sensitive half, so a fixed bit budget does better giving keys moderately more. D-Quant (2026) gives the mechanism. An error in a key perturbs the attention logits and can be amplified by the exponential in softmax, while an error in a value enters after softmax and gets averaged by the attention weights.
KIVI (2024) proposed the split most recent works adopt: quantize the key cache per channel and the value cache per token. Per-channel grouping keeps the large key outliers inside their own channels so they do not distort the normal ones. Values show no clear outlier pattern, and per-token grouping keeps each token’s error within that token, which matters because the attention output mixes the values of a few important tokens. KVQuant confirmed the split independently, with a 3.82 perplexity improvement on LLaMA-7B at 3 bits over per-token quantization of both. QServe (2024) observed key outlier channels about 10 times larger than most activation values, with no such pattern in values.
Experiments that give keys and values different bit budgets point the same way. In Kitty (2025), KIVI-K4V2 outperformed KIVI-K2V4. In Anbeeld’s benchmark, the asymmetric q5_0-q4_0 pair has the same size as symmetric q4_1 (31.3% of the bf16 cache) and beat it on all three 99.9% tail tables. The alignment study found that key-only quantization caused 76 to 102% of the alignment damage in eight of nine primary models.
The rule has limits. TurboAngle (2026) found a value-dominated bottleneck on TinyLlama and a key-dominated one on Mistral-7B. Anbeeld also found that raising keys to q8_0 over a q4_1 value cache was no better than a cheaper symmetric pair. Start with keys at higher precision, then confirm per model, because the asymmetry is a strong default and not a law.
Memory saved and quality lost by KV cache quantization bit width
The table collects what the sources report at each cache bit width. The numbers come from different schemes, models and metrics, so read each row as evidence about a named method, not about the bit width itself.
| KV bits | Memory reported | Quality reported | What else the sources show |
|---|---|---|---|
| 8-bit | Not reported in these sources | 8-bit KV largely preserves refusal behaviour (alignment study, 2026); q8_0 output 81.6% similar to f16 (Inventive HQ) | FP8 e5m2 KV in vLLM flipped 30.3% of Qwen’s refusals, and e4m3 flipped 7.1% |
| 4-bit | 3.7x with outlier handling (KVQuant, 2024); about 2.5x for int4 (Hugging Face) | Under 0.02 perplexity cost (KVQuant); almost no accuracy loss on Qwen3-8B and LLaMA3-8B (Kitty, 2025) | Mistral-7B lost 15.2% of refusals at 1.03x perplexity (alignment study, 2026); q4_0 output only 8.3% similar to f16 (Inventive HQ) |
| 3-bit | 4.8x (KVQuant); more than fivefold (QJL, 2024) | Under 0.1 perplexity cost (KVQuant); no accuracy drop versus 16 bits on the tasks tested (QJL) | Results are mostly reported as perplexity or task F1 |
| 2-bit | 6.9x (KVQuant); nearly 8x with channel boosting (Kitty) | Under 0.5 perplexity cost (KVQuant); average drops of 15.23 and 10.15 points for Qwen3-8B and LLaMA3-8B (Kitty) | Refusal collapse in LLaMA (58.1%) and Gemma (91.8%) at 2 bits |
The 4-bit row shows why the scheme matters more than the bit count. KVQuant’s near-lossless 4-bit result uses per-channel pre-RoPE keys, non-uniform datatypes and outlier extraction, and it is measured as perplexity. Inventive HQ’s q4_0 cliff is textual similarity to an f16 reference on a 12-prompt suite, and it describes the drop from q8_0 as “a cliff, not a slope”. Both can be true at once. “4-bit KV” names a storage budget, not a behaviour, so one scheme’s result is not a forecast for another. The alignment study adds that no single bit-width is safe for every model, with collapse onsets spread over a full four bits.
Does quantizing the KV cache change latency, or only memory capacity?
Both, and the direction of the latency change depends on the hardware, the kernel and the batch size. QServe gives the clearest case. Replacing the static 8-bit KV path in a TensorRT-LLM decoding kernel with dynamic per-head 4-bit KV made it 1.7x faster on an L40S and 1.2x slower on an A100. The authors trace the A100 slowdown to dequantization. Each INT4 value needs about 5 ALU operations, and the roofline turning point of the A100’s FP32 CUDA cores is 9.8 Ops/Byte, so the fused KV4 attention kernel becomes compute-bound.
| Setup | Result | Source |
|---|---|---|
| L40S, decode attention kernel, KV8 replaced by per-head KV4 | 1.7x faster | QServe, 2024 |
| A100, same replacement | 1.2x slower | QServe, 2024 |
| Apple Silicon unified memory, fused Metal kernel, Gemma-3 1B | 3 to 8% lower ms/token than fp16 | Apple Silicon int4 KV study, 2026 |
| Apple Silicon, q4_0 cache kernels on Metal | about 42% throughput drop (54 to 31 tok/s) | Inventive HQ |
| Hugging Face Transformers, int4 cache, rising batch size | about 2.5x memory saving, generation slows as batch grows | Hugging Face |
On unified memory, a 3x compressed cache moves 3x less data per decode step, and the fused kernel’s cost stays below that bandwidth saving. The two Apple Silicon rows use different kernels, and Inventive HQ attributes its slowdown to Metal’s slow q4_0 cache kernels. Time-to-first-token barely moves: a workload benchmark of KIVI, TurboQuant, SnapKV and CaM found it merely affected by KV cache optimisations.
Capacity turns into throughput when memory is what limits the batch. With similar peak memory, KIVI allowed up to 4x larger batches and 2.35x to 3.47x higher throughput. At short context the trade is small: Inventive HQ measured savings of 208 MB for q8_0 and 320 MB for q4_0 against f16 at 8K context on a 7B model. The same source notes that the payoff grows sharply with context length. KV quantization pays off in batch serving when memory caps the batch size, and in long-context work when the cache dominates. It rarely pays off for a single user at short context. How a quantized cache fits with prefix reuse and tiered storage is covered in hierarchical caching for low-latency LLM inference.
How should a KV-cache calibration corpus be chosen?
Choose for context length first, then for content. For key scales, a short corpus can be the wrong shape no matter what text it contains. The PM-KVQ authors show why: rotary position embedding gives some key channels periods longer than 32K tokens, so calibrating on 2K-token sequences misses their real distribution. Their fix, positional interpolation with a 2,048-token calibration length, reached accuracy comparable to calibrating on 8,192-token sequences.
In KVQuant, the two halves of the cache are calibrated differently. KVQuant computes per-channel key scales offline from a calibration set, because computing them online would mean updating every key already cached, while per-token value scales can be computed online. In serving, vLLM can estimate FP8 KV scales from a curated calibration dataset through the llm-compressor library.
In KVQuant’s test, calibration content mattered little. KVQuant’s perplexity stayed similar when models calibrated on Wikitext-2 were evaluated on C4 and the other way round. Both are general-text corpora, so that robustness does not cover code, structured output or tool calls, and we would include those formats if the deployment serves them.
A low calibration error does not prove quality. In the 2026 Apple Silicon study, a learned rotation cut calibration MSE by 84.9% but made perplexity 0.017 worse. For behaviour, the alignment study’s diagnostic needed only 20 calibration prompts, increased to 50 for models with very low flip rates. The MLPerf Inference rules allow calibration only on the dataset the benchmark owner provides. Keeping calibration data separate from evaluation data is the same discipline at deployment scale.
Signs that a quantized KV cache is hurting output quality
The strongest evidence that the usual metrics miss damage is the 2026 alignment study’s Mistral-7B result: with a 4-bit cache, perplexity stayed at 1.03x the baseline while 15.2% of AdvBench prompts the model had refused flipped to compliance. Dutta et al. found the same pattern for compressed models, with accuracy differences of 2% or less hiding answer flips of 5% or more. Anbeeld’s benchmark explains why perplexity hides this: it averages over all tokens, so a single broken JSON key gets diluted, and the 99.9% KL divergence against the bf16 baseline is what catches it.
A diagnostic checklist for a deployed KV preset:
- Output divergence on your own prompts. Compare each quantized preset against a full-precision reference on the tasks that matter (correct answers, code that compiles, structured output), not on textual similarity alone.
- Tail KL, not mean KL. In Anbeeld’s runs most presets kept mean KLD below 0.01, while the 99.9% KLD rose from 0.099 at q5_0 to 0.130 at q4_0.
- Answer flips. Count answers that change from correct to incorrect and the other way round between baseline and quantized model.
- Refusal and instruction-following behaviour. Production FP8 KV in vLLM is not exempt: e5m2 flipped 30.3% of Qwen’s refusals.
- Evaluation at the context lengths you serve. Short-context multiple-choice benchmarks and a 30-question passkey test could not tell the cache presets apart.
- Global-context tasks. In the 2026 workload benchmark, KIVI at 4 bits was the most stable across tasks, while aggressive TurboQuant settings lost quality on generation-heavy workloads and summarization was the most sensitive.
- Throughput is not a quality signal. Inventive HQ’s q4_0 run produced output 8.3% similar to f16 at 80.3 tok/s, against 81.8 tok/s for the f16 reference.
Serving stacks give some controls. vLLM’s --kv-cache-dtype-skip-layers flag keeps sensitive layer types, such as sliding-window attention, at the model’s native dtype. For local inference presets, Anbeeld recommends q8_0/q6_0 as the high-end choice and q5_0/q5_0 as the normal quality preset, and places q4 value tiers below that ladder as memory-saving options. Inventive HQ advises using q4_0 only after testing it against f16 on your own task. Taken together, these results point to treating q8_0 as a starting candidate and q4_0 as an exception you have measured, never as a default.
What a benchmark run shows about precision, and what it leaves to your own KV tests
LynxBenchAI does not run KV-cache quantization schemes, and a 26Q3 run does not quantize or validate your model; it runs the fixed catalogue of the 26Q3 release. What it shares with the KV-cache question is the bookkeeping: what was prepared, under whose rules, and what threshold a result had to clear. LynxBenchAI prepares the benchmarked models once, before any target device is known, sends the identical prepared artefacts to every vendor’s hardware, and applies the stricter rule to all vendors wherever their kernels disagree on what an approximation may do. It runs inference at fp16, int8 and fp8, reports each precision separately, and enforces a per-precision correctness threshold before a result counts at all. A workload that fails or times out is recorded as a zero instead of being dropped. MLPerf takes a parallel line for KV caches: its rules handle KV-cache entries under the activation quantization rules, and a quantized model must keep model equivalence and meet the accuracy threshold.
The 26Q3 results show the effect. In the 26Q3 results the NVIDIA L40S scores 752.2 on DistilGPT2 inference at fp16 and 457.3 at fp8. Aggregation weights throughput by the memory a run actually moves, so a lower-precision run has to earn its speed-up. The NVIDIA A100-SXM4-80GB records 0.0 at fp8 on inference models, meaning that precision produced no result on that executor. These scores are comparable within the 26Q3 release only. The same reasoning about what a benchmark must expose before its number can settle a cost question is set out in what an AI performance benchmark has to measure.
To see how your own device handles those precisions, run the LynxBenchAI Personal Edition, the released edition. It runs the 26Q3 catalogue on your machine. Your KV preset still needs its own test on your prompts, because the release measures the executor, not your cache configuration.
Putting a KV precision decision into a serving stack
Calibrating key scales at the context length you deploy, deciding which layers stay at native precision, and checking whether a KV4 attention kernel is compute-bound on the target GPU is engineering work on a specific system. TechnoLynx does that work through its GPU performance engineering practice. The real question for a serving team is not whether 4-bit is safe. It is which failure the product can tolerate, among tail errors in structured output, lost refusals and drift on long reasoning, and whether the evaluation suite would catch it before users do.
