Pick an open-weight model on parameter count and benchmark scores alone and you will size your serving hardware from the weight footprint. That is the wrong number. On a fixed GPU budget, the key/value cache is usually what caps concurrency — and the cache is a property of the attention mechanism, not of the parameter count printed on the model card.
Multi-head Latent Attention (MLA), the attention variant used in DeepSeek’s models, exists to attack exactly that ceiling. Instead of caching a full key and value tensor per head per token, MLA projects keys and values down into a single shared low-rank latent vector, caches that, and reconstructs the per-head keys and values on the fly during decoding. The cached object shrinks; the reconstruction cost moves into compute.
Why the KV cache, not the weights, limits serving capacity
During autoregressive decoding, every token already generated must remain addressable by attention. Standard multi-head attention keeps this history as explicit key and value tensors: KV cache bytes grow linearly with context length, number of layers, number of heads, head dimension, batch size, and bytes per element — while model weights stay constant regardless of how much text is in flight. Weights are a fixed rent. The cache is a variable one, and it is the variable term that decides how many concurrent requests fit in the same HBM.
The consequence shows up as a specific planning error we see regularly. A team checks that a model’s weights fit in device memory, provisions accordingly, and only discovers the real limit in load testing — when long prompts and a realistic batch size push the cache past whatever headroom was left. Nothing about the model changed; the sizing exercise simply measured the wrong quantity. This is the same architecture-selection discipline our generative AI engineering practice argues for at the family level — GAN versus diffusion versus autoregressive — pushed one level down, inside the transformer family itself.
What is Multi-head Latent Attention, in one paragraph?
MLA replaces per-head key/value caching with a low-rank bottleneck. Each token’s hidden state is projected into a compressed latent vector — one per token per layer, shared across heads — and that latent is what the cache stores. When attention runs, up-projection matrices expand the latent back into per-head keys and values. In DeepSeek’s reported configurations, the low-rank latent cache reduces per-token KV cache bytes by roughly an order of magnitude relative to full multi-head attention, which is why the same GPU can hold substantially more concurrent sequences or longer contexts. The compression is learned and lossy in the linear-algebra sense: the up-projection reconstructs an approximation, and the model is trained with that approximation in the loop rather than having it bolted on afterwards.
How MLA differs from MQA and GQA
MLA is not the first attempt to shrink the cache. Multi-query attention (MQA) and grouped-query attention (GQA) got there earlier by sharing key/value heads rather than compressing them. The distinction matters when you are comparing two open-weight models and trying to work out which will actually serve cheaper.
| Mechanism | What gets cached | How the saving is achieved | Main trade-off |
|---|---|---|---|
| Multi-head attention (MHA) | Full K and V per head, per token, per layer | No saving — baseline | Largest cache; concurrency ceiling hit first |
| Multi-query attention (MQA) | One shared K/V head for all query heads | Head-count reduction to 1 | Most aggressive sharing; quality loss reported at the extreme |
| Grouped-query attention (GQA) | One K/V head per group of query heads | Head-count reduction to g groups | Tunable middle ground; still linear in group count |
| Multi-head latent attention (MLA) | One low-rank latent vector per token, per layer | Dimensional compression, then up-projection at attention time | Extra projection compute; positional-encoding handling needs care |
The structural point: MQA and GQA reduce how many K/V tensors you keep, while MLA reduces how large the cached representation is. They are different axes, and MLA keeps the full complement of query heads active — the expressive width of attention is preserved, and only the cached intermediate is squeezed.
What MLA costs
Nothing about this is free, and treating MLA as a strictly dominant choice is the mirror of the mistake it fixes.
- Projection compute. Down-projecting on write and up-projecting on read adds matrix multiplications on every decode step. On memory-bandwidth-bound decoding this is often a good trade, since you are converting bytes moved into FLOPs executed — but it is a trade, and it can invert on hardware where compute, not bandwidth, is the binding constraint.
- Positional encoding interaction. Rotary position embeddings (RoPE) apply a position-dependent rotation to keys, which does not commute cleanly with an arbitrary learned up-projection. MLA implementations handle this with a decoupled path — carrying position-bearing components separately from the compressed latent — and that structural detail is where naive re-implementations tend to go wrong.
- Quantisation interaction. A compressed latent has less redundancy to spare than a full K/V tensor, so aggressive cache quantisation stacked on top of MLA is not additive in the way teams hope. Measure the combination; do not assume the savings multiply.
- Quality is an empirical question. The reconstruction is approximate. Whether that approximation costs anything on your task is something to evaluate, not something to infer from a leaderboard row.
Where MLA sits in the taxonomy — and when it should move a decision
MLA is not a new generative model family. It is an efficiency change inside the autoregressive transformer, sitting alongside quantisation, attention kernel choices such as FlashAttention, and paged cache management in inference servers like vLLM or TensorRT-LLM. It changes the serving economics of a family; it does not create one.
That bounds when it should influence a model choice:
- Relevant when you serve long contexts, high concurrency, or both on a fixed GPU allocation, and cost per million output tokens is a live constraint.
- Relevant when two candidate models are close on task quality and the decision comes down to what each will cost to run at your target load.
- Largely irrelevant for short-context, low-concurrency, or batch-offline workloads, where the cache never approaches the ceiling and prefill compute dominates.
- Irrelevant as a proxy for capability. MLA says something about memory behaviour, not about whether the model is good at your task.
What to measure before you believe the benefit
Vendor-reported ratios are a hypothesis about your deployment, not a result from it. Three measurements, taken at the context length you actually serve, settle it:
- Bytes of KV cache per token per layer — instrument the serving runtime rather than deriving it from the config file, since paged allocators round up.
- Tokens per second at target concurrency — not single-stream latency, which hides the concurrency effect entirely.
- Cost per million output tokens — the number that survives contact with finance, and the one that reveals whether the extra projection compute ate the memory saving on your specific hardware.
Run all three before and after the architecture change, on the same stack, with the same batching policy. When we are asked whether a shortlisted generative model can be served inside an existing infrastructure budget, this is the measurement set that answers it — and attention-level memory behaviour is a routine part of that assessment rather than a footnote.
Frequently Asked Questions
What is Multi-head Latent Attention (MLA), and why does it cut inference memory cost versus standard multi-head attention? MLA projects each token’s keys and values into a single shared low-rank latent vector and caches that vector instead of full per-head key and value tensors, reconstructing them with up-projection matrices at attention time. Because the cached object is dimensionally compressed rather than duplicated per head, per-token cache bytes drop sharply — roughly an order of magnitude in DeepSeek’s reported configurations.
How does the KV cache grow in standard multi-head attention, and why does it — not the model weights — usually limit serving capacity? Cache size scales with context length, layer count, head count, head dimension, batch size, and element width, so it grows with the amount of text in flight. Model weights are constant no matter how many requests you serve, which means the cache is the term that expands until it exhausts device memory and caps concurrency.
How does MLA’s low-rank latent projection differ from multi-query attention (MQA) and grouped-query attention (GQA)? MQA and GQA reduce how many key/value heads are cached by sharing them across query heads. MLA instead reduces how large the cached representation is through dimensional compression, keeping the full set of query heads active. They act on different axes and can, in principle, be reasoned about independently.
What does MLA cost in return: extra projection compute, quality impact, and interaction with positional encoding such as RoPE? It adds down-projection and up-projection matrix multiplications on every decode step, which is favourable when decoding is bandwidth-bound and less so when it is compute-bound. Rotary position embeddings do not commute cleanly with a learned up-projection, so implementations carry position-bearing components on a decoupled path. Reconstruction is approximate, so task-level quality impact must be measured rather than assumed.
Where does MLA sit in the generative model taxonomy — is it a new architecture family or an efficiency change inside autoregressive transformers? It is an efficiency change inside the autoregressive transformer family, not a new family alongside GANs, diffusion models, or VAEs. It belongs in the same category as quantisation, attention kernel selection, and paged cache management — changes to what a family costs to serve.
When is MLA the right reason to prefer one open-weight model over another, and when is it irrelevant to the decision? It matters when long contexts or high concurrency on a fixed GPU allocation make cost per million output tokens a binding constraint, and two candidates are otherwise close on quality. It is largely irrelevant for short-context or offline batch workloads, and it is never evidence of task capability.
What should a team measure to verify the memory and throughput benefit on their own hardware and context lengths? Measure bytes of KV cache per token per layer from the running serving stack, tokens per second at your target concurrency rather than single-stream latency, and cost per million output tokens. Take all three before and after the change, at the context length you actually serve, on an otherwise identical stack.
Attention-level memory behaviour is one dimension of a larger question — which generative architecture belongs in a given system at all — and we develop that decision space in our taxonomy of generative AI model architectures. The open question worth carrying forward: on your hardware, is decoding bandwidth-bound enough that trading bytes for FLOPs still pays?
When Multi Head Latent Attention is worth it
Multi Head Latent Attention is rarely the hard part — knowing which of its failure modes you can live with is. That answer is workload-specific, and it is worth writing down before you build.
