A model that does not fit does not have a throughput number.

That sounds trivial until you watch a team spend two weeks tuning attention kernels for a deployment that was never compute-bound in the first place — it was one long request away from an out-of-memory error, and every “optimisation” they shipped was really a capacity workaround wearing a performance costume. Precision is usually discussed as a speed knob or an accuracy risk. Before it is either of those, it is the knob that decides how much of your workload is resident on the device at once.

That is a different question from “is FP8 faster than FP16”, and it has a different answer shape. Speed questions resolve into a ratio. Capacity questions resolve into a binary — it runs or it doesn’t — and only after that binary flips do the ratios start to matter.

Three footprints, three behaviours

The habit that causes the most estimation error is treating “model memory” as one number. Device memory during inference holds at least three distinct populations, and a precision change moves each of them for a different reason.

Weights are static. Their footprint is parameter count times bytes per parameter, plus whatever the quantisation format carries as overhead — scales, zero-points, block metadata. This is the part everyone estimates, and it is the part that behaves most predictably: halving the bit width roughly halves the weight footprint, minus the metadata tax. Block-scaled 4-bit formats carry a scale per block of weights, so the effective bits-per-weight is always somewhat above the nominal number. A format advertised as 4-bit with a per-32-element scale group is closer to 4.5 effective bits once you count the scales.

Activations are transient and shaped by batch size, sequence length, and the graph’s fusion behaviour rather than by parameter count. Whether they shrink with precision depends on where the runtime materialises intermediates. A fused attention implementation such as FlashAttention never materialises the full attention matrix, so the “activation savings” you would predict from a naive layer-by-layer accounting simply are not there to collect — the memory was already not being spent. Meanwhile, if the runtime keeps a high-precision accumulator or dequantises a tile into a scratch buffer, part of the activation path stays at the wider format regardless of the storage precision you chose.

KV cache is the population that most often turns out to be the binding constraint in long-context serving, and it is the one that scales with concurrency and context length rather than with model size. It grows linearly in sequence length, linearly in batch size, and linearly in the number of key-value heads. On a model using grouped-query attention, the head count in that product is already reduced by architecture; on an older multi-head model it is not. This is why two models with identical parameter counts can have KV footprints that differ by several times at the same context length.

Quick answer — where does the memory actually go?

• Weights: static, scales cleanly with bit width, plus quantisation metadata (scales, zero-points) that does not.
• Activations: depend on batch, sequence length, and kernel fusion — not on parameter count. Savings are conditional on whether the intermediate was ever materialised.
• KV cache: scales with context × batch × KV heads × layers. Dominant in long-context serving; often the reason a workload does not fit.
• Fixed overheads: CUDA context, allocator fragmentation, framework workspace, communication buffers. These do not shrink when you change precision.

Why parameter count × bytes per parameter is not a capacity estimate

The most common sizing method — multiply parameters by bytes, add “some headroom” — is wrong in a specific and predictable direction. It underestimates, because it accounts only for the population that shrinks cleanly and ignores every population that does not.

What a usable capacity estimate has to include beyond weights:

1. KV cache at your worst realistic concurrency and context, not your average. Capacity planning against averages produces a system that OOMs under exactly the traffic you built it for.
2. Quantisation metadata, which is proportional to the number of blocks, not to the bit width. Go from 8-bit to 4-bit with the same block size and the metadata cost stays where it was while the payload halves — so it becomes a larger fraction of a smaller total.
3. Runtime and allocator overhead. A CUDA context costs device memory before your model loads. Caching allocators fragment. Multi-GPU setups hold NCCL communication buffers. None of this responds to precision.
4. Peak, not steady state. Load-time dequantisation, graph capture, and the first forward pass through an unfused path can all transiently exceed the resident footprint you measured afterwards.

Point 3 deserves emphasis because it is where “some memory savings scale with bit width and some do not” becomes operationally sharp. If roughly a fifth of your occupied memory is fixed overhead — and on a 16 GB consumer card with a modest model it can easily be that or more — then moving from FP16 to INT8 does not free half your memory. It frees half of the part that was elastic. We see teams plan against the first number and discover the second one at deployment.

What you can buy with freed headroom — pick one

Suppose the estimate works out and you have genuinely freed several gigabytes. There are three obvious things to spend it on, and they compete for the same pool.

Spend the headroom on	What you gain	What you give up	When it’s the right call
Larger batch	Higher aggregate throughput on a memory-bound decode phase; better arithmetic intensity per weight read	Per-request latency rises; tail latency rises faster than mean	Offline or high-QPS serving where throughput is the SLO
Longer context	Larger usable window per request; fewer retrieval workarounds	KV cache growth is linear in context and consumes headroom fast; attention cost grows faster than linearly	Document-scale workloads where truncation is a correctness problem
Second resident model	Avoids model-swap latency; enables draft-model speculative decoding or a reranker alongside the generator	Both models now share bandwidth and SM time, not just capacity	Pipelines where swap cost dominates, or speculative decoding pays for itself

These are alternatives. A precision change that frees 8 GB does not let you simultaneously quadruple batch, double context, and co-resident a second model — it lets you do roughly one of those things well. The framing error we encounter most often in capacity reviews is a plan that quietly assumes all three (observed across engagements; not a benchmarked distribution). Write down which one you are buying before you change the format.

Why do memory-bound and compute-bound workloads respond differently to the same precision change?

Because they are limited by different resources, and precision touches both — unevenly.

In the decode phase of autoregressive generation at low batch size, the device spends most of its time reading weights and KV cache out of HBM to do a small amount of arithmetic per byte. This is memory-bandwidth-bound. Halving the storage precision of the weights roughly halves the bytes that have to cross the memory bus per token, and the speed-up can track that reduction reasonably closely — right up until the dequantisation work on the compute side becomes the new limit, at which point the curve flattens.

In the prefill phase, or in decode at high batch, the same weights are read once and reused across many tokens. Arithmetic intensity is high, the workload is compute-bound, and the benefit of lower precision comes from whichever tensor-core path the format unlocks — not from the reduced traffic. If the hardware has no native kernel for the format you chose, you can shrink the footprint and lose speed, because the runtime now dequantises on every use.

Which means the same precision change can produce a near-linear speed-up, a flat result, or a regression, depending entirely on where the workload sat on the roofline before you touched it. Memory savings do not translate proportionally into speed-ups. They are two different effects that happen to share a cause. Our companion discussion of how precision trades against model accuracy covers the other half of this decision; here the concern is strictly what fits. If you are still deciding whether precision belongs in your test matrix at all, precision as a benchmark variable treats the measurement design question this article assumes.

KV cache quantisation is not weight quantisation with a different target

Both reduce a footprint. The risk profiles are not comparable, and treating them as one decision is how teams end up with a serving stack that degrades in ways their offline evaluation never catches.

Weight quantisation is a one-time, offline, inspectable transformation. You quantise once, you can measure perplexity or task accuracy on the quantised artifact, and the error it introduces is fixed for the lifetime of that artifact. Calibration data matters, outlier channels matter, and the failure modes are well-mapped by now.

KV cache quantisation is online and cumulative. Every token’s keys and values are quantised as they are written, and every subsequent token attends over that quantised history. Error does not sit still — it participates in the next step’s attention computation. Two properties follow. First, degradation tends to grow with sequence length, so a short-prompt evaluation can look clean while a long-context workload drifts. Second, keys and values behave differently: attention scores are a dot product against keys, so key quantisation error propagates through a softmax that can amplify it, while value error enters as a weighted average that tends to be more forgiving. Implementations in vLLM and llama.cpp expose key and value bit widths separately for exactly this reason, and the asymmetric configurations exist because the symmetric assumption is wrong.

The practical consequence: if you quantise the KV cache, your evaluation has to run at the context lengths and concurrency you actually serve. A benchmark that measures a 512-token prompt tells you nothing about what happens at 32k.

A capacity diagnostic before you touch the format

Run this before choosing a precision, not after the OOM.

• What fraction of occupied memory is weights? If it’s under half at your target context and batch, weight quantisation is not your main lever — the KV cache is.
• What is the KV footprint at p99 context × p99 concurrency? Not the mean. Not the design target. The realistic worst case you will actually see.
• Does the model use grouped-query or multi-query attention? If yes, the KV term is already reduced by architecture and further quantisation buys less than the arithmetic suggests.
• How much is fixed overhead? Measure occupied device memory with the runtime initialised and no model loaded. That number does not respond to precision.
• Is the target format natively supported by the kernels you will run? A format without a native path costs dequantisation compute for its capacity win. Check the runtime, not the hardware datasheet.
• Where does the workload sit on the roofline today? Memory-bound decode and compute-bound prefill respond differently; a mixed serving pattern responds as a weighted mixture you have to actually measure.
• Are you spending the freed headroom on batch, context, or a second model? Name one. If the answer is “all of them”, the plan is over-committed.

Six of these seven can be answered from a profiler and a spec sheet in an afternoon. The seventh is a product decision.

Measuring capacity effects instead of estimating them

Here is where the estimate has to give way to measurement. Per-precision numbers taken at a saturated workload size are the only place the capacity effect becomes observable rather than modelled, because saturation is the condition under which the memory system is actually the constraint. A run at batch size 1 with a 128-token prompt tells you about kernel launch overhead, not about capacity. Finding that saturation point is its own discipline — see our treatment of scale-aware saturation in benchmark design for how the workload size is chosen rather than guessed.

There is a reporting consequence too. If you aggregate results by averaging throughput across configurations, the capacity effect disappears from the summary — a configuration that barely fits and a configuration with generous headroom both contribute one number each. Weighting throughput by the memory a run actually moves keeps the capacity dimension visible in the aggregate, because a run that moved four gigabytes per token step and a run that moved one are no longer treated as equivalent evidence. This is part of why sustained measurement at realistic scale produces different conclusions than transient peak figures: the memory system behaves differently when it is full.

The broader point about what unit you are even measuring — hardware plus the software stack that drives it, not hardware alone — is developed in our treatment of the AI Executor as the unit of performance. Precision formats are a property of that pair, not of the silicon. A format your GPU supports and your runtime does not is a format you do not have.

For teams whose next step is executing a capacity reduction on a live system rather than measuring one, the applied side of this work — locating whether the real bottleneck is memory or compute before spending engineering time on either — sits with TechnoLynx’s inference cost audit work. Fitting on a device and performing well on it are different achievements, and the second one is where that work starts. How the benchmark itself keeps the capacity dimension visible rather than averaging it away is set out in the LynxBenchAI methodology.

FAQ

How does numerical precision change the memory footprint of weights, activations, and the KV cache separately?

Weights scale close to linearly with bit width, minus quantisation metadata such as per-block scales that stays fixed regardless of the payload width. Activations depend on batch size, sequence length, and whether the kernel materialises intermediates at all — a fused attention path has no full attention matrix to shrink. KV cache scales with context length, batch, KV head count, and layer count, which is why it can dominate the total in long-context serving even for a small model.

Why does precision often decide whether a workload fits on a device before it decides how fast that workload runs?

Capacity is a binary and throughput is a ratio. A configuration that exceeds device memory produces no performance number at all, so the fit question is logically prior. Once the workload is resident, precision’s effect on speed depends on where it sits on the roofline — which is a separate question with a separate answer.

What can freed memory headroom be spent on, and why are batch, context, and a second resident model alternatives rather than a combined gain?

Larger batch raises aggregate throughput at the cost of tail latency; longer context consumes headroom linearly through the KV cache; a second resident model removes swap latency but shares bandwidth and compute. All three draw from the same freed pool, so a precision change that frees a given amount of memory lets you do roughly one of them well. Naming the intended spend before changing the format prevents an over-committed capacity plan.

At what point does cutting precision stop making a decode workload faster?

Low-batch decode is bandwidth-bound: fewer bytes crossing HBM per token translates fairly directly into speed, until dequantisation compute becomes the new limit — that inflection is the ceiling. Prefill and high-batch decode never had the same starting point, because they are compute-bound, where the benefit comes from whichever tensor-core path the format unlocks rather than from reduced traffic. With no native kernel for the chosen format, the same change can shrink footprint while making the workload slower.

How does quantising the KV cache differ in risk from quantising the weights?

Weight quantisation is offline, one-time, and inspectable — you can evaluate the resulting artifact directly and the error is fixed. KV cache quantisation happens online and accumulates: each token attends over an already-quantised history, so degradation tends to grow with sequence length and short-prompt evaluations can miss it entirely. Keys and values also behave differently under quantisation, which is why serving runtimes expose their bit widths separately.

Which memory savings scale with bit width, and which stay fixed regardless of the format chosen?

Weight payload and KV cache payload scale with bit width. Quantisation metadata scales with the number of blocks rather than the width, so it becomes a larger fraction of a smaller total as you go lower. Runtime context, allocator fragmentation, framework workspace, and communication buffers do not respond to precision at all — which is why halving the format does not halve occupied memory.

What should a capacity estimate include beyond parameter count multiplied by bytes per parameter?

KV cache at worst-case concurrency and context rather than average, quantisation metadata, fixed runtime and allocator overhead, and transient peaks during load-time dequantisation or the first forward pass. Parameter count times bytes per parameter covers only the population that shrinks cleanly, which is why that method underestimates in a consistent direction.

The question the fit answer does not settle

The interesting failures we see are not the ones where a model refuses to load. Those announce themselves. The expensive ones are the deployments that fit — barely — and then spend their lives one long prompt away from the edge, with fragmentation slowly closing the gap and a capacity plan nobody wrote down.

So the question worth asking of any precision decision is not “does it fit” but “how much of the headroom did I intend to spend, and on what”. If you cannot answer the second half, the format change was a reprieve rather than a plan. Measure the footprint at the concurrency and context you actually serve, decide in advance what the freed memory is for, and treat fitting on the device as the beginning of the performance question rather than the end of it.