A matrix multiplication that doesn’t need all those bits

Consider a transformer’s self-attention computation. The query-key dot products, followed by softmax, followed by the value projection. In FP32, each of these operations uses 32 bits per element — full IEEE 754 floating-point precision, with roughly 7 decimal digits of accuracy.

But the softmax output is a probability distribution. Its values fall between 0 and 1, and the model’s downstream behavior depends on the relative ordering and rough magnitudes of these probabilities far more than on their exact values. Whether the softmax assigns 0.0312 or 0.0314 to a particular position almost never changes the model’s prediction. That difference — two parts in ten thousand — disappears in the noise of the subsequent matrix multiplication.

This observation is the foundation of mixed precision: not all operations require equal precision, and the operations that tolerate lower precision are often the most compute-intensive ones.

The numerical tolerance landscape

A neural network is not a uniform numerical system. Different components exhibit different sensitivities to precision changes.

Attention and feedforward layers — the bulk of computation in transformer models — are dominated by matrix multiplications where the outputs are subsequently normalized (by LayerNorm or similar operations). The normalization absorbs small numerical errors by rescaling the outputs. This makes these layers relatively tolerant of reduced-precision arithmetic. BF16 and FP16 work well here because the normalization step cleans up the rounding errors that accumulate during the lower-precision matrix multiply.

Loss computation and gradient accumulation — in training — require higher precision because they deal with small values that grow through summation over many elements. Accumulating thousands of small gradients in FP16 risks overflow or catastrophic cancellation. This is why training frameworks keep a master copy of weights in FP32 and perform gradient reduction in FP32, even when the forward and backward passes run in BF16.

Embedding lookups and final projection layers tend to be more sensitive in some architectures because they operate at the boundaries where small numerical changes can shift which token gets selected or which class gets predicted. These layers sometimes benefit from staying at higher precision even when the rest of the model has been reduced.

The practical consequence is that precision is not a global setting to be applied uniformly. It’s a resource to be allocated selectively — high precision where sensitivity demands it, low precision where tolerance allows it.

Operation sensitivity and typical precision assignment

Operation type	Precision sensitivity	Typical mixed-precision assignment
Matrix multiplications (attention, FFN)	Low — normalization absorbs rounding errors	BF16 or FP8 on tensor cores
Loss computation	High — small values accumulated over many elements	FP32 always
Gradient accumulation	High — overflow and cancellation risk	FP32 master weights + FP32 reduction
Embedding / final projection	Moderate — boundary layers where small changes shift output	Often kept at higher precision
Activation functions, normalizations	Low to moderate — depends on architecture	Framework auto-classification (torch.amp allowlists)

How frameworks implement it

Modern frameworks make mixed precision largely automatic for the common case. PyTorch’s torch.amp (Automatic Mixed Precision) wraps the forward pass in a context that casts operations to lower precision where it’s been determined to be safe, while keeping certain operations — cumulative sums, log operations, loss functions — in FP32.

Under the hood, the decision about which operations run at which precision is based on empirically validated allowlists. NVIDIA’s documentation classifies operations into categories: operations that are safe in FP16/BF16 (most matrix multiplies and convolutions), operations that should remain in FP32 (reductions, normalizations, log-domain math), and operations where either precision is acceptable depending on context.

What AMP does not decide is the part that matters most in production. It picks a cast per operation from a fixed table; it does not know your dataset’s numerical distribution, it does not know whether your custom kernel behaves like the op it resembles, and it will not tell you when an output has drifted. Those judgements stay with the engineer.

This automated approach works well for standard architectures. It becomes less reliable with custom operations, unusual architectures, or numerical edge cases specific to certain datasets. When we’ve evaluated non-standard architectures, we’ve sometimes found that the default allowlists are too aggressive or too conservative — a custom attention variant that needs FP32 for stability, or a normalization layer that works fine in BF16 despite being categorized as FP32-required. (Observed across our evaluation work; not a published failure rate.)

The automation is a useful starting point, not a guarantee. Validation on the target workload remains necessary, just as precision being a design parameter rather than a fixed quality level means the design must be verified for each deployment.

What about loss scaling?

Loss scaling multiplies the loss by a constant before the backward pass so that small gradient magnitudes land inside FP16’s representable range instead of flushing to zero, then divides the gradients back down before the optimizer step. It exists because the backward pass produces the smallest values in the whole computation. An inference-only mixed-precision setup has no backward pass and no gradient accumulation, so there is nothing to rescue — which is why loss scaling is a training concern that quietly disappears from inference discussions, and why teams reading training guidance sometimes look for a knob their serving path does not have.

The performance case

The motivation for mixed precision is straightforward: lower precision means less memory, less bandwidth, and more throughput.

A BF16 matrix multiply uses half the memory bandwidth of FP32 and can execute up to 2× faster on hardware with dedicated BF16 tensor cores (Ampere, Hopper). FP8 on Hopper-generation hardware offers another 2× over BF16 for supported operations.

The choice between formats is itself a tolerance question. BF16 is often preferred over FP16 for mixed precision because it keeps FP32’s 8-bit exponent — the same dynamic range — and trades away mantissa bits instead. That wider range is what prevents the overflow and underflow that FP16’s narrower exponent invites during reduced-precision matrix multiplies, which is why BF16 frequently runs without the loss-scaling machinery FP16 needs to stay stable. FP16 buys more mantissa precision in exchange, but only across a smaller representable range. FP8, in turn, sits a step further down the same ladder: it assumes the operation tolerates not just coarser rounding but a markedly smaller dynamic range than FP16 or BF16, so it is reserved for the matrix multiplies whose outputs get normalized away and is paired with per-tensor scaling to keep values inside its limited range. Each format down the ladder asks more of the numerical tolerance you are exploiting.

Memory savings compound the throughput benefit. A model stored in BF16 uses half the HBM of FP32. This means larger batch sizes fit in memory, which improves GPU utilization. Or it means larger models fit on a single GPU, avoiding the communication overhead of model parallelism.

For inference specifically, where the workload is often memory-bandwidth-bound (reading model weights from HBM for every token), reduced precision directly translates to higher tokens-per-second because each token generation reads half (BF16 vs FP32) or quarter (FP8 vs FP32) the data from memory.

Mixed precision is not quantization

The two get conflated constantly, and the conflation produces wrong expectations about stability. Mixed precision routes different operations to different floating-point formats and keeps the arithmetic floating-point end to end; the error at each step is bounded by that format’s rounding and range. Quantization maps values onto a much smaller discrete set — commonly integers — which changes the representation itself and introduces calibration, scale selection, and outlier handling as separate problems. A team that has validated a BF16/FP32 mixed scheme has learned very little about how the same model behaves in INT8, and vice versa.

Why does mixed precision work without degrading model quality?

The reason mixed precision works reliably in practice — despite reducing numerical precision for most computations — is that the precision reduction is selective, not uniform.

The operations that are most vulnerable to precision errors (gradient accumulation, loss computation, certain normalizations) retain full precision. The operations that generate the vast majority of compute load (matrix multiplications, convolutions) use reduced precision because the magnitude of their rounding errors is small relative to the signal they carry, and subsequent operations (normalization, activation functions) absorb or mask those errors.

This selective strategy means the model’s numerical behavior in mixed precision closely tracks its behavior in full precision. The errors introduced by lower-precision arithmetic are absorbed at every normalization boundary, and the accumulated effect on the final output is typically within the noise floor of other sources of inference variability.

When the strategy fails — when mixed precision produces materially different outputs than full precision — it’s almost always because a specific operation was incorrectly classified as precision-tolerant when it wasn’t. This is diagnosable (compare layer-by-layer outputs between mixed and full precision) and fixable (keep that layer at higher precision while leaving the rest at lower precision). The fix is surgical, not a retreat to full FP32.

One boundary is worth naming plainly. A tolerance you measured is a property of the whole AI Executor — device, backend, driver, framework, and runtime together — not of the silicon underneath it. The same model at the same nominal precision can round differently under a different cuDNN version or a different graph compiler, so a stability result carried across stacks is a hypothesis, not evidence.

Mixed precision isn’t a hack or a shortcut. It’s an engineering exploitation of a real property of neural networks: uneven numerical sensitivity. Understanding where the tolerance lives — and validating that the framework’s assumptions match your workload’s reality, as discussed in how hardware constraints shape precision choices — is what makes it work reliably. LynxBenchAI deliberately does the opposite of a mixed design: it holds one precision fixed per test and reports each level separately, which is what leaves clean per-precision endpoints on a real machine for a mixed scheme to be argued against rather than an average that hides where the tolerance lived.

So the open question for any team is not whether mixed precision works in general — it does — but which of your layers you have actually measured, and on which executor you measured them.

Frequently Asked Questions

Why does mixed precision inference work in practice, given that not all operations tolerate the same numerical loss?

It works because the precision reduction is selective, not uniform. Operations vulnerable to precision errors — loss computation, gradient accumulation, certain normalizations — retain full precision, while the bulk of compute (matrix multiplications, convolutions) runs at lower precision where their rounding errors are small relative to the signal and get absorbed by subsequent normalization steps. The model’s numerical behavior in mixed precision closely tracks its behavior in full precision because the errors never accumulate past a normalization boundary.

Where in a typical model is higher precision usually retained, and why?

Higher precision is retained in loss computation and gradient accumulation (small values summed over many elements, where FP16 risks overflow or catastrophic cancellation), in reductions and log-domain operations, and often in embedding lookups and the final projection layer. Those boundary layers can shift which token or class gets selected with small numerical changes, so the cost of keeping them at higher precision is paid willingly.

Why is mixed precision not universally stable, even when a framework supports it automatically?

Framework allowlists like torch.amp’s are empirically validated for standard architectures, but they can be too aggressive or too conservative for custom operations, unusual architectures, or edge-case datasets. A custom attention variant may need FP32 for stability; a normalization layer may run fine in BF16 despite being classified otherwise. The automation is a useful starting point, not a guarantee — validation on the target workload remains necessary.

How is mixed precision different from quantization in what it actually does to the numerics?

Mixed precision routes different operations to different floating-point formats (FP32, BF16, FP16, FP8) based on their tolerance, keeping the underlying arithmetic floating-point throughout, with error bounded by each format’s rounding and representable range. Quantization maps values onto a much smaller discrete set and brings calibration, scale selection, and outlier handling with it. Validating one tells you very little about the other, and conflating them produces wrong expectations about where each technique is safe.

How does FP8 fit into the mixed precision picture, and what additional tolerance assumptions does it require compared with FP16/BF16?

FP8 sits a step further down the same precision ladder, giving another throughput gain over BF16 for supported operations on Hopper-generation hardware. It assumes the operation tolerates not just coarser rounding but a markedly smaller dynamic range than FP16 or BF16, so it is reserved for the matrix multiplies whose outputs get normalized away. To keep values inside its limited range, FP8 is paired with per-tensor scaling — each format down the ladder asks more of the numerical tolerance you are exploiting.

What does automatic mixed precision (AMP) in a framework actually decide on your behalf, and which decisions still have to be made by hand?

AMP decides the cast for each operation it recognises, using empirically validated allowlists: matrix multiplies and convolutions go to BF16 or FP16, reductions and log-domain math stay in FP32. What it does not decide is whether those defaults hold for your architecture, your custom kernels, or your data’s numerical distribution — and it will not signal drift when they don’t. Choosing the format, keeping specific layers at higher precision, and running the layer-by-layer comparison against a full-precision reference remain manual work.

What is loss scaling, and why does an inference-only mixed precision setup usually not need it while a training setup does?

Loss scaling multiplies the loss by a constant before the backward pass so small gradient magnitudes fall inside FP16’s representable range instead of flushing to zero, then rescales the gradients before the optimizer step. It exists because gradients are the smallest values in the computation, and FP16’s narrow exponent cannot hold them. Inference has no backward pass and no gradient accumulation, so the failure mode loss scaling protects against does not arise — which is also why BF16, with FP32’s exponent range, often removes the need even in training.