A matrix multiplication that doesn’t need all those bits

Consider a transformer’s self-attention computation. The query-key dot products, followed by softmax, followed by the value projection. In FP32, each of these operations uses 32 bits per element — full IEEE 754 floating-point precision, with roughly 7 decimal digits of accuracy.

But the softmax output is a probability distribution. Its values fall between 0 and 1, and the model’s downstream behavior depends on the relative ordering and rough magnitudes of these probabilities far more than on their exact values. Whether the softmax assigns 0.0312 or 0.0314 to a particular position almost never changes the model’s prediction. That difference — two parts in ten thousand — disappears in the noise of the subsequent matrix multiplication.

This observation is the foundation of mixed precision: not all operations require equal precision, and the operations that tolerate lower precision are often the most compute-intensive ones.

The numerical tolerance landscape

A neural network is not a uniform numerical system. Different components exhibit different sensitivities to precision changes:

Attention and feedforward layers — the bulk of computation in transformer models — are dominated by matrix multiplications where the outputs are subsequently normalized (by LayerNorm or similar operations). The normalization absorbs small numerical errors by rescaling the outputs. This makes these layers relatively tolerant of reduced-precision arithmetic. BF16 and FP16 work well here because the normalization step cleans up the rounding errors that accumulate during the lower-precision matrix multiply.

Loss computation and gradient accumulation — in training — require higher precision because they deal with small values that grow through summation over many elements. Accumulating thousands of small gradients in FP16 risks overflow or catastrophic cancellation. This is why training frameworks keep a master copy of weights in FP32 and perform gradient reduction in FP32, even when the forward and backward passes run in BF16.

Embedding lookups and final projection layers tend to be more sensitive in some architectures because they operate at the boundaries where small numerical changes can shift which token gets selected or which class gets predicted. These layers sometimes benefit from staying at higher precision even when the rest of the model has been reduced.

The practical consequence is that precision is not a global setting to be applied uniformly. It’s a resource to be allocated selectively — high precision where sensitivity demands it, low precision where tolerance allows it.

Operation sensitivity and typical precision assignment

Operation type	Precision sensitivity	Typical mixed-precision assignment
Matrix multiplications (attention, FFN)	Low — normalization absorbs rounding errors	BF16 or FP8 on tensor cores
Loss computation	High — small values accumulated over many elements	FP32 always
Gradient accumulation	High — overflow and cancellation risk	FP32 master weights + FP32 reduction
Embedding / final projection	Moderate — boundary layers where small changes shift output	Often kept at higher precision
Activation functions, normalizations	Low to moderate — depends on architecture	Framework auto-classification (torch.amp allowlists)

How frameworks implement it

Modern frameworks make mixed precision largely automatic for the common case. PyTorch’s torch.amp (Automatic Mixed Precision) wraps the forward pass in a context that casts operations to lower precision where it’s been determined to be safe, while keeping certain operations — cumulative sums, log operations, loss functions — in FP32.

Under the hood, the decision about which operations run at which precision is based on empirically validated allowlists. NVIDIA’s documentation classifies operations into categories: operations that are safe in FP16/BF16 (most matrix multiplies and convolutions), operations that should remain in FP32 (reductions, normalizations, log-domain math), and operations where either precision is acceptable depending on context.

This automated approach works well for standard architectures. It becomes less reliable with custom operations, unusual architectures, or numerical edge cases specific to certain datasets. When we’ve evaluated non-standard architectures, we’ve sometimes found that the default allowlists are too aggressive or too conservative — a custom attention variant that needs FP32 for stability, or a normalization layer that works fine in BF16 despite being categorized as FP32-required.

The automation is a useful starting point, not a guarantee. Validation on the target workload remains necessary, just as precision being a design parameter rather than a fixed quality level means the design must be verified for each deployment.

The performance case

The motivation for mixed precision is straightforward: lower precision means less memory, less bandwidth, and more throughput.

A BF16 matrix multiply uses half the memory bandwidth of FP32 and can execute up to 2× faster on hardware with dedicated BF16 tensor cores (Ampere, Hopper). FP8 on Hopper-generation hardware offers another 2× over BF16 for supported operations. The improvement comes from both reduced data movement (less bandwidth consumed per operation) and dedicated hardware units (tensor cores with native lower-precision arithmetic).

Memory savings compound the throughput benefit. A model stored in BF16 uses half the HBM of FP32. This means larger batch sizes fit in memory, which improves GPU utilization. Or it means larger models fit on a single GPU, avoiding the communication overhead of model parallelism.

For inference specifically, where the workload is often memory-bandwidth-bound (reading model weights from HBM for every token), reduced precision directly translates to higher tokens-per-second because each token generation reads half (BF16 vs FP32) or quarter (FP8 vs FP32) the data from memory.

Why does mixed precision work without degrading model quality?

The reason mixed precision works reliably in practice — despite reducing numerical precision for most computations — is that the precision reduction is selective, not uniform.

The operations that are most vulnerable to precision errors (gradient accumulation, loss computation, certain normalizations) retain full precision. The operations that generate the vast majority of compute load (matrix multiplications, convolutions) use reduced precision because the magnitude of their rounding errors is small relative to the signal they carry, and subsequent operations (normalization, activation functions) absorb or mask those errors.

This selective strategy means the model’s numerical behavior in mixed precision closely tracks its behavior in full precision. The errors introduced by lower-precision arithmetic are absorbed at every normalization boundary, and the accumulated effect on the final output is typically within the noise floor of other sources of inference variability.

When the strategy fails — when mixed precision produces materially different outputs than full precision — it’s almost always because a specific operation was incorrectly classified as precision-tolerant when it wasn’t. This is diagnosable (compare layer-by-layer outputs between mixed and full precision) and fixable (keep that layer at higher precision while leaving the rest at lower precision). The fix is surgical, not a retreat to full FP32.

Mixed precision isn’t a hack or a shortcut. It’s an engineering exploitation of a real property of neural networks: uneven numerical sensitivity. Understanding where the tolerance lives — and validating that the framework’s assumptions match your workload’s reality, as discussed in how hardware constraints shape precision choices — is what makes it work reliably.

Related deep-dives

• Half-precision floating-point: why FP16 needs mixed precision to be stable — the FP16 reference anchor and why mixed-precision schemes exist around it.

LynxBenchAI measures performance at each precision regime separately — so mixed-precision and single-precision results are reported under declared conditions rather than merged into an average that obscures where the tolerance actually lived. It is a benchmarking methodology for AI hardware — measuring sustained performance across the complete hardware-and-software stack, reported per precision, with bounded optimisation.

Frequently Asked Questions

Why does mixed precision inference work in practice, given that not all operations tolerate the same numerical loss?

It works because the precision reduction is selective, not uniform. Operations vulnerable to precision errors — loss computation, gradient accumulation, certain normalizations — retain full precision, while the bulk of compute (matrix multiplications, convolutions) runs at lower precision where their rounding errors are small relative to the signal and get absorbed by subsequent normalization steps. The model’s numerical behavior in mixed precision closely tracks its behavior in full precision because the errors never accumulate past a normalization boundary.

How does mixed precision exploit uneven numerical sensitivity across the operations of a model?

A neural network is not a uniform numerical system. Attention and feedforward layers dominate compute and tolerate BF16 or FP8 because their outputs are normalized downstream, which rescales away the rounding noise. Sensitive operations — reductions, log-domain math, gradient accumulation — are kept at FP32. Precision becomes a resource allocated selectively rather than a global setting, which is why mixed precision exists as a category at all.

Where in a typical model is higher precision usually retained, and why?

Higher precision is retained in loss computation and gradient accumulation (small values summed over many elements, where FP16 risks overflow or catastrophic cancellation), in reductions and log-domain operations, and often in embedding lookups and the final projection layer. Those boundary layers can shift which token or class gets selected with small numerical changes, so the cost of keeping them at higher precision is paid willingly.

Why is mixed precision not universally stable, even when a framework supports it automatically?

Framework allowlists like torch.amp’s are empirically validated for standard architectures, but they can be too aggressive or too conservative for custom operations, unusual architectures, or edge-case datasets. A custom attention variant may need FP32 for stability; a normalization layer may run fine in BF16 despite being classified otherwise. The automation is a useful starting point, not a guarantee — validation on the target workload remains necessary.

How is mixed precision different from quantization in what it actually does to the numerics?

Mixed precision routes different operations to different floating-point formats (FP32, BF16, FP16, FP8) based on their tolerance, keeping the underlying arithmetic floating-point throughout. The numerical behavior closely tracks full precision because the errors are bounded by the format’s representable range at each step. Quantization is a different mechanism that maps values to a much smaller discrete set, and conflating the two leads to incorrect expectations about stability and where each technique applies.

What residual stability risks should a team still evaluate when adopting mixed precision in production?

The main risk is an operation being incorrectly classified as precision-tolerant when it isn’t — particularly in custom layers, non-standard attention variants, or workloads with unusual numerical distributions. The diagnostic is to compare layer-by-layer outputs between mixed and full precision, identify where they diverge materially, and keep the offending layer at higher precision while leaving the rest reduced. The fix is surgical rather than a retreat to FP32, but the validation step is non-negotiable.