In the BF16 vs FP16 choice, BF16 is the sensible default for training on an Ampere-class or newer NVIDIA GPU, because it keeps the dynamic range of FP32 and removes the need for loss scaling. Three cases favour FP16 instead. The first is pre-Ampere hardware such as the V100 or T4, which has no BF16 Tensor Cores. The second is RL fine-tuning, where training and inference engines have to agree numerically. The third is serving, when outputs should track an FP32 reference as closely as possible. Both formats use 16 bits in total. What differs is how those bits are split between range and precision, and that split decides which failure you are more likely to hit.
BF16 vs FP16 bit layouts: range against precision
Exponent bits set the range a format covers (how large or small a value can get), and mantissa bits set the precision (how finely nearby values can be told apart). FP16, the IEEE 754 half-precision format, gives 5 bits to the exponent and 10 to the mantissa. BF16 keeps FP32’s 8-bit exponent and cuts the mantissa to 7 bits, so it gives up nothing in range compared with FP32.
| Exponent / mantissa bits | 5 / 10 | 8 / 7 |
| Smallest positive normal | ≈6.1×10⁻⁵ | ≈1.2×10⁻³⁸ |
| Largest representable value | 65,504 | ≈3.4×10³⁸ |
| Next value above 1.0 | ≈1.000977 | ≈1.007812 |
| Machine epsilon | 2⁻¹¹ ≈ 4.9×10⁻⁴ | 2⁻⁸ ≈ 3.9×10⁻³ |
The last two rows show the precision gap. With 10 mantissa bits against 7, FP16 resolves 2¹⁰ values per binade where BF16 resolves 2⁷, which makes it eight times finer. The first three rows show the range gap. FP16 overflows past 65,504, and an overflow produces Inf, which can then spread into NaN and wreck model accuracy. At the small end, gradients below FP16’s minimum representable value round to zero. Loss scaling fixes this by multiplying the loss by a large factor before backpropagation, which shifts small gradients back into representable range.
Google built BF16 on the view that neural networks are far more sensitive to exponent size than to mantissa width. One detail to keep in mind: BF16 flushes denormals to zero, while FP32 keeps them. For a wider survey of what each format gives up, see our guide to floating-point formats in AI.
In Kalamkar et al. (2019), BF16 training matched FP32 results across vision, speech, language, generative and recommendation workloads in the same number of iterations, with no hyperparameter changes, while FP16 needed a tuned loss-scaling hyperparameter. A survey of memory-efficient transformer training notes that the BLOOM-176B training report suggests BF16 as the better option for avoiding numerical issues. A 2025 survey of low-precision LLM training concludes that BF16 has gradually replaced FP16 as the default 16-bit training format. FP16 training still works with the recipe it was born with: FP32 master weights, loss scaling and FP32 accumulation.
Is BF16 or FP16 faster, and does a T4 run BF16 natively?
On the A100 and H100, NVIDIA rates neither format faster than the other. NVIDIA’s Ampere documentation says BF16/FP32 mixed-precision Tensor Core operations run at the same rate as FP16/FP32, with both rated at 312 dense TFLOPS on the A100. NVIDIA rates H100 FP16 and BF16 Tensor Core throughput as identical too. Both 16-bit formats sit well above the alternatives on the A100: NVIDIA quotes up to 2x the throughput of TF32 and up to 16x that of FP32.
Equal peak rates do not guarantee equal measured speed on every chip. In a 2026 study of cross-precision inference, FP16 ran Llama-3.2-3B on MATH-500 about 24% faster than BF16 on an A10G (Ampere, SM 86). The authors suggest a faster FP16 FMA path on that microarchitecture as the likely cause. Treat it as a reason to measure, not as a general rule.
The T4 answer is short: it has no native BF16. Turing (SM 75) predates native BF16 support, which starts with Ampere (SM 80). A 2025 hardware survey table shows BF16 Tensor Core support arriving with the A100 and FP16 Tensor Cores with Volta. When a pipeline asks for BF16 on a T4, one 2026 guide says it falls back to FP32, and another practitioner source describes slower emulated BF16 instead. Whichever happens, the Tensor Core advantage is gone. On T4-class hardware, FP16 is the 16-bit format with Tensor Core support.
Why mixed precision can come out slower than FP32 or TF32 in PyTorch
When mixed precision brings no speed-up, or a slowdown, the cause can be one of a few mechanisms rather than the format itself. The first is a dtype the hardware cannot accelerate, such as BF16 on Turing. The second is casting overhead between precisions, which later mixed-precision work addressed with graph-level placement of operations. The third is latency-limited operations: the original mixed-precision paper notes that speed-ups shrink when operations are limited by latency rather than by memory or arithmetic bandwidth. FP8 in particular needs large matrices to pay off. On H800 and RTX 4090, FP8 trailed FP16 at small matrix sizes because of conversion overhead.
Our position is that the precision choice should follow profiling of the specific workload, not a rule of thumb. In practice that means checking a profiler such as Nsight before assuming any throughput gain from a format.
Running a BF16-pretrained model in FP16: where the overflow comes from
One risk is overflow. Unsloth reported that Gemma 3 activations, from the 1B to the 27B model, exceed FP16’s maximum of 65,504. With float16 mixed precision, gradients and activations turned into infinity on T4, RTX 20-series and V100 GPUs, which only have float16 Tensor Cores. Unsloth says the problem does not occur on GPUs with BF16 Tensor Cores. The 2026 inference study notes that when BF16-trained weights cast to FP16 produce NaN or extreme intermediate activations, body-layer error dominates over the lm_head.
Serving stacks already lean towards matching the training format. vLLM’s --dtype auto setting uses BF16 for BF16 models and FP16 for FP32 and FP16 models. Unsloth’s workaround on FP16-only GPUs keeps intermediate activations in BF16 (or FP32, at twice the memory), runs matrix multiplies in FP16 on the Tensor Cores with manual up- and down-casting, and upcasts non-matmul operations such as layernorms to FP32. NVIDIA’s TensorRT guidance follows the same principle: keep precision-sensitive calculations in FP32 and put linear operations in reduced precision, since Tensor Cores accelerate them.
Even without overflow, BF16 and FP16 copies of the same weights are not identical. Storing the same FP32 weights in BF16 versus FP16, both computed in FP32, still gave only 0.88 exact agreement, a gap the study calls irreducible because it comes from the stored weights.
When FP16’s extra mantissa bits matter more than BF16’s range
Range protects you from overflow. Precision protects you from rounding error that compounds. The evidence points to three places where compounding rounding error is the bigger risk.
RL fine-tuning, where two engines must agree
The first case is reinforcement learning fine-tuning of LLMs. RL frameworks often run training and rollout on different engines or kernels. Even with both set to BF16, differences in CUDA kernels and parallel strategies produce different rounding errors, and these accumulate over autoregressive sampling until the training and inference policies diverge. A 2025 arXiv paper, Defeating the Training-Inference Mismatch via FP16, found that switching both sides to FP16 virtually eliminated the mismatch. Under BF16, by contrast, sequence-level mismatch grew exponentially larger with response length. In one configuration, BF16 LoRA training collapsed after roughly 600 steps while FP16 stayed stable. Keeping BF16 training and moving inference to FP32 did make runs stable, but FP32 inference was nearly three times slower than FP16 or BF16.
The result held across several algorithms (GRPO, GSPO, PG and others), model families, LoRA, dense 14B and MoE models, and two frameworks, VeRL and Oat. Its findings concern RL fine-tuning specifically. For pretraining, the same authors credit BF16’s wide range with stability. If your team runs reinforcement learning on agents with separate rollout and training stacks, this paper is a documented reason to run an FP16 trial.
Serving, when outputs should match an FP32 reference
For inference, the 2026 study Greedy Decoding Is Not Precision-Invariant found that the same model, prompt and greedy decoding produce different outputs in BF16 and FP16 on the same hardware. Between 49% and 100% of prompts diverged across six models of 1.1B to 7B parameters. Aggregate task accuracy hides this. On Qwen2.5-3B with GSM8K, BF16 and FP16 scored 27% and 28%, yet 19% of prompts flipped correctness between the two formats. Where the serving format is a free choice, the authors find FP16 the closer single-format proxy for FP32. They also report that reasoning-tuned models show 10 to 15 points lower exact agreement than instruction-tuned models of the same size.
For 7B-scale BF16-saturated models, the authors’ first reproducibility measure is to pin the serving precision. For low-batch greedy decoding, recomputing the lm_head in FP32 only when the top-two logit margin is small raised exact agreement by 22 to 36 points on an A10G at under 4% latency overhead. That fix only works at batch size 4 or below. Our view is that the serving dtype belongs with the other pinned settings in an MLOps deployment pipeline, versioned next to the model, so that a framework default changing cannot quietly change what users see.
Long-context attention with RoPE
BF16’s low precision also has a documented cost in long-context training. Researchers found that RoPE in BF16 drifts from its intended relative positional encoding, that the error grows with context length, that the first token contributes most of it, and that the effect disappears in FP32. Applying the rotation in FP32 does not fully avoid the problem, because FlashAttention2 still requires the query-key inner product in BF16.
How TF32 and FP8 sit next to the two 16-bit formats
TF32 sits apart from BF16 and FP16 as a mode for single-precision workloads. It is a Tensor Core compute mode with 8 exponent bits and 10 mantissa bits, so it has FP32’s range and FP16’s precision. TensorRT enables it by default. NVIDIA rates TF32 on the A100 at 156 dense TFLOPS, half the FP16 and BF16 rate. NVIDIA also describes 16-bit mixed precision as the faster training option, since it uses 2x faster Tensor Cores and reads half the bytes. TF32 accelerates single-precision workloads as they are. FP16 or BF16 mixed precision buys up to a further 2x.
FP8 repeats the range-versus-precision trade at 8 bits. Hopper’s two variants are E4M3, with less range and more precision, and E5M2, with more range and less precision. One documented training recipe uses E4M3 in the forward pass and E5M2 in the backward pass. NVIDIA says FP8 halves storage and doubles throughput compared with FP16 or BF16. Independent measurements on H800 and RTX 4090 were less tidy: FP8 trailed FP16 at small matrix sizes and reached almost twice FP16 throughput at N=16384. On a full transformer layer, FP8 beat FP16 above hidden size 4096 but did not reach double. FP8 needs SM 89 or newer, and its per-logit perturbation is 2 to 5 times that of FP16.
BF16 or FP16 by task and GPU generation
| Situation | Choice | Main reason |
|---|---|---|
| Pretraining or general fine-tuning on Ampere, Hopper or newer | BF16 | FP32 range, no loss scaling, same Tensor Core rate as FP16 |
| Training on V100, T4 or RTX 20-series | FP16 with loss scaling and FP32 master weights | No BF16 Tensor Cores before Ampere |
| RL fine-tuning with separate rollout and training engines | FP16 on both sides, verified on your stack | BF16 rounding drives training-inference mismatch |
| Serving a BF16-pretrained model on Ampere or newer | BF16 by default | Matches training format, avoids FP16 overflow |
| Serving where outputs must track FP32 | FP16, if activations stay below 65,504 | FP16 is the closer single-format proxy for FP32 |
| Serving a BF16-pretrained model on a T4 | FP16 matmuls, FP32 norms, overflow checks | BF16 is not native on Turing |
| Long-context training with RoPE | BF16, aware that RoPE drift grows with context even with the rotation in FP32 | BF16 precision distorts RoPE as context grows |
The hardware you serve from settles the format
The table gives defaults, not answers. Speed parity on paper sat alongside a 24% FP16 advantage on one Ampere part. FP16’s precision helped RL and FP32 fidelity, and it broke Gemma 3 on older GPUs. Whether your workload is range-bound or precision-bound shows up in its activation magnitudes, its sequence lengths and how far apart its training and serving paths sit. Those things are measured, not looked up. On one 3B model, 27% against 28% accuracy hid 19% of prompts changing correctness.
FAQ
Is FP4 better than FP16?
Not for accuracy. In a limited DistilBERT fine-tuning setting from a 2025 survey, FP16, BF16 and FP8 stayed close to FP32 accuracy while INT8 and especially 4-bit floating point (FP4) degraded more.
What is FP16 used for?
FP16 was the format behind the original mixed-precision training method, which nearly halved memory and sped up arithmetic on GPUs that support it. FP16 remains a common inference format for its speed and memory savings, as long as values stay inside its range.
Why is FP16 faster than FP32?
On NVIDIA Ampere GPUs, 16-bit Tensor Cores run twice as fast as TF32 mode, and FP16 tensors move half the bytes through memory. The compute gain depends on the operations actually running on those Tensor Cores.
