Three years ago, the debate was FP32 vs FP16
The decision was conceptually simple: full precision or half precision? FP32 offered numerical safety; FP16 offered double the throughput with known stability risks. BF16 arrived as a pragmatic compromise — same 16-bit width as FP16, but with FP32’s exponent range, trading mantissa bits for dynamic range. Most teams adopted BF16 for training and inference on Ampere-generation hardware, and the conversation moved on.
Then FP8 entered production hardware. Hopper-generation GPUs (H100, H200) include native FP8 tensor cores, and the conversation shifted from a binary choice to a three-way comparison where each option encodes fundamentally different assumptions about numerical behaviour. Framing them as three rungs on one ladder is the mistake we see most often. They are three regimes.
Format properties shape operating regimes
Each format’s characteristics create a distinct operating envelope — not a good-better-best ranking, but genuinely different trade-off profiles:
FP32 (IEEE 754 single precision): 8 exponent bits, 23 mantissa bits. Wide dynamic range and high precision. The baseline against which everything else is measured. No hardware-accelerated throughput advantage on tensor cores — it is the slow, safe option.
FP16 (IEEE 754 half precision): 5 exponent bits, 10 mantissa bits. Limited dynamic range (max ~65,504) with moderate precision. The narrow dynamic range makes it problematic for training, where gradients can overflow or underflow, but workable for inference on well-behaved models. Historically the first “fast” option on tensor cores.
BF16 (Brain Float 16): 8 exponent bits, 7 mantissa bits. Same dynamic range as FP32 but substantially less precision. The engineering insight was that matching FP32’s range eliminates most of the overflow and underflow that plague FP16, even though individual values are coarser. That makes BF16 the practical default for training and many inference workloads on hardware that supports it.
FP8 (E4M3 and E5M2 variants): Two sub-formats coexist. E4M3 has 4 exponent bits and 3 mantissa bits — narrower range than BF16, slightly finer values than E5M2. E5M2 has 5 exponent bits and 2 mantissa bits — wider range, much coarser values. Hardware implementations typically support both, and inference frameworks can choose per-layer or per-tensor which variant to use.
These are not points on a linear scale. Moving from BF16 to FP8 does not merely halve the bit width; it changes which values can be represented, where rounding errors fall, how scale factors must be managed, and what the hardware does with values outside the representable range.
Precision format comparison
| Format | Bits | Exponent / Mantissa | Dynamic range | Key trade-off |
|---|---|---|---|---|
| FP32 | 32 | 8 / 23 | Very wide | Maximum precision; no tensor core throughput advantage |
| BF16 | 16 | 8 / 7 | Same as FP32 | Eliminates overflow/underflow; reduced per-value precision |
| FP16 | 16 | 5 / 10 | Narrow (~65K max) | Higher per-value precision than BF16; overflow risk in training |
| FP8 E4M3 | 8 | 4 / 3 | Moderate (~448 max) | Requires scale management; highest throughput on Hopper |
| FP8 E5M2 | 8 | 5 / 2 | Wide (~57K max) | Very coarse values; used where range matters more than precision |
Format properties are published specifications, not measurements — the numbers above are drawn from the IEEE 754 and OCP FP8 definitions, and they say nothing about how a given device behaves under load.
Throughput and density: the FP8 proposition
FP8’s primary appeal is raw throughput. On H100 tensor cores, FP8 matrix multiplications are designed to run at up to roughly 2x the rate of BF16 and 4x the rate of FP32, per NVIDIA’s published architectural targets (vendor specification, not a measured result on your stack). For memory-bandwidth-bound inference, FP8 also halves the bytes read from HBM per weight compared with BF16, which directly improves tokens-per-second for autoregressive decoding.
The density benefit is equally significant. A 70B-parameter model in FP8 fits in roughly 70 GB of HBM — about one H100 80GB card. The same model in BF16 needs roughly 140 GB, forcing multi-GPU deployment with the associated communication overhead through NCCL. FP8 does not just make each GPU faster; it changes the deployment topology.
Those are real advantages. They arrive with constraints BF16 does not impose.
Stability and risk: what FP8 asks in return
FP8 E4M3 can represent values up to 448. Activations that exceed this range must be clipped or scaled. Unlike BF16, which shares FP32’s exponent range and rarely encounters overflow in practice, FP8 requires per-tensor scale factors to map the activation range into the representable interval.
That scaling is not optional. If scale factors are poorly calibrated, the model produces garbage. If they are calibrated on data that does not represent the production distribution, the model works on the calibration set and fails on edge cases — a pattern we see repeatedly when a TensorRT or Transformer Engine deployment passes validation and then degrades on live traffic.
The E5M2 variant extends the range (max ~57,344) at the cost of further reduced precision. Two mantissa bits means each representable value is an extremely coarse approximation. E5M2 is sometimes used for gradient representation in training, where range matters more than per-value accuracy, but it is aggressive even for inference on precision-sensitive tasks.
The practical risk profile is asymmetric: BF16 is numerically robust by default and needs minimal per-deployment validation, while FP8 is numerically viable but requires careful calibration, per-task validation, and awareness of failure modes that do not exist at higher precisions.
Is FP8 the same kind of decision as INT8?
No, and conflating the two is a common error. It is worth separating this float-versus-float comparison from the float-versus-integer one. INT8 discretises values onto a uniform integer grid with a fixed scale and zero point rather than carrying a per-value exponent. FP8 keeps a floating exponent, so it preserves dynamic range across activations of widely different magnitudes — something INT8 only approximates through per-channel scaling. INT8 can edge out FP8 on raw throughput and memory on hardware tuned for integer matmul, but it assumes the activation distribution can be captured by a single linear quantisation map. Choosing between FP8 and INT8 is therefore not a finer point on the same precision scale; it is a decision about whether your workload tolerates a uniform grid or needs the floating exponent’s adaptive range.
FP4 sits one step further along the same FP8 logic and makes the assumption harder. At four bits, the representable set is small enough that per-tensor scaling is rarely sufficient on its own — block-level scaling and much tighter outlier handling become part of the format’s operating requirements rather than an optimisation. The question a four-bit regime asks is not “how much precision can I lose” but “is this tensor’s distribution narrow and stable enough to survive a handful of representable values”.
Hardware support defines what is viable
Precision format choice is not purely a software decision. The hardware must have dedicated execution units for the target format, or the throughput benefit disappears.
Tensor cores on Ampere (A100) natively accelerate FP16, BF16, and TF32. They do not accelerate FP8 — running FP8 on A100 means software emulation with no throughput advantage. Hopper (H100, H200) adds native FP8 tensor cores. Older V100 silicon supports only FP16 on tensor cores, with no BF16 acceleration.
So the viable format is hardware-conditional. A deployment decision that assumes FP8 availability but targets A100 gains nothing. A deployment optimised for BF16 on Hopper leaves FP8 throughput on the table. The format choice has to be made jointly with hardware selection, not independently of it — which is precisely the interplay explored in how hardware architecture constrains precision decisions.
Availability is part of what defines a regime, and a benchmark that reports per format has to say so explicitly. A 26Q3 LynxBenchAI run covers fp64/fp32/fp16/bf16 on compute, fp32/bf16 on training, and fp16/int8/fp8 on inference, reporting each on its own rather than folding them into a single figure. Where a device cannot execute one of those formats, the result is a zero rather than a silent omission — the distance between “this regime exists” and “this regime is available here” stays legible in the output. Those figures belong to the named release that produced them; a number from one release name is not comparable against a number from another.
How do you choose between FP8, BF16, and FP16?
The temptation is to arrange the formats as a progression — FP32 → BF16 → FP8, each step “better” — and that framing misleads because it implies one axis of comparison where the reality is multi-dimensional.
FP8 is not a universal improvement over BF16. It is a different operating regime, offering higher throughput and density at the cost of narrower representable range, mandatory scale-factor management, and higher sensitivity to calibration quality. Where those constraints are manageable — well-behaved activations, representative calibration data, tasks with high quantisation tolerance — FP8 is the efficiency choice. Where precision sensitivity is high, activation distributions are unpredictable, or deployment conditions vary, BF16 provides robustness FP8 does not.
The engineering discipline is selecting the format that matches the workload’s requirements and the hardware’s capabilities, rather than defaulting to the newest or most aggressive option. Each format is a tool for a specific set of conditions, and the conditions determine the choice. As explored in the economic implications of precision decisions, the cost of getting this wrong is not only numerical — it is operational and financial.
LynxBenchAI treats each precision format as a distinct measurement regime: results are reported per format, so hardware comparisons preserve the conditions that determine the right choice instead of averaging across them. If your own stack were measured that way — same model, same workload scaled until throughput plateaued, each format reported separately — which regime would the numbers actually support?
Frequently Asked Questions
What does each of FP8, FP16, and BF16 represent as a numerical operating regime, beyond raw bit width?
Each format encodes a different split between exponent and mantissa bits, which determines dynamic range, per-value precision, and how the hardware handles values outside the representable interval. FP16 packs more precision into a narrow range; BF16 trades mantissa bits to keep FP32’s range; FP8 (in E4M3 or E5M2 variants) compresses further and shifts the burden onto scale-factor management. Bit width is the cheapest part of the description — the real distinction is which assumptions about numerical behaviour each format bakes in.
How do FP16 and BF16 trade stability for efficiency differently, even though both are 16 bits?
FP16 keeps 10 mantissa bits but only 5 exponent bits, so it has higher per-value precision than BF16 inside a narrow dynamic range (max ~65,504) and is prone to overflow and underflow under training-grade gradient magnitudes. BF16 inverts the trade: 8 exponent bits give it the same dynamic range as FP32, while 7 mantissa bits make individual values coarser. The result is that BF16 is robust by default for training and most inference, whereas FP16 demands loss-scaling and careful range management to stay stable.
Where does FP8 prioritise throughput and density, and what does that prioritisation cost?
On Hopper tensor cores, FP8 matmul targets roughly 2x BF16 and 4x FP32 throughput, and halves HBM bytes per weight — enough that a 70B model fits on a single 80GB H100 instead of spanning two cards in BF16. The cost is a much narrower representable range (~448 for E4M3, ~57K for E5M2), mandatory per-tensor scale factors, and a class of calibration failures that simply does not exist at BF16. Get the calibration wrong on unrepresentative data and the model degrades on edge cases that never appeared in the calibration set.
Why does each precision format encode implicit assumptions about the numerical behaviour of the workload?
FP16 assumes activations and gradients stay inside a narrow range. BF16 assumes you would rather lose precision than risk overflow. FP8 assumes activation distributions are well-behaved enough that a calibrated scale factor maps them into a tiny representable interval without material loss. Choosing a format is therefore choosing which of those assumptions you are willing to underwrite for a specific workload — and which failure modes you are taking on if the assumption breaks.
Which axes — stability, dynamic range, throughput, hardware support — actually matter when comparing precision formats, and why is reducing them to a single “which is best” question the wrong frame?
All four matter, and they interact. Stability and dynamic range decide whether the format is numerically viable for the model’s activations and gradients; throughput decides whether the format improves cost-per-token or only looks good on paper; hardware support decides whether that throughput exists at all, since FP8 on A100 reverts to emulated execution. A single “which is best” question collapses these independent axes into one, which is why it cannot be answered — the formats are distinct regimes, and the right one is whichever matches the workload and the available silicon.
How does FP8 compare to INT8 for inference, and why is integer-vs-float a different axis than the FP8/FP16/BF16 comparison?
INT8 places values on a uniform integer grid with a fixed scale and zero point, whereas FP8 keeps a floating exponent that preserves dynamic range across activations of very different magnitudes. INT8 can match or beat FP8 on raw throughput and memory where hardware is tuned for integer matmul, but only if a single linear quantisation map captures the activation distribution. The FP8/FP16/BF16 comparison is about how a floating exponent and mantissa are split; FP8-versus-INT8 is the orthogonal question of whether a floating exponent is needed at all.
Where does FP4 sit relative to FP8 as an operating regime, and what changes about the numerical assumptions when a format drops to four bits?
FP4 continues the FP8 logic but tightens every assumption behind it: the representable set becomes small enough that per-tensor scaling alone is rarely sufficient, so block-level scaling and explicit outlier handling become part of the operating requirements rather than optional tuning. Where FP8 asks whether a calibrated scale factor can map a distribution into a narrow interval, FP4 asks whether the distribution is narrow and stable enough to survive a handful of representable values at all. Availability is also part of the regime — a device that cannot execute the format produces a zero, not a quietly missing row.
