Every floating-point format from the double precision floating point format (FP64) down to FP4 is an operating regime, not a rung on a quality ladder. Each one decides how many bits go to range and how many to precision, what supporting machinery the workload has to carry (scaling, wide accumulation, master weights), and which silicon runs it at full rate. The format that fits a workload is the one whose assumptions about value range and rounding error that workload actually meets, on a device that executes it natively.

It is tempting to treat fewer bits as the same computation done more cheaply. That reading stops working once you look at where the bits go. FP16 and BF16 are both 16 bits wide and allocate them in opposite ways, and the two FP8 encodings, also identical in width, are recommended for different kinds of tensor.

The format ladder: bits, range and typical role

A binary floating-point value is a sign, an exponent and a mantissa. Exponent bits set the range of representable values and mantissa bits set how finely values can be told apart within that range.

Format	Sign / exponent / mantissa bits	Largest normal value	Typical role
FP64 (binary64)	1 / 11 / 52	far beyond FP32	Scientific computing that needs at least double precision; the refinement step in mixed-precision solvers
FP32 (binary32)	1 / 8 / 23	3.4 × 10^38	Reference baseline for deep learning; master weights and accumulation in mixed precision
FP16 (binary16)	1 / 5 / 10	65,504	Mixed-precision training with loss scaling; inference
BF16	1 / 8 / 7	3.4 × 10^38	Large-scale training without loss scaling
FP8 E4M3	1 / 4 / 3	448	Weights and activations
FP8 E5M2	1 / 5 / 2	57,344	Gradients
FP4 E2M1	1 / 2 / 1	6	Only with a shared scale per block (MXFP4: 32 values, E8M0 scale; NVFP4: 16 values, E4M3 scale)
INT8 (comparison point)	integer	127	Post-training quantised inference

Two rows are easy to misread. BF16 keeps FP32’s 8 exponent bits, so its range matches FP32 while its precision drops to between two and three decimal digits. E4M3 reaches 448 only because it gives up infinities and keeps a single NaN pattern, which buys it seven extra magnitudes. FP4 E2M1 has only fifteen distinct values, and both MXFP4 and NVFP4 pair it with block scaling.

Double precision floating point format: what FP64 is still for

IEEE 754-2019 specifies interchange and arithmetic formats for binary and decimal floating-point, along with exception conditions and how they are handled by default. Double precision is its 64-bit binary format, binary64, with 1 sign bit, 11 exponent bits and 52 stored mantissa bits. The standard also fixes a property that matters a great deal for reproducibility: numerical results and exceptions are uniquely determined by the input values, the sequence of operations and the destination formats.

Many scientific computing applications need at least FP64 to keep their accuracy, while deep learning generally manages with FP32. Running in FP64 costs speed. A 2020 survey of mixed-precision numerical methods notes that 32-bit operations are often at least twice as fast as 64-bit ones on modern architectures, partly because the arithmetic itself is faster and partly because only half as many bytes move through memory. The survey also notes that casting from FP64 to FP32 halves the data but loses about 8 decimal digits.

In mixed-precision solvers, FP64 serves as the anchor of the regime rather than the precision for everything. The same survey reports that factorising in FP16 on GPU Tensor Cores and refining towards FP64 reached FP64 accuracy for matrices with condition numbers up to about 10^5, with solvers up to four times faster than an optimised double-precision solver on a single V100. Our explainer on HPL-MxP for GPU cluster benchmarking applies mixed precision to a whole cluster.

Whether you get usable FP64 throughput depends on the device. In the 26Q3 results, the Dense MatMul fp64 test scores 414.4 on the NVIDIA H100 80GB HBM3 and 9.5 on the NVIDIA GeForce RTX 4090. The same two devices score 1264.3 and 295.4 on Dense MatMul fp16. In those figures, the ratio of fp16 to fp64 score is about 3 on the H100 and about 31 on the RTX 4090, an order of magnitude apart.

Should I train in BF16 or FP16?

For large-scale pre-training, BF16 is the more forgiving regime because its 8-bit exponent matches FP32’s range. FP16 keeps three more mantissa bits (10 against 7), trading range for higher precision.

FP16’s 5-bit exponent limits normal values to roughly 2^-14 to 2^16, and small gradients underflow below that range. To handle this, the mixed-precision recipe of Micikevicius et al. (2018) keeps an FP32 master copy of the weights, scales the loss, and accumulates FP16 products into FP32. In their experiments, an SSD detector failed to train in FP16 without loss scaling.

BF16 avoids the need to tune loss scaling. Kalamkar et al. (2019) report that BF16 training matched FP32 results in the same number of iterations with no hyperparameter changes. Google designed the format for its TPUs on the view that neural networks are far more sensitive to exponent size than to mantissa size. Google notes that multiplier area grows with the square of the mantissa width, so a BF16 multiplier takes about half the silicon of an FP16 one. When FP16 caused irreversible divergences in Bloom above 100B parameters, Bloom and other large models such as Gopher and Chinchilla moved to BF16. The cost of BF16 is its short mantissa: work on Gopher found BF16 degrading against FP32 even with stochastic rounding.

A study of reinforcement-learning fine-tuning brought the precision side back into focus. It traced the mismatch between the training and inference engines to BF16 rounding, and reported that FP16, with 8 times the precision (2^10 mantissa values against 2^7), kept that mismatch about 24 times lower. The authors argue that the model’s range is already settled during pre-training, so BF16’s extra range buys nothing at that stage, while its rounding error compounds over long responses.

Put simply, BF16 assumes the hard problem is range, and FP16 assumes range is already under control and precision is the scarce resource. Even within BF16, conversions are not identical across hardware: Google TPUs round to nearest even, Arm uses a non-IEEE round-to-odd mode, and NVIDIA supports round-to-nearest-even. The BF16 vs FP16 comparison of dynamic range and precision works through range-bound and precision-bound cases in more depth, and the guide to why FP16 needs mixed precision to stay stable covers loss scaling step by step.

What FP8 buys, and the scaling it demands

Throughput and density come first in the FP8 regime. The OCP 8-bit Floating Point Specification (revision 1.0, 2023), written by authors from AMD, Arm, Google, Intel, Meta and NVIDIA, defines the E4M3 and E5M2 interchange encodings. Micikevicius et al. (2022) recommend E4M3 for weight and activation tensors and E5M2 for gradients, with the aim of smaller, more power-efficient math pipelines and less pressure on memory bandwidth.

That density has a price. Each of these costs comes with the format:

• Scaling. Casting tensors to FP8 without a scale factor caused a significant accuracy loss in Micikevicius et al.’s experiments, and some networks need a scale factor per tensor to keep their accuracy.
• Wide accumulation. FP8 matrix products are accumulated in FP16, BF16 or FP32, because E4 lacks the range and E5 lacks the precision to accumulate in FP8. A 2025 study of datacenter cost for LLM inference found that Hopper GPUs use a 14-bit accumulator for FP8 GEMMs, while Gaudi HPUs always accumulate in FP32.
• Variants that are not interchangeable. E4M3 tops out at 448 on NVIDIA GPUs and at 240 on Gaudi 2, which follows IEEE conventions for special values.
• Long-run stability. Fishman et al. (2024) trained LLMs in FP8 on up to 2 trillion tokens and found instabilities that shorter runs never showed, which they traced to the SwiGLU activation amplifying outliers.

The throughput gain depends on conditions. The 2025 cost study notes that FP8 may offer up to twice BF16’s theoretical throughput, but that reduced utilisation and bottlenecks in memory bandwidth, special function units or scheduling often keep it out of reach. In that study’s decode-phase measurements, Gaudi 2 gained nearly 50% throughput from FP8 over BF16, while the H100 and H200 showed little to no improvement. A 2024 microbenchmark study found FP8 slower than FP16 on small matrices, and close to twice FP16 throughput on the H800 and RTX 4090 at N=16384. With short sequences in memory-bound decode, the same study saw no significant compute advantage from FP8.

So the FP8 question is not just whether the chip supports it. It is also whether the matrix shapes, the accumulation path and the scaling recipe let the workload actually reach that support. That constraint is covered in more detail in how hardware architecture constrains precision choices.

Is FP8 better than INT8 for inference?

It depends on how severe the model’s outliers are.

Integer versus float is a separate axis from FP8 versus FP16. Integer formats space their values evenly, while floating-point formats space them exponentially. As a result, a float keeps a roughly constant signal-to-noise ratio across its range, while an integer has only a narrow band with high SNR. In one modelled tensor with outliers, FP8 gave 635 times the SNR of INT8.

Two lines of evidence pull in different directions:

• The case for FP8: Kuzmin et al. (2022) conclude that, in post-training quantisation, FP8 beats INT8 on accuracy, and that the right number of exponent bits depends on how severe the outliers are. Micikevicius et al. (2022) found that FP8 post-training quantisation kept BERT’s accuracy while INT8 lost a significant amount.
• The case for INT8: van Baalen et al. (2023) argue that, in dedicated hardware, FP8-E4 needs 53% more gates than INT8 with a 16-bit accumulator and 183% more with a 32-bit one. They find INT8 more accurate for well-behaved networks, with FP8 ahead only on layers with heavy outliers, and they argue those outliers are better fixed in the model than worked around with a different format.

The two papers are less opposed than they first look. The 2023 paper cites the 2022 one. Both papers find that quantisation-aware training largely closes the accuracy gap between the formats. The hardware-cost argument concerns dedicated hardware implementations of each format. A buyer choosing an existing device is looking at a fixed FP8 rate and a fixed INT8 rate, and should compare those two directly. The claim that FP8 matches INT8 throughput rests on the same condition: it holds only where the hardware offers both at the same rate. INT8 is also a well-established regime: a 2020 study of integer quantisation for inference kept models within 1% of their floating-point baselines, including MobileNets and BERT-large, which are known to be hard to quantise.

What it takes to make 4-bit floating point usable

At four bits, E2M1 spans only about -6 to 6, so values need a scale stored outside them. In practice, 4-bit floating point becomes a small code paired with a scale shared across a block. E2M1 has 1 sign bit, 2 exponent bits and 1 mantissa bit, which gives fifteen distinct values between 0 and ±6. MXFP4 shares one power-of-two E8M0 scale across 32 values. NVFP4 shares an E4M3 scale across 16 values and adds a per-tensor FP32 scale on top. NVIDIA puts NVFP4’s storage at about 4.5 bits per value, about 3.5 times smaller than FP16 and 1.8 times smaller than FP8.

Getting the arithmetic to hold up takes a lot of supporting technique. NVIDIA pretrained a 12B model on 10 trillion tokens in NVFP4 and reached 62.58% on MMLU-pro, against 62.62% for the FP8 baseline. That result relied on Random Hadamard transforms to spread outliers, 2D block scaling for weights, stochastic rounding on gradients, and about 16% of the linear layers kept in BF16. Removing any one of those techniques made convergence worse. A separate 2025 study of fully quantised FP4 training also reached downstream performance comparable to BF16, using stochastic rounding only in the backward pass.

NVIDIA’s Blackwell is the first GPU to support FP4 matrix multiplication in hardware, and before formats like NVFP4 arrived on that hardware, FP4 was usable only for storage. Several NVFP4 results cited here, including the 12B pretraining result against FP8, come from NVIDIA’s own research on Blackwell, so we would read them as strong but vendor-generated. The 26Q3 release includes no FP4 test, so there is no LynxBenchAI FP4 score to quote.

Comparing formats per workload: a decision table

Range, precision, throughput and hardware support are four separate axes, and a precision choice is an explicit trade-off among them. Collapsing them into one “which is best” ranking discards exactly the information the decision needs.

If the workload…	The regime whose assumptions it meets	What to verify first
Solves ill-conditioned linear systems or needs at least double precision	FP64, or a lower-precision factorisation refined in FP64	The device’s FP64 rate, which varies widely between devices
Pre-trains a large model whose gradients span a wide range	BF16 with FP32 accumulation	Native BF16 support and the accumulation precision
Fine-tunes with separate training and inference engines that must agree	FP16, with loss scaling available	Whether the mismatch between engines is the actual instability
Runs large, compute-bound GEMMs in LLM linear layers	FP8 E4M3 with per-tensor or finer scaling	An FP8 tensor path, matrix sizes large enough to use it, and how the device accumulates
Is dominated by memory-bound decode	Lower precision mainly for smaller weights; compute gains may not appear	Measured throughput on thin GEMMs, not peak TFLOPS
Serves well-behaved CNNs	INT8 post-training quantisation	Calibration method and per-channel weight scales
Has outlier-heavy transformer layers	FP8 E4M3, or INT8 with outlier handling	Whether the outliers can be fixed in the model
Is limited by the size of its weights, on FP4-capable hardware	NVFP4 with sensitive layers kept in higher precision	A native FP4 datapath and which layers must stay in BF16

Reading per-precision benchmark scores, including the zeros

Numerical precision is a first-class design trade-off in LynxBenchAI, not a detail folded into one score. A 26Q3 run covers fp64, fp32, fp16 and bf16 on compute workloads, fp32 and bf16 on training, and fp16, int8 and fp8 on inference, and reports each precision on its own instead of folding them into one figure. Each result must pass a per-precision correctness threshold before it counts. Aggregation weights throughput by the memory a run actually moves, so a lower-precision run has to earn its speed-up rather than getting credit just for being cheaper. When a device cannot execute a precision, the result is scored as zero, not quietly left out.

In the 26Q3 results, the NVIDIA A100-SXM4-80GB scores 0.0 on the DistilBERT inference test in fp8, against 1267.4 in fp16 and 614.0 in int8. The NVIDIA H100 80GB HBM3 scores 1362.7 on the same fp8 test and 2611.7 in fp16. The A100’s zero means that executor produced no fp8 result in the release. Because that zero appears in the output instead of a blank, you can see that FP8 being defined as a format is not the same as FP8 being available on this device. The H100’s lower fp8 score is reported next to its fp16 score as it stands; the release does not turn it into a verdict on FP8. The compute side follows the same rule: the Apple M1 scores 0.0 on Dense MatMul fp64.

These scores have declared limits. They reflect the fixed model catalogue of the 26Q3 release, not your application, and each test is a single timed measurement window, not a median across repeated runs. They can be compared only with other 26Q3 figures. The LynxBenchAI Personal Edition benchmarks AI performance across precisions on your own hardware. It measures the device and its software stack on the release’s catalogue, not your model. Choosing and validating a regime for a specific model and stack (quantising, calibrating, porting kernels) is engineering work rather than benchmarking, and TechnoLynx’s engineering services take that on.

Start from the tensor with the widest range

The most useful question to ask before comparing formats is which tensor in your workload sets the range requirement and which sets the precision requirement. The tensor that settles it is not always the one you would expect. After a softmax, attention probabilities sit in [0, 1], bunched near zero, so a naive 4-bit scale wastes almost all of its range on them. In DeepSeek-V3, the linear layers run in FP8 while the GEMMs inside attention stay in BF16 to preserve accuracy. The models cited here run as a mix of regimes, one per tensor class, rather than in a single format. The open question is how far down that mix can go.