Someone on your team flips a flag, watches the model shrink to a quarter of its FP16 footprint, and declares FP4 a solved problem. Then the task accuracy quietly drops two points and nobody notices until a customer does. That gap — between “4-bit floating point is a free 4x win” and what actually happens on real hardware — is where most FP4 decisions go wrong. The naive reading treats FP4 as a switch: pick the format, cut memory and bandwidth by four, ship. The expert reading starts one level down, with the format itself. FP4 spends its four bits on a sign, a small exponent, and a tiny mantissa. That layout is not a detail — it is the whole trade. It decides which tensors survive the conversion and which ones fall apart, and it decides whether your target GPU accelerates the format in silicon or quietly emulates it and hands the savings back. What matters most about 4-bit floating point in practice? A floating-point number is a sign bit, an exponent field that sets the scale (the dynamic range), and a mantissa that sets the resolution within that scale (the precision). FP16 has 5 exponent bits and 10 mantissa bits. FP8 comes in two common flavours — E4M3 and E5M2 — trading range against precision. FP4 has only four bits total to divide across all three roles. The most common FP4 layout is E2M1: one sign bit, two exponent bits, one mantissa bit. That gives you a handful of representable magnitudes and almost no resolution between them. An E2M1 value can express a coarse ladder of numbers around zero, and that is it. There is no version of FP4 that quietly preserves the fidelity of a 16-bit tensor; the information is simply not there to store. This is why FP4 is never used raw. Every practical FP4 scheme pairs the 4-bit values with a scaling factor — often per-block or per-group of weights — stored at higher precision. The scale stretches the coarse FP4 ladder to cover the actual range of a given tensor block. The 4-bit values carry shape; the scale carries magnitude. Get the scaling granularity wrong and the format collapses regardless of how the arithmetic units are built. This is the same precision-versus-range reasoning that governs how memory bandwidth, not raw capacity, tends to bound inference throughput — the bits you move matter more than the bits you own. What is the E2M1 bit layout, and how do sign, exponent, and mantissa split across four bits? Here is the split, laid out explicitly so it is extractable without the surrounding prose: Format Sign Exponent Mantissa Rough character FP16 1 5 10 Wide range, high precision FP8 (E4M3) 1 4 3 Moderate range, usable precision FP8 (E5M2) 1 5 2 Wide range, coarse precision FP4 (E2M1) 1 2 1 Narrow range, minimal precision — needs a scale INT4 — — — 16 evenly spaced integers, symmetric grid The row that matters is E2M1. Two exponent bits mean roughly four exponent levels; one mantissa bit means a single step of resolution within each level. The representable set is a short, non-uniform ladder — values cluster near zero and spread out as magnitude grows, which is exactly the distribution most neural-network weights follow. That alignment is not an accident; it is why a floating-point 4-bit format can beat a fixed integer grid for weight-heavy tensors even at the same bit width. How does FP4 differ from INT4 and FP8, and when should you choose one? The common shortcut is “they’re all 4 bits, so they’re interchangeable.” They are not. INT4 lays down 16 evenly spaced integer levels — a uniform grid. FP4 lays down a non-uniform ladder that is denser near zero. For tensors whose values pile up near zero and trail off in a long tail (typical of LLM weight matrices), the floating-point ladder wastes fewer levels on regions that hold no data. INT4 with a good per-group scale can still be competitive and is often simpler to implement, which is why quantisation toolkits have supported it for years. FP8 is the more conservative step. It keeps enough mantissa (E4M3) to survive activation quantisation, where FP4 usually cannot. A common and defensible pattern in configurations we have worked through is to keep activations and sensitive layers at FP8 while pushing the bulk of the weights to FP4 — a mixed scheme, not a global one. Decision axis INT4 FP4 (E2M1) FP8 (E4M3) Value distribution fit Uniform grid Non-uniform, dense near zero Non-uniform, wider range Best for Weights with tight range Weight tensors with long tails Activations, sensitive layers Memory vs FP16 ~4x smaller ~4x smaller ~2x smaller Hardware acceleration Widely supported Newest tensor cores only Broadly supported on recent GPUs Accuracy risk without care Moderate High — needs fine scaling Low The choice is per-tensor and per-layer, not per-model. That is the single most important correction to the “just quantise it” instinct: FP4 is a targeting decision backed by calibration, not a checkbox. Which GPUs and compute APIs actually accelerate FP4 in hardware? This is where teams get burned. FP4 delivers its bandwidth win only when the arithmetic units read and multiply 4-bit values natively. Hardware FP4 tensor cores are recent — NVIDIA’s Blackwell generation introduced native FP4 matrix support, and older architectures without those units will emulate FP4 by unpacking to a wider type before computing. Emulation preserves the memory footprint savings but hands back much of the compute-throughput advantage, because the multiply still happens at higher precision under the hood. The compute-API layer sits on top of this. On the NVIDIA path, native FP4 kernels arrive through CUDA, cuDNN, and inference runtimes like TensorRT that expose the low-precision paths; the format is only as fast as the kernel that implements it. If you are working across vendors — the CUDA versus OpenCL trade-off when porting AI workloads — FP4 support is far more uneven than FP16 or INT8, and an OpenCL or SYCL path may have no native FP4 kernel at all yet. The rule is simple: confirm your exact GPU generation and your runtime’s kernel support before you assume the savings are real. For teams weighing whether this precision work is worth it, the honest first step is measuring where the workload is actually bound — memory bandwidth, compute, or something upstream. FP4 only pays off on bandwidth-bound stages, which is the same lens as reading DGX Spark memory bandwidth as an inference bottleneck signal. If your bottleneck is elsewhere, cutting weight precision buys nothing. What accuracy loss should you expect, and how do scaling and calibration keep it bounded? There is no universal number. FP4’s accuracy impact depends on the model, the layers you convert, and the calibration quality. What is reliable is the shape of the outcome: with careful per-block scaling and a calibration pass over representative data, well-behaved LLM inference can often hold task-metric degradation to under one percent (an observed pattern across low-precision engineering work, not a benchmarked guarantee for any specific model). Skip the calibration, apply one global scale, and the same model can lose several points and produce visibly worse outputs. Calibration is the mechanism that makes the bound real. It measures the actual value distribution of each tensor on sample inputs and sets the scaling factors so the coarse FP4 ladder lands where the data actually lives. This is why FP4 is a per-tensor decision: the calibrator finds which layers tolerate the crush and which need to stay at FP8 or FP16. The bounded accuracy delta is something you establish and verify, not something you assume. For which layers is FP4 safe, and where should you keep higher precision? As a working rubric — validate against your own model, do not treat it as law: Usually safe at FP4: the bulk of large linear/projection weight matrices in transformer blocks, where values are numerous and near-zero-clustered. Keep higher precision: activations feeding sensitive operations, normalisation layers, embeddings, and the final output projection, where small errors compound into visible degradation. Measure before deciding: attention weight matrices — sometimes fine at FP4, sometimes not, depending on the model. The discipline is to convert incrementally, re-run the task metric after each layer group, and stop where the delta exceeds your budget. What memory, bandwidth, and throughput gains does FP4 realistically deliver? The memory story is the clean part. FP4 halves the footprint of an FP8 model and cuts it to roughly a quarter of FP16 — that follows directly from the bit width, before any hardware question enters. Smaller weights mean less data crossing the memory bus per token, which is what lifts tokens-per-second on bandwidth-bound inference. That is the ROI anchor: throughput and memory saved per accelerator, against a bounded and verified accuracy delta. The throughput story is conditional. You realise the compute-throughput gain only on hardware with native FP4 units and a runtime that uses them; on everything else you keep the memory savings and forfeit the compute win. And even the memory win only converts to end-to-end speed on stages that were bandwidth-bound to begin with. This is the same distinction that separates peak specifications from delivered performance across every precision format. FAQ How does 4 bit floating point work? FP4 stores a number in four bits split across a sign, a small exponent, and a tiny mantissa, giving only a coarse ladder of representable values. In practice it is always paired with a higher-precision scaling factor per block of weights: the 4-bit values carry shape and the scale carries magnitude, so FP4 is never used raw. What is the bit layout of common FP4 formats like E2M1, and how do sign, exponent, and mantissa split across just 4 bits? The common FP4 layout, E2M1, uses one sign bit, two exponent bits, and one mantissa bit. Two exponent bits give roughly four scale levels and one mantissa bit gives a single resolution step, producing a short non-uniform ladder that clusters values near zero — which happens to match how most neural-network weights are distributed. How does FP4 differ from INT4 and FP8, and when should I choose one over the others? INT4 is a uniform 16-level integer grid; FP4 is a non-uniform ladder denser near zero, which fits long-tailed weight distributions better. FP8 keeps more mantissa and survives activation quantisation where FP4 usually cannot. Choose FP4 for bulk weight tensors, FP8 for activations and sensitive layers, and treat it as a per-layer decision rather than a global switch. Which GPUs and compute APIs actually accelerate FP4 in hardware versus emulating it, and how does that interact with my CUDA/OpenCL/SYCL choice? Native FP4 tensor cores are recent — NVIDIA’s Blackwell generation introduced hardware FP4 matrix support; older architectures emulate it by unpacking to a wider type, keeping the memory savings but losing the compute win. On the CUDA path, cuDNN and TensorRT expose native FP4 kernels; OpenCL and SYCL paths often have no native FP4 kernel yet, so confirm both GPU generation and runtime kernel support first. What accuracy loss should I expect from FP4 inference, and how do scaling and calibration keep it bounded? There is no universal number — it depends on the model, layers, and calibration quality. With careful per-block scaling and a calibration pass over representative data, well-behaved LLM inference can often hold task-metric degradation under one percent (an observed pattern, not a benchmarked guarantee), whereas a single global scale with no calibration can cost several points. For which layers or tensors is FP4 safe, and where should I keep higher precision? FP4 is usually safe for the bulk of large linear and projection weight matrices in transformer blocks. Keep activations feeding sensitive operations, normalisation layers, embeddings, and the final output projection at higher precision. Convert incrementally, re-check the task metric after each group, and stop where the delta exceeds your budget. What memory, bandwidth, and throughput gains does FP4 realistically deliver for ML inference? FP4 roughly halves memory footprint versus FP8 and cuts it to about a quarter of FP16, reducing data moved per token and lifting tokens-per-second on bandwidth-bound workloads. The compute-throughput gain only materialises on hardware with native FP4 units and a runtime that uses them; elsewhere you keep the memory saving and forfeit the compute advantage. Where FP4 belongs in a GPU cost decision FP4 is real, and on the right workload and the right silicon it lowers cost per inference in a way few other levers can. But it is a precision-engineering decision, not a flag. The teams that get burned are the ones that adopt it without knowing which tensors tolerate the crush, whether their GPU accelerates the format or emulates it, and what accuracy delta their calibration actually produced. FP4 viability — workload, target hardware, and compute API all agreeing that the bandwidth savings are attainable at an acceptable accuracy cost — is exactly the kind of input a GPU performance and cost review weighs before recommending it. Measure the bound, calibrate the layers, verify the delta. Then decide.
