A team flips a quantisation flag, moves a model to FP4, and expects a free doubling of throughput. Sometimes they get it. Often they get an accuracy hit and almost no speedup — because FP4’s value was never about the numbers being smaller. That is the misconception worth clearing up first. The naive read of 4-bit floating point is “smaller numbers, cheaper inference,” and the only trade-off anyone bothers to check is accuracy. The expert read starts one layer down: FP4 changes what a GPU actually moves and computes per cycle. Whether that helps you depends entirely on whether your workload was memory-bandwidth-bound or compute-bound in the first place. If your fleet is already leaving memory bandwidth on the table, FP4 can roughly double effective throughput on the same silicon. If it is not, you inherit accuracy risk for very little gain. So FP4 is not a universal speed lever. It is a lever that reclaims a specific kind of wasted capacity. This article explains how the format is built, where it loses dynamic range, how it differs from its neighbours, and — the part that actually decides ROI — how to tell whether it reclaims capacity or just relocates your bottleneck. How does 4-bit floating point work in practice? A floating-point number is three parts: a sign bit, an exponent field, and a mantissa (also called the significand). The exponent sets the scale — roughly, where the decimal point sits — and the mantissa sets the precision within that scale. FP16 spends 16 bits on this. FP4 spends four. With four bits total, one goes to sign, leaving three to split between exponent and mantissa. The common layout, E2M1, gives two exponent bits and one mantissa bit. That is not many values. E2M1 represents on the order of 16 distinct numbers across its whole range, including zero and the signed duplicates. You are no longer describing a smooth number line; you are choosing from a tiny menu of representable points. In practice this means FP4 is almost never used as a standalone format for a whole tensor. It is paired with a scaling factor — a higher-precision multiplier applied per block of weights — so the tiny FP4 menu can be positioned over the range where a given block’s values actually live. NVIDIA’s Blackwell generation implements this as micro-scaling (the MXFP4 / NVFP4 families), where each small block of FP4 values carries its own scale. Without that scaling, the format is close to useless for real models; with it, FP4 becomes a viable inference numeric. How is FP4 laid out in bits, and where does its dynamic range come from? The dynamic range of a floating-point format comes almost entirely from the exponent bits, not the mantissa. Two exponent bits in E2M1 give you a narrow window of scales — you can represent both small and larger magnitudes, but the gaps between representable values are wide, and the mantissa’s single bit means each exponent step offers only two significand choices. This is the structural fact that matters: FP4 buys its four-bit footprint by throwing away resolution, and it recovers usable range only through the external scaling factor. The bits inside the value are not where the range lives once you are in production; the per-block scale is. That is why “FP4 inference” in any serious runtime is really “FP4 values plus a companion scaling scheme,” and why comparing raw FP4 to raw FP8 on paper misses the point. The comparison that matters is the packaged format — the values and their scaling — against your accuracy tolerance. How does FP4 differ from INT4, FP8, and FP16? The four formats occupy different points on a footprint-versus-fidelity curve, and each earns its place under different conditions. Format Bits Structure Dynamic range Typical fit FP16 16 1 sign / 5 exp / 10 mantissa Wide Safe default; training and accuracy-sensitive inference FP8 8 1 sign / (4 or 5) exp / (3 or 2) mantissa Moderate Strong inference default on recent GPUs; low accuracy risk INT4 4 Integer, no exponent None (uniform steps + scale) Weight-only quantisation where value distribution is well-behaved FP4 4 1 sign / 2 exp / 1 mantissa + block scale Narrow, scale-recovered Bandwidth-bound inference at a tolerated accuracy cost The INT4-versus-FP4 distinction is the one most often flattened. INT4 has uniform spacing between representable values — every step is the same size within a scaled block. FP4, because it has an exponent, spaces its values non-uniformly: finer resolution near zero, coarser further out. For weight distributions that are peaked near zero — common in transformer weights — the floating-point spacing can preserve the important small values better than uniform INT4 at the same bit count. That is the reason a 4-bit float exists at all rather than everyone just using INT4. FP8 sits in a different role. On current hardware it is often the pragmatic inference default because the accuracy hit is small and predictable. FP4 is the step you take when FP8’s memory footprint is still the thing standing between you and higher throughput — and only then. The general framing here connects to the broader question of what memory capacity actually buys you versus the real bottleneck, which is rarely capacity itself. Which workloads benefit most from FP4? Here is the part the format-comparison tables never tell you. FP4 helps most when the GPU is spending its cycles waiting for memory, not doing arithmetic. Large-language-model decoding is the canonical example. During autoregressive token generation, each step reads the full weight set (and a growing KV cache) but does comparatively little math per byte moved. The arithmetic intensity is low, so the memory subsystem — HBM bandwidth — is the ceiling. Halving the bytes per weight, which is roughly what FP4 does versus FP8, directly raises how many useful FLOPs you extract per second, because you were never compute-limited to begin with. Now the opposite case. A large-batch, compute-dense workload — say a heavily batched prefill or a convolution-heavy vision model saturating the tensor cores — is already using its arithmetic units near capacity. Shrinking the operands does not help; the bottleneck is the math, not the fetch. Here FP4 delivers the accuracy risk with little of the throughput reward. FP4 inference can roughly halve memory footprint and memory-bandwidth demand versus FP8, and cut it about 4x versus FP16 (an observed-pattern framing from the format’s bit widths and how block-scaled inference behaves, not a benchmark of your specific model). Whether that halving becomes throughput depends on one property of your workload: was memory the ceiling? This is exactly why a GPU performance audit matters before a format switch — the audit tells you whether the capacity FP4 reclaims was actually being wasted. If a profiler shows your decode step at, say, 30% tensor-core utilisation with HBM near saturation, you have a strong FP4 candidate. If tensor cores are already pinned, you do not. What accuracy trade-offs does FP4 introduce, and how do scaling and calibration mitigate them? FP4’s one mantissa bit means large quantisation error per value in isolation. What makes it survivable in production is that the error is managed, not eliminated. Three mechanisms do the managing. Per-block scaling positions the tiny FP4 menu over each block’s actual value range, so the format spends its resolution where the numbers are. Calibration — running representative data through the model to fit those scales, and sometimes to identify outlier channels that need higher precision — is what separates a usable FP4 deployment from a broken one. And selective precision keeps sensitive layers (often the first and last, or attention projections) at FP8 or FP16 while the bulk of the weights go to FP4. The failure mode to watch: teams quantise uniformly, skip calibration, measure perplexity on a benchmark set that looks fine, and then discover degradation on the long-tail inputs that calibration never saw. In our experience, the accuracy surprises with FP4 come less from the format’s headline error and more from unrepresentative calibration data — the model looks acceptable on the eval set and drifts on production traffic. Treat the calibration set as a first-class artifact, not an afterthought. FAQ What’s worth understanding about 4-bit floating point first? FP4 uses four bits — one sign, and three split between exponent and mantissa (commonly E2M1: two exponent, one mantissa). That yields on the order of 16 representable values, so in practice FP4 is always paired with a higher-precision per-block scaling factor that positions those few values over each weight block’s actual range. It is a numeric for inference, not a standalone precision for whole tensors. How is FP4 laid out in bits — how are exponent and mantissa allocated, and where does its dynamic range come from? The E2M1 layout is one sign bit, two exponent bits, and one mantissa bit. Dynamic range comes almost entirely from the exponent, but two exponent bits give only a narrow window; the usable range in production is recovered by the external per-block scaling factor, not by the bits inside the value. That is why serious FP4 is really “FP4 values plus a scaling scheme.” How does FP4 differ from INT4, FP8, and FP16, and when does each make sense? FP16 is the wide-range safe default; FP8 is a strong, low-risk inference default on recent GPUs; INT4 uses uniform value spacing plus a scale; FP4 uses non-uniform (exponent-based) spacing with a block scale. FP4’s floating spacing preserves small near-zero weights better than uniform INT4 at the same four bits, which is why a 4-bit float exists. FP4 is the step past FP8 when memory footprint is still the throughput bottleneck. Which workloads benefit most from FP4 — and how does GPU utilisation determine whether it actually helps? Memory-bandwidth-bound workloads benefit most — notably LLM autoregressive decoding, where each step moves the full weight set but does little math per byte. Compute-bound workloads (large-batch prefill, tensor-core-saturated vision models) see little gain because the bottleneck is arithmetic, not fetch. Utilisation is the deciding signal: low tensor-core use with saturated HBM means FP4 reclaims wasted capacity; pinned tensor cores mean it does not. What accuracy trade-offs does FP4 introduce, and how do scaling and calibration mitigate them? FP4’s single mantissa bit produces large per-value quantisation error, managed rather than eliminated by three mechanisms: per-block scaling, calibration on representative data, and keeping sensitive layers at higher precision. The most common accuracy surprise comes from unrepresentative calibration data — the model looks fine on the eval set and drifts on production traffic. Treat the calibration set as a first-class artifact. How do I decide whether FP4 reclaims wasted GPU capacity or just shifts the bottleneck? Profile first: measure tensor-core utilisation against HBM bandwidth on your actual workload. If arithmetic units sit idle while memory bandwidth is saturated, FP4 turns reclaimed bandwidth into throughput. If the arithmetic units are already near capacity, FP4 mostly adds accuracy risk. The format only reclaims capacity a utilisation audit proves is being wasted. Deciding without guessing The honest version of the FP4 decision is short. Do not ask “should we use FP4.” Ask “is this workload memory-bandwidth-bound at the operating point we care about, and is our accuracy tolerance wide enough to absorb four-bit weights with proper calibration.” Both answers come from measurement, not from the format’s spec sheet. If your instinct is to reach for a numeric format before you have profiled where the cycles actually go, the failure class to name is optimising a bottleneck you have not located. A GPU performance audit — the same utilisation work that underpins any FP4 candidacy call — is what turns that instinct into an evidence-backed decision about whether a format switch reclaims wasted silicon or just moves the wall a few metres down the road.
