Someone reads that a model quantised to FP4 halves memory over FP8 and concludes the latency will halve too. It usually does not. FP4 is a genuine lever, but it moves cost only in specific places, and treating it as a universal 2x is where the reasoning goes wrong. The temptation is understandable. Four bits is half of eight bits, so surely everything scales linearly: half the memory, half the bandwidth, half the latency, half the bill. Memory footprint does roughly halve. The rest depends entirely on where your inference workload actually spends its time — and on whether the silicon in front of you can do FP4 math natively or only pretends to by upconverting on the fly. What should you know about 4-bit floating-point in practice? A 4-bit floating-point number spends its four bits on a sign, a few exponent bits, and one or two mantissa bits. That is almost no information. An FP4 value like the common E2M1 layout (2 exponent bits, 1 mantissa bit) can represent only a handful of distinct magnitudes. You are not storing a precise number; you are storing a coarse bucket that a scaling factor later stretches back toward the original value. This is the part people skip. FP4 on its own has a dynamic range so narrow that quantising real model weights directly into it destroys them. A single outlier weight — and transformer weight distributions are full of outliers — either saturates the format or forces the scale so wide that every ordinary weight collapses to zero. Raw per-tensor FP4 is not usable for most models. What makes it usable is block-level scaling, and that is where MXFP4 and NVFP4 come in. The practical meaning: FP4 is a storage-and-transport format wrapped around a scaling scheme. The four bits carry the shape; the scale carries the magnitude. Judge any FP4 claim by how the scaling is done, not by the bit count alone. How does FP4 differ from INT4, and why does floating-point matter at 4 bits? INT4 spaces its 16 representable values evenly across a range. FP4 spaces them logarithmically — closer together near zero, further apart as magnitude grows. For weight distributions that pile up near zero with a long tail (which is most of them), the floating-point spacing wastes fewer codes on ranges that barely occur. That is the whole argument for floating-point at four bits: the non-uniform spacing matches the data. INT4 with a good per-channel scale and clipping strategy is often competitive and sometimes better on specific layers, so this is not a blanket win. The reason FP4 has gained traction is less about accuracy and more about hardware: the FP4 tensor cores shipping on recent NVIDIA architectures accept the floating-point format directly, so you get the throughput without a conversion step. INT4 on those same cores may not map as cleanly. If you are choosing a quantisation format on a workload where the numeric spacing is the deciding factor, the underlying question is where the cost sits — the same question we work through in what “computationally expensive” means in an inference path. Format choice without a cost map is guesswork. What are MXFP4 and NVFP4, and why does block-level scaling make FP4 usable? Both are microscaling formats. Instead of one scale factor for a whole tensor, they attach a shared scale to a small block of consecutive values — typically 32 elements for MXFP4, with NVFP4 using a smaller block and a higher-precision scale. Each block gets its own magnitude, so an outlier in one block no longer flattens the values in every other block. That single design decision is what turns FP4 from a research curiosity into something you can serve. A per-tensor scale has to accommodate the largest weight anywhere in the tensor; a per-block scale only has to accommodate the largest weight in its 32-element neighbourhood. The effective precision within each block goes up dramatically because the scale is local. MXFP4 — Open Compute Project microscaling standard. Block size of 32, a shared power-of-two (E8M0) scale per block. Broad, vendor-neutral. NVFP4 — NVIDIA’s variant. Smaller blocks and a higher-fidelity FP8-style scale factor, which tends to preserve accuracy better at the cost of slightly more scale overhead. The trade-off between them is precision versus metadata: smaller blocks and richer scales recover more accuracy but add bytes and bookkeeping. In our experience reviewing quantisation setups, the block format is the first thing worth checking — a model that “fails at FP4” has often just been quantised with too coarse a scaling scheme, not hit a fundamental limit of four bits. This is an observed pattern across the configurations we have looked at, not a benchmarked failure rate. When does FP4 actually reduce inference latency, and when does it only save memory? Here is the distinction that the naive 2x story erases. Memory savings and latency savings are different outcomes with different preconditions. FP4 always saves memory — weights take roughly half the bytes of FP8. That lets you fit a larger model on a given GPU, or fit more concurrent requests, or hold a longer KV cache. Real value, but it is a capacity win, not automatically a speed win. FP4 reduces latency only when one of two things is true: The stage is memory-bandwidth bound. LLM decode — generating tokens one at a time — is dominated by reading weights from HBM, not by arithmetic. Halving the bytes read per token directly cuts the time per token. This is where FP4 earns its reputation. The hardware has native FP4 tensor-core support and the stage is compute bound. On silicon with FP4 tensor cores, compute-heavy stages like long-context prefill can run the matmuls at higher effective throughput. Without native support, the values get upconverted to a wider type before the math, so you pay for the memory saving but get none of the compute speedup. Where FP4 moves latency vs where it only saves memory Stage / condition Memory saved Latency moved Why LLM decode (token generation) Yes Yes Bandwidth-bound; fewer bytes per weight read per token Long-context prefill, native FP4 HW Yes Yes Compute-bound matmuls run on FP4 tensor cores Long-context prefill, no native FP4 Yes No Values upconvert before math; compute cost unchanged Small batch, already compute-light Yes Marginal Little bandwidth pressure to relieve Accuracy-sensitive layer forced to FP4 Yes Yes, but Speed gained, accuracy lost — net negative if it breaks the task Evidence class for this table: observed-pattern from quantisation reviews and the published behaviour of memory-bound decode; not a single named benchmark. The operational point: if your latency lives in a compute-bound stage and your hardware upconverts FP4, you will save memory and see the same latency. That is the exact trap the parent lever-thinking mistake sets. The way out is to profile where the time goes before you quantise — which is what a GPU performance audit is for, and why the DGX Spark memory-bandwidth analysis is a useful companion read on how bandwidth limits actually show up. What accuracy trade-offs should I expect from FP4, and which layers are most sensitive? FP4 degrades accuracy in proportion to how much information a layer needs to preserve. Most transformer weight matrices tolerate 4-bit surprisingly well once block scaling is in place. The layers that do not tolerate it are the usual suspects. Attention and the layers around it — where small numeric errors compound across long sequences. The final output projection / LM head — precision here bleeds straight into token probabilities. Anything with heavy activation outliers — some attention projections carry activations orders of magnitude larger than the norm, and FP4 has no headroom for them. The practiced answer is mixed precision: quantise the bulk of the weights to FP4, keep the sensitive layers in FP8 or higher. The measurable outcome you should hold onto is the before/after accuracy delta on your actual task, checked against the latency and cost-per-inference reduction. If task accuracy drops by an amount your product cannot absorb, the memory win does not matter. Quantify the trade — do not assume it. Which GPUs have native FP4 tensor-core support, and what happens without it? Native FP4 tensor-core support arrived with NVIDIA’s Blackwell generation; earlier architectures like Hopper (H100/H200) do FP8 natively but not FP4 at full tensor-core throughput. On hardware without native FP4, a runtime such as TensorRT-LLM or a similar serving stack will typically store weights in FP4 to save memory and upconvert to FP8 or BF16 before the matmul. That upconversion is the quiet failure mode. You configured FP4, your memory dropped, and you reasonably expected faster tokens — but the arithmetic never ran at 4 bits, so the compute-bound part of your workload is unchanged. Without checking the kernel path, the memory graph looks like success while the latency graph looks like nothing happened. Two things worth confirming before you commit: that your target GPU actually has FP4 tensor cores, and that your inference runtime dispatches to an FP4 kernel rather than upconverting. Both are checkable. Neither is safe to assume from a spec-sheet TFLOPS number. How do I decide between FP8 and FP4 for a given inference workload? Treat it as a measurement question, not a preference. FP8 is the conservative default: wider dynamic range, easier to keep accuracy, native on more hardware. FP4 is the aggressive option that pays off in a narrower set of conditions. FP8 vs FP4 quick decision rubric Reach for FP4 when: Your dominant cost is memory-bandwidth-bound decode, and Your GPU has native FP4 tensor cores (or you only need the capacity win), and Your task tolerates the measured accuracy delta with sensitive layers kept higher-precision. Stay on FP8 when: Your latency lives in compute-bound stages on hardware without native FP4, or The model is accuracy-sensitive and the FP4 delta breaches your target, or You have not yet profiled where latency is spent. The last bullet is the one that matters most. Choosing FP4 without measuring where latency is actually spent repeats the mistake the whole cost-per-inference discussion warns against: treating a lever as universal when it is conditional. FP4 is most relevant to LLM and generative serving, where 4-bit weight quantisation is a primary way to close the prototype-to-production cost gap — the same territory covered across the GPU engineering work we do. FAQ What does working with 4-bit floating-point involve in practice? FP4 spends four bits on a sign, exponent, and one or two mantissa bits, so it stores only a coarse bucket of magnitude rather than a precise value. In practice it is a storage-and-transport format wrapped around a scaling scheme: the four bits carry the shape and a separate scale factor restores the magnitude. Judge any FP4 claim by how the scaling is done, not the bit count alone. How does FP4 differ from INT4, and why does floating-point matter at 4 bits? INT4 spaces its 16 values evenly; FP4 spaces them logarithmically, closer together near zero, which matches weight distributions that pile up near zero with a long tail. The bigger practical reason FP4 has gained traction is hardware: recent NVIDIA FP4 tensor cores accept the floating-point format directly for throughput. INT4 remains competitive on specific layers, so this is not a blanket win. What are MXFP4 and NVFP4, and why does block-level scaling make FP4 usable? Both are microscaling formats that attach a shared scale to a small block of values instead of one scale per tensor. MXFP4 uses 32-element blocks with a power-of-two scale; NVFP4 uses smaller blocks and a higher-fidelity scale that preserves accuracy better. Block-level scaling stops a single outlier from flattening every other value, which is what turns FP4 from a research curiosity into something servable. When does FP4 actually reduce inference latency, and when does it only save memory? FP4 always saves memory — weights take roughly half the bytes of FP8 — but that is a capacity win, not automatically a speed win. It reduces latency only when the stage is memory-bandwidth bound (like LLM decode) or when the hardware has native FP4 tensor cores for compute-bound stages. Without native support the values upconvert before the math, so you save memory but latency is unchanged. What accuracy trade-offs should I expect from FP4, and which layers are most sensitive? Most transformer weight matrices tolerate FP4 well once block scaling is in place, but attention-adjacent layers, the final output projection, and layers with heavy activation outliers are sensitive. The standard practice is mixed precision: FP4 for the bulk, FP8 or higher for sensitive layers. Always measure the before/after accuracy delta on your actual task against the latency and cost reduction rather than assuming the trade is acceptable. Which GPUs have native FP4 tensor-core support, and what happens without it? Native FP4 tensor-core support arrived with NVIDIA’s Blackwell generation; Hopper (H100/H200) does FP8 natively but not full-throughput FP4. On hardware without native support, runtimes typically store weights in FP4 to save memory and upconvert to FP8 or BF16 before the matmul, so compute-bound stages see no speedup. Confirm both that the GPU has FP4 tensor cores and that the runtime dispatches an FP4 kernel rather than upconverting. How do I decide between FP8 and FP4 for a given inference workload? Treat it as a measurement question. Choose FP4 when decode is memory-bandwidth bound, the GPU has native FP4 tensor cores (or you only need the capacity win), and the task tolerates the measured accuracy delta. Stay on FP8 when latency lives in compute-bound stages without native FP4, the model is accuracy-sensitive, or you have not yet profiled where latency is spent. Before quantising anything, the harder question is not “FP8 or FP4” but “where does a single token’s latency actually go” — because the answer decides whether four bits buys you speed, capacity, or a quieter version of the same problem. That profiling step is exactly what a GPU performance audit is meant to settle.
