Someone on the team reads that a new GPU supports FP4, flips the model from FP16 to 4-bit, and expects a clean 4x throughput win to fall out. It rarely does, and the reason it doesn’t is the whole point of understanding the format. FP4 — four-bit floating point — is not a switch you flip on an existing kernel. It is a change to how weights are laid out in memory, how scaling factors are computed, and how much accuracy budget you are willing to spend. Treat it as a kernel tweak and it fails silently. Treat it as an algorithmic decision and it becomes one of the largest cost levers available for memory-bound inference. That distinction is what this article is about: what FP4 actually encodes, why the decision to adopt it belongs in the same tier as re-architecting your data layout rather than in the same tier as tuning register allocation, and how to tell — before you commit — whether your workload will actually get faster. How Does 4-Bit Floating Point Work in Practice? Start with what four bits can hold. A floating-point number splits its bits between a sign, an exponent (dynamic range), and a mantissa (precision within that range). At 32 bits you have room to spare. At 4 bits you have almost none, so every bit allocation is a hard trade-off. The most common FP4 format is E2M1: one sign bit, two exponent bits, one mantissa bit. That gives you a grand total of sixteen representable values. Compared to an integer format like INT4, the floating-point layout spends bits on an exponent so it can represent both small and large magnitudes — useful when a tensor’s values span several orders of magnitude, which weight distributions in transformers routinely do. Variants shift the split (more exponent bits buy range at the cost of the single mantissa bit, and vice versa), but at four bits total there is no configuration that is precise and wide. You are choosing which failure you can tolerate. This is why FP4 never travels alone. A single global scale factor across a whole tensor would waste most of those sixteen values on a handful of outliers. Instead, FP4 deployments use block scaling: the tensor is partitioned into small blocks (say, 16 or 32 elements), and each block carries its own higher-precision scale factor. The 4-bit values encode the shape within a block; the block scale restores the magnitude. The scaling metadata is small relative to the payload, so the memory win survives — but it means an FP4 tensor is a two-part data structure, not a flat array of nibbles. Any kernel that reads it has to understand both parts. That two-part structure is the first hint that this is not a drop-in change. The layout in HBM is different, the memory access pattern is different, and the compute path has to unpack scales before it can do arithmetic. Why FP4 Is an Algorithmic Decision, Not a Kernel Tune Here is the trap teams fall into. They have an FP16 inference path that is running at, say, 40% of the GPU’s theoretical throughput. The instinct is to profile the kernel, chase occupancy, fuse a few operations, and claw back the gap. Sometimes that helps. But if the workload is memory-bound — if the GPU is waiting on HBM bandwidth to feed the tensor cores rather than waiting on the tensor cores themselves — no amount of register allocation recovers bandwidth that the format is spending. The ceiling is the format. FP4 attacks that ceiling directly. It roughly halves the memory footprint versus 8-bit and quarters it versus FP16 (observed-pattern — this follows from the bit width and holds when block-scale overhead is small relative to payload). On a bandwidth-bound path, moving a quarter as many bytes per token translates to proportionally more tokens per second and proportionally lower dollars-per-million-tokens. That is a different class of intervention than kernel micro-tuning, and it should be classified that way before anyone touches code. We treat format selection as sitting in the same tier as data-layout redesign because it changes the same three things at once: the bytes moved across the memory hierarchy, the block structure the compute reads, and the accuracy budget you are spending. That is the algorithmic tier. Contrast it with the micro-tuning tier — occupancy, launch configuration, kernel fusion — which reshuffles work that is already the right size. Our GPU performance audit’s optimization roadmap classifies interventions this way precisely because putting an FP4 migration in the micro-tuning bucket leads teams to under-scope it: they budget for a kernel change and get blindsided by a calibration effort. The related question of when raw memory capacity is the constraint versus bandwidth is covered in what a 128GB GPU means in practice, and the two failure modes are easy to confuse. Frameworks reflect this reality. In PyTorch, an FP4 path is not a dtype cast you sprinkle over the model — it involves quantization tooling that computes block scales from calibration data, and a runtime like TensorRT or a quantized kernel library that knows how to consume the packed format. NVIDIA’s Hopper and Blackwell tensor cores expose low-precision datapaths in hardware, but the software stack has to hand them correctly packed tensors and matching scale factors for any of that hardware capability to show up as throughput. The memory-bandwidth constraints that make FP4 worthwhile are the same constraints that determine whether the win materializes at all. When Does FP4 Actually Deliver a Speedup? The honest answer: only when the workload was memory-bound to begin with, and only when the model tolerates the quantization error with proper scaling. Both conditions have to hold. Miss the first and you have added calibration complexity for a compute-bound path that never touched its bandwidth ceiling. Miss the second and you have shipped a cheaper model that quietly gives worse answers. The following rubric captures the decision. It is a screen you run before committing engineering time, not a guarantee. FP4 Adoption Decision Rubric Condition Check If yes If no Memory-bound? Profile shows the path stalled on HBM bandwidth, not tensor-core occupancy FP4 can move the ceiling FP4 saves memory but not latency; reconsider Large weights dominate the footprint? Weight bytes » activation/KV-cache bytes in the hot path Quantizing weights to FP4 pays off Quantize the actual bottleneck (e.g. KV cache) instead Error-tolerant layers? Task metric survives quantization of the target layers Proceed with block scaling Keep sensitive layers at higher precision (mixed precision) Calibration data available? Representative inputs exist to compute block scales Calibrate, then measure Do not deploy FP4 uncalibrated — this is the silent-failure path Hardware + runtime support? GPU tensor cores and the serving stack consume the packed format Expect real throughput You are emulating FP4 in software; the win evaporates The single most common way FP4 fails is skipping the calibration row. An uncalibrated FP4 conversion still runs — it produces tokens, the pipeline is green, nothing throws — but the block scales are wrong, the effective precision collapses on outlier-heavy tensors, and task quality degrades in ways that a smoke test misses. This is the “fails silently” case, and it is why we insist on treating calibration as part of the format decision rather than a follow-up. How Do Block Scaling and Calibration Keep Accuracy in Budget? Block scaling is the mechanism that makes four bits usable; calibration is the process that makes block scaling correct. They work together, and neither alone is enough. Calibration runs representative inputs through the model and observes the actual distribution of values in each tensor block. From those observations it derives the per-block scale factors that map the real magnitudes onto the sixteen FP4 levels with the least distortion. The goal is a bounded accuracy delta — for example, holding task-metric regression under roughly 1% (observed-pattern; the acceptable threshold is workload-specific and should be set by the product, not by the format). Sensitive layers — often the first and last layers, or attention projections with heavy outliers — can be held at higher precision in a mixed-precision scheme while the bulk of the weights go to FP4. That is not a compromise on the technique; it is the technique working as intended. The practical discipline is: quantize, calibrate, measure the task metric, and only then read the throughput number. If the accuracy regression exceeds budget, you widen the precision on the offending layers and re-measure. The trade-off is explicit and numeric, not a vague hope that “4-bit is probably fine.” How Do I Measure Whether an FP4 Migration Actually Paid Off? Two numbers, weighed against each other. Not “is it faster” — that question is too loose to act on. The first number is cost: tokens per second per GPU, and its dollar translation, dollars per million tokens. This is a benchmark-class measurement when you take it on your own deployed serving stack under representative load — name the model, the batch size, the sequence lengths, and the hardware, and the figure is reproducible. Peak-throughput microbenchmarks do not count here; sustained throughput under your real request mix is what moves the bill. The second number is the accuracy delta: the change in your task metric between the FP16 baseline and the FP4 build, measured on a held-out evaluation set. A migration “paid off” only when the cost number improved and the accuracy delta stayed inside the budget you set before starting. Worked Example (Illustrative) Assume an FP16 serving path measured at 100 tokens/sec/GPU, bandwidth-bound at ~80% of HBM roofline, weights dominating the footprint. Converting the weight-heavy layers to FP4 with 32-element block scaling and calibration on 512 representative prompts. If the resulting build measures ~180 tokens/sec/GPU with a 0.6% drop on the task metric, the migration paid off: cost-per-token fell by roughly 45% (benchmark-class if taken on your own stack) against a sub-1% accuracy cost. If instead throughput barely moved, that is the signal the path was not bandwidth-bound in the way the profile suggested — go back to the first row of the rubric rather than pushing FP4 harder. The reason to name both numbers explicitly is that FP4 makes it easy to report one and hide the other. A throughput chart with no accuracy column is not a result; it is half a result. FAQ How does 4-bit floating point work in practice? FP4 stores a number in four bits split between sign, exponent, and mantissa, giving sixteen representable values. In practice it is never used alone: tensors are partitioned into small blocks, each carrying a higher-precision scale factor, so the 4-bit values encode shape within a block and the scale restores magnitude. That makes an FP4 tensor a two-part data structure that both memory layout and compute kernels must understand. What do the FP4 formats (E2M1 and variants) actually encode, and how do exponent and mantissa bits trade dynamic range against precision at 4 bits? E2M1 uses one sign bit, two exponent bits, and one mantissa bit. Exponent bits buy dynamic range — the ability to represent both small and large magnitudes — while mantissa bits buy precision within a range. At four bits total there is no split that is both wide and precise, so choosing a format means choosing which failure you can tolerate for your value distribution. Why is switching to FP4 an algorithmic decision about data layout and bandwidth rather than a kernel-tuning optimization? FP4 changes the bytes moved across the memory hierarchy, the block structure the compute reads, and the accuracy budget — all at once. That is the same class of change as a data-layout redesign, not the same class as tuning occupancy or fusing kernels. If a path is memory-bound, no register-allocation work recovers the bandwidth that the format itself is spending, so the format must be classified as an algorithmic intervention before any code changes. When does FP4 deliver real GPU speedups, and when is the workload not memory-bound enough for it to matter? FP4 delivers speedups when the path is stalled on HBM bandwidth and large weights dominate the footprint, because moving a quarter of the bytes translates to proportionally more tokens per second. If the path is compute-bound — waiting on tensor cores rather than memory — FP4 saves memory but does little for latency, and the added calibration effort is not justified. Profiling to confirm the bottleneck is the prerequisite check. How do block scaling and calibration keep FP4 accuracy loss within a bounded budget? Block scaling gives each small block its own scale factor so a few outliers don’t waste the sixteen available levels. Calibration runs representative inputs to derive those scales from the real value distribution, and sensitive layers can be held at higher precision in a mixed-precision scheme. The discipline is to quantize, calibrate, measure the task metric against a preset budget (for example, sub-1% regression), and widen precision on offending layers if the budget is exceeded. How do I measure whether an FP4 migration actually improved cost-per-token without an unacceptable accuracy regression? Measure two numbers and weigh them together: sustained tokens-per-second-per-GPU (and its dollars-per-million-tokens translation) on your own serving stack under representative load, and the change in your task metric versus the FP16 baseline on a held-out set. The migration paid off only when the cost number improved and the accuracy delta stayed inside the budget set before starting. A throughput result reported without the accuracy column is only half a result. The trade-off worth naming out loud FP4 is neither magic nor marketing. It is a bandwidth lever with a numeric price, and the price is paid in calibration effort and accuracy budget. The teams that get it wrong are the ones that classify it as a kernel change and skip the calibration step; the teams that get it right classify it as an algorithmic intervention, profile to confirm they are memory-bound, and report cost and accuracy as a pair. The harder question is not how to convert to FP4 — the tooling handles that. It is whether the layer you are about to quantize is the one that actually feeds your bottleneck. Get that classification right and the format choice becomes obvious; get it wrong and you will have spent a quarter of the bytes on the wrong tensor.
