Four bits is not a lot of room. When someone says “just quantize the model to FP4 and ship it,” the mental model is a free 4x memory cut with a rounding error you can ignore. That is not what happens on outlier-heavy layers, and the gap between the naive read and the working one is where most FP4 projects either save real money or quietly regress. The right starting question is not “how much memory do I save” but “what can four bits actually represent.” Everything downstream — accuracy retention, which layers you can cast, what hardware you need — follows from that answer. So start there. What matters most about 4-bit floating-point in practice? A floating-point number is split into three parts: a sign bit, a set of exponent bits that set the magnitude (the dynamic range), and a set of mantissa bits that set the precision within that magnitude. FP16 has one sign bit, five exponent bits, and ten mantissa bits — plenty of both. FP4 has to fit all three concerns into four bits total. The most common FP4 layout is E2M1: one sign bit, two exponent bits, one mantissa bit. That is the entire budget. Two exponent bits give you four exponent levels; one mantissa bit gives you a single fractional step within each. The result is a format that can represent only about 15 or 16 distinct values across its whole range. When you cast a weight tensor to FP4, every value in it snaps to one of those points. That is the mechanism, and it explains the whole discipline around FP4. You are not shaving a few bits off a fine-grained number — you are projecting a smooth distribution of weights onto a tiny, uneven grid. Whether that projection preserves model behavior depends entirely on how you manage the grid, which is where scaling comes in. What do the exponent and mantissa bits represent in an FP4 format like E2M1? It helps to be concrete about what E2M1 buys you. The two exponent bits control how far apart the representable magnitudes sit; because floating-point spacing is logarithmic, FP4 packs its values more tightly near zero and spreads them out as magnitude grows. That is genuinely useful for neural-network weights, which cluster near zero — a floating-point grid spends its scarce points where the data actually lives, unlike a uniform integer grid. The single mantissa bit is the constraint that bites. Within any given exponent band, you get exactly one intermediate step. So the precision inside a magnitude range is coarse in a way FP8 or INT8 never are. This is the core trade FP4 forces: you keep some of floating-point’s dynamic-range advantage over integers, but you pay for it with almost no mantissa precision. Anyone deciding whether FP4 fits their model needs to hold both halves of that trade in mind rather than treating “4-bit” as a single number. FP4 versus FP8, INT4, and FP16: what actually differs The formats people compare FP4 against are not interchangeable. They differ on three axes that matter for inference — dynamic range, within-range precision, and memory footprint — and the right choice depends on which of those your bottleneck actually is. Format Bit layout (typical) Dynamic range Precision Memory vs FP16 FP16 S1 E5 M10 Wide High 1x (baseline) FP8 (E4M3) S1 E4 M3 Moderate Moderate ~2x smaller INT4 4-bit integer None (uniform grid) Uniform, no range adaptation ~4x smaller FP4 (E2M1) S1 E2 M1 Narrow but present Very coarse ~4x smaller Two observations fall out of this table. First, FP4 and INT4 land at roughly the same memory footprint — about 4x smaller than FP16 (a spec-level consequence of the bit count, not a benchmark). The difference is representational: INT4 spaces its 16 levels uniformly, while FP4 spaces them logarithmically, which tends to fit weight distributions better but handles a fixed known range worse. Second, FP8 sits in a different tier entirely — roughly 2x smaller than FP16 with far more headroom — and is often the safer default when accuracy is fragile. The hardware-specs side of this comparison, and why raw format specs do not directly predict delivered throughput, is something we unpack in more detail in how hardware specs shape AI infrastructure performance. The practical read: reach for FP4 when memory or bandwidth is the binding constraint and the model is large enough to tolerate aggressive compression on its bulk layers. Reach for FP8 when you want a meaningful cut with a smaller accuracy risk. INT4 competes with FP4 specifically on models where a well-calibrated uniform grid holds up. Why does per-block or per-tensor scaling matter for FP4 accuracy? Here is where the naive approach breaks. If you cast an entire weight tensor to FP4 with one global scale factor, a single large outlier drags the scale up, and every ordinary weight in that tensor loses what little precision FP4 had to begin with. Outlier-heavy layers — and transformer attention and projection layers are notoriously outlier-heavy — collapse under uniform casting. The model does not fail loudly; it just gets worse in ways that pass a smoke test and fail in production. Scaling is the fix. Instead of one scale for the whole tensor, you split it into blocks — say 16 or 32 values each — and give every block its own scale factor, often stored in a higher-precision format like FP8 or even FP16. Now an outlier only distorts its own small block, and the rest of the tensor keeps its resolution. This is why formats like NVIDIA’s NVFP4 and the open MXFP4 standard are defined as block formats, not bare 4-bit numbers: the scale metadata is part of the format, not an optional add-on. The cost is honest and worth stating. Per-block scale factors add a few percent of memory back, so real-world FP4 does not hit a clean 4x — it lands close to it. That is the trade being made explicit rather than assumed. We treat the block-scaling overhead as a first-class part of the memory budget, not a footnote, because pretending FP4 is exactly 4x is how teams end up surprised at deployment. Where does FP4 make sense in AI inference, and where does it fail? FP4 earns its keep on large-model inference where the weights dominate memory and bandwidth, and where you can afford a calibration pass. Large language models are the clearest fit: their weight matrices are enormous, much of the network tolerates coarse weights, and the throughput gain on FP4-capable hardware is real. FP4 inference can cut model memory footprint by roughly 4x versus FP16 and materially raise tokens-per-second on hardware with native FP4 support (a directional, spec-level expectation — the delivered number depends on your kernels and your model, not the format alone). The measurable outcome is throughput and memory reduction at a bounded accuracy delta. It fails, or should be avoided, in a few recognizable places. Small models have no slack — coarse weights hurt them proportionally more. Layers that carry disproportionate signal (embeddings, final projection heads, normalization-adjacent paths) usually need to stay in FP8 or FP16; a good FP4 deployment is almost always mixed precision, not uniform. Numerically sensitive workloads — anything where a small output shift compounds — are poor candidates. And casting without calibration on a representative dataset is the single most common way to ship a silent regression. A short diagnostic before you commit to FP4 Run through these before quantizing anything to FP4: Is memory or bandwidth actually your bottleneck? If compute is the limit and memory is fine, FP4 buys you less than you think. Is the model large enough to absorb coarse weights? Billions of parameters, yes; a few hundred million, be cautious. Do you have a calibration set that reflects production inputs? Without it, you cannot measure the accuracy delta you are accepting. Have you identified the layers that must stay higher-precision? If your plan is uniform FP4 across every layer, it is not a plan yet. Does your target hardware have native FP4 execution? Emulated FP4 gives you the memory saving but not the throughput, which changes the ROI math. Can you measure quality end-to-end, not just per-layer error? Per-layer MSE looks fine right up until the full model does not. If you cannot answer most of these with a clear yes, FP8 is a lower-risk place to start. What hardware and software support is needed to actually benefit from FP4? The memory saving from FP4 is available almost anywhere — smaller weights are smaller weights. The throughput saving is not. To get faster inference rather than just a smaller model, you need silicon with native FP4 tensor execution. On NVIDIA’s side, the Blackwell generation introduced hardware FP4 support; on older architectures, FP4 weights typically get upcast for compute, so you keep the memory win but lose most of the speed win. That distinction changes the entire business case, so confirm it before you promise anyone a cost-per-inference reduction. Software matters just as much. You need format-aware kernels that can consume block-scaled FP4 without materializing everything back to FP16 — the work happens in inference stacks like TensorRT-LLM and in quantization tooling that understands block formats such as MXFP4 and NVFP4. Frameworks like PyTorch expose the tensor plumbing, but the accuracy-preserving path runs through calibration and mixed-precision tooling, not a single .to(fp4) call. Treating FP4 as a one-line cast is exactly the mental model that produces regressions. The broader point — that a format’s headline spec never predicts delivered performance on its own — is the theme running through much of how we think about running large models affordably, and it sits alongside the other infrastructure trade-offs we cover across our generative AI work. FAQ How does 4-bit floating-point work? FP4 splits four bits into a sign, exponent, and mantissa, giving only about 15–16 distinct representable values. Casting a weight tensor to FP4 snaps every value onto that tiny logarithmic grid. In practice it means real memory and bandwidth savings, but only if you manage the grid with scaling and mixed precision rather than casting uniformly. What do the exponent and mantissa bits represent in an FP4 format like E2M1? In E2M1, one sign bit sets direction, two exponent bits set magnitude (dynamic range), and one mantissa bit sets precision within a magnitude band. The exponent bits let FP4 pack values tightly near zero where weights cluster; the single mantissa bit means precision inside any range is very coarse. That is the trade FP4 forces — floating-point range at almost no mantissa precision. How does FP4 compare to FP8, INT4, and FP16 on range, precision, and memory? FP4 and INT4 both land around 4x smaller than FP16, but FP4 spaces its 16 levels logarithmically while INT4 spaces them uniformly. FP8 is roughly 2x smaller than FP16 with far more range and precision headroom, making it the safer default when accuracy is fragile. Choose FP4 when memory or bandwidth is the binding constraint and the model is large enough to absorb coarse weights. Why does per-block or per-tensor scaling matter for FP4 accuracy? A single global scale lets one outlier drag the scale up and strip precision from every ordinary weight, which collapses outlier-heavy layers. Block scaling gives each small block (e.g. 16–32 values) its own higher-precision scale factor, so an outlier only distorts its own block. That is why block formats like MXFP4 and NVFP4 bundle scale metadata into the format itself. Where does FP4 make sense in AI inference, and where does it fail? FP4 works well on large-model inference — LLMs especially — where weights dominate memory and much of the network tolerates coarse quantization. It fails on small models, on high-signal layers like embeddings and output heads that need higher precision, and on numerically sensitive workloads. A sound FP4 deployment is almost always mixed precision with calibration, never a uniform cast. What hardware and software support is needed to actually benefit from FP4? The memory saving is available anywhere, but the throughput saving requires native FP4 tensor execution — NVIDIA’s Blackwell generation introduced hardware FP4 support, while older architectures upcast FP4 for compute and lose most of the speed. On the software side you need format-aware kernels and quantization tooling that understand block formats, found in stacks like TensorRT-LLM and quantization libraries around MXFP4 and NVFP4. Before anyone signs off on FP4 as a cost-reduction lever, the sharper question is not “how much memory does it save” but “which layers can we afford to make this coarse, and how will we know if we got it wrong.” FP4 rewards teams that treat that as a measurement problem with calibration and mixed precision — and quietly punishes the ones who treat it as a one-line cast.
