4-Bit Floating Point (FP4) Explained: How It Works in Practice

Someone reads that a model now runs in “4-bit floating point” and does the obvious arithmetic: four bits instead of sixteen, so the model must be four times cheaper. The number shrank, so the cost shrank with it. That intuition is where the trouble starts.

FP4 is not a smaller version of a big number. It is a specific format — a sign bit, a couple of exponent bits, and one or two mantissa bits — and the way those bits are split, plus the scaling scheme layered on top, decides whether a 4-bit model stays usable or falls apart. Teams that drop to FP4 without per-block scaling or calibration watch accuracy collapse. Teams that treat FP4 as a hardware-aware format with proper scaling keep quality within tolerance and collect the memory and throughput gains. Same four bits, opposite outcomes. The difference is entirely in the strategy.

This is a concept explainer, not an implementation recipe. The goal is to make clear what FP4 actually is, where its real constraints live, and why the interesting decisions are not about the bit-width at all.

What does 4-bit floating point actually store?

Start with the anatomy, because the whole misconception dissolves once you see it. A floating-point number splits its bits into three roles: a sign bit (positive or negative), exponent bits (the scale — how big or small the number is), and mantissa bits (the precision — how finely you can distinguish values at that scale). FP16 gives you 5 exponent bits and 10 mantissa bits. FP4 has four bits total to distribute across all three roles.

With one sign bit spent, you have three bits left. That is the entire budget for both dynamic range and precision. You cannot have both. If you spend two bits on the exponent and one on the mantissa, you get a format usually written E2M1: a modest range with almost no precision at each scale. If you spend three bits on the exponent and none on the mantissa (E3M0), you get wider range but the mantissa is effectively gone — the format can only represent powers of two and their negatives.

An E2M1 value can represent roughly sixteen distinct magnitudes. Not sixteen values in a range you choose freely — sixteen fixed points on a curve. That is the physical reality behind “4-bit.” The bits do not encode a shrunk version of your weight; they encode which of a handful of representable buckets your weight rounds to.

This is why the format choice matters more than the bit-count. E2M1 and E3M0 are both “FP4,” and they behave nothing alike. One trades range for a scrap of precision; the other throws away precision to cover a wider spread of values. Neither is universally correct — the right split depends on the distribution of the numbers you are actually storing.

Why FP4 needs scaling to be usable at all

Here is the part the naive read misses entirely. Sixteen representable buckets is not enough to encode a real tensor of neural-network weights, whose values span several orders of magnitude within a single layer. Left alone, FP4 would round most of a weight matrix to zero or saturate it at the top of its tiny range. Accuracy would not degrade gracefully; it would collapse.

The fix is scaling. Instead of storing weights directly in FP4, you store a block of weights in FP4 plus a separate, higher-precision scale factor for that block. The FP4 values become relative — “this weight is 0.75 of the block’s scale” — and the scale factor, often in FP8 or FP16, carries the absolute magnitude. This is block-wise or micro-scaling: partition the tensor into small groups (a block might be 16 or 32 elements), give each its own scale, and now each block’s FP4 buckets are positioned where that block’s values actually live.

Block scaling is why microscaling formats — the MX family that hardware vendors and the Open Compute Project have standardized around — are the practical form FP4 takes in production. A raw E2M1 element on its own is nearly useless. An E2M1 element inside a block with a shared FP8 scale is a viable way to store a weight. The bit-width headline hides the scale factors that make the format work.

Calibration is the second half. Choosing the right scale for each block, deciding which tensors can tolerate FP4 and which must stay in FP8, and handling outlier values that would otherwise wreck a block’s scale — this is quantization strategy, and it is where the accuracy-versus-cost trade-off is actually decided. In our experience helping teams reason about low-precision inference, the failures almost never come from the format itself; they come from quantizing uniformly, ignoring outliers, or skipping calibration and hoping the numbers survive. They usually do not.

How does FP4 compare to FP8 and FP16?

The reason anyone tolerates this complexity is the payoff, and the payoff is real when the hardware supports FP4 natively. The comparison below states the storage arithmetic directly — these are format definitions, not measured results — and frames the trade-offs qualitatively.

FP4 vs FP8 vs FP16: what changes

Format definitions and storage arithmetic; the “~4x smaller” figure is the raw element ratio, not a measured end-to-end footprint, since block scale factors add overhead.

Two claims are worth extracting cleanly. First: FP4 inference can roughly halve the memory footprint versus FP8 and cut it about 4x versus FP16, with corresponding throughput and tokens-per-second gains on hardware with native FP4 support. That is the ROI anchor — less memory per model instance, more tokens served per second, lower serving cost per token. Second: those gains are contingent on native hardware support. Emulating FP4 on hardware without FP4 arithmetic units gives you the memory savings but not the throughput; you spend cycles packing and unpacking bits.

The block scale factors are also not free. A block of 32 E2M1 elements at 4 bits each is 128 bits, plus one FP8 scale (8 bits) — so the effective cost is closer to 4.25 bits per element than a clean 4. The “4x smaller than FP16” headline is the element ratio; the real footprint is slightly higher once scales are counted. This is the kind of specification detail worth checking against a vendor’s published format before you build a memory budget on it — the same discipline that applies when reading CPU specifications for AI workloads, where the headline number rarely tells the whole story.

What is the accuracy trade-off, concretely?

The honest answer: it depends on what you are quantizing and how. FP4 applied naively to every tensor in a model — attention projections, MLP weights, embeddings, the lot — typically produces visible quality loss, because some tensors have distributions that FP4’s sixteen buckets simply cannot represent even with block scaling. FP4 applied selectively, with the sensitive tensors kept in FP8 and outliers handled, can land within a tolerance many production systems accept.

This is a genuine trade-off surface, not a solved problem. The decisions that move the needle are: block size (smaller blocks fit local distributions better but add scale-factor overhead), which layers get FP4 versus FP8, whether you use post-training quantization or quantization-aware training, and how you detect and isolate outlier channels. Frameworks and runtimes — the FP4 paths appearing in TensorRT-LLM, in NVIDIA’s Transformer Engine, and in the microscaling support landing across PyTorch and inference stacks — expose these knobs, but they do not choose the settings for you. The tooling is maturing quickly; the judgment about what precision each part of your model can survive is still yours.

Which hardware and use cases actually benefit?

Native FP4 arithmetic is a recent hardware capability — it shows up in the latest datacenter accelerator generations built specifically for low-precision inference, where the tensor cores can multiply FP4 operands directly. On that hardware, FP4 is a lever for exactly one class of problem: large-model inference where memory capacity and bandwidth, not compute, are the binding constraint. Fitting a bigger model on one accelerator, or more concurrent model instances on a fleet, or serving more tokens per second per dollar — these are the wins.

Where FP4 is not the answer: training (which needs the dynamic range and precision FP4 cannot supply for gradients), small models that already fit comfortably, and workloads where accuracy tolerance is razor-thin and every fraction of quality matters more than serving cost. Choosing precision is a workload-specific decision, and it interacts with the rest of your infrastructure — the same reasoning that governs how you manage a vector database’s memory footprint when retrieval and generation share a serving budget. If you are just beginning to map out where these choices live, our guides and how-to explainers are meant to give you the vocabulary before the tuning starts.

FAQ

What should you know about 4 bit floating point in practice?

FP4 stores a number in four bits split between a sign, a small exponent, and one or two mantissa bits — giving only about sixteen representable magnitudes per element. In practice it never works alone: weights are grouped into blocks that share a higher-precision scale factor, so each block’s small set of FP4 values is positioned where that block’s numbers actually live. It means roughly 4x less memory than FP16, contingent on scaling and calibration being done well.

What is the difference between FP4 formats like E2M1 and E3M0?

Both are 4-bit floating point, but the exponent/mantissa split differs. E2M1 spends two bits on the exponent and one on the mantissa, giving a modest range with a scrap of precision at each scale. E3M0 spends three bits on the exponent and none on the mantissa, widening the range but reducing the format to powers of two. Neither is universally better — the right choice depends on the distribution of values being stored.

How does FP4 compare to FP8 and FP16 for memory and throughput?

FP4 uses roughly half the memory of FP8 and about a quarter of FP16 at the element level, with corresponding throughput and tokens-per-second gains on hardware with native FP4 arithmetic. Those gains are contingent on native support; emulated FP4 gives the memory savings but not the speed. Block scale factors add a small overhead, so the effective cost is slightly above a clean 4 bits per element.

What is the accuracy trade-off when quantizing a model to FP4?

Applied uniformly to every tensor, FP4 usually causes visible quality loss because some distributions cannot fit its sixteen buckets even with scaling. Applied selectively — sensitive tensors kept in FP8, outliers isolated, blocks calibrated — it can land within a tolerance many production systems accept. It is a genuine trade-off surface, decided by block size, layer selection, and calibration method, not a solved problem.

Why do FP4 formats need scaling or block-wise calibration to stay usable?

Sixteen representable buckets cannot encode a real weight tensor whose values span several orders of magnitude; left alone, FP4 rounds most weights to zero or saturates them. Block-wise scaling stores small groups of FP4 values relative to a shared higher-precision scale factor, positioning each block’s buckets where its values actually are. Calibration then chooses the right scales and handles outliers, which is where accuracy is actually preserved.

Which hardware and use cases actually benefit from native FP4 support?

The recent datacenter accelerator generations with FP4 tensor cores benefit most, because they multiply FP4 operands directly rather than emulating them. The winning use case is large-model inference bound by memory capacity or bandwidth — fitting bigger models, more instances, or more tokens per second per dollar. FP4 is not suited to training, small models that already fit, or workloads with razor-thin accuracy tolerance.

The decision was never about four bits

If there is one thing to carry away, it is that “FP4” names a family of formats plus a scaling discipline, not a fixed cheaper number. The bit-width is the headline; the exponent/mantissa split, the block-scaling scheme, and the calibration strategy are the article. A team that understands this asks the right question — which parts of my model can survive four bits, and how do I scale the rest? — instead of the wrong one, which is how much does the number shrink? The savings are real, but they are earned by the strategy layered on top of the format, not handed over by the bit count alone.