Drop the model to 4 bits and you get a 4x memory and latency win for free. That is the pitch, and it is wrong often enough to sink an edge deployment. “4-bit” is not a single lever. FP4 and INT4 are different numeric formats with different accuracy behaviour, and hardware support for them varies sharply across the edge targets a computer vision system actually ships to. Pick a format without characterising per-layer sensitivity and you ship a model that either misses the accuracy target or — worse — runs no faster than it did at 8 bits because the silicon has no native 4-bit kernels. That gap between the promise and the practice is what this article is about. FP4 is a genuinely useful tool for fitting a CV model into a tight memory and latency envelope. It is also one facet of a broader edge trade-off, not a universal shortcut, and treating it as one is the fastest way to burn a deployment window. What matters most about 4-bit floating-point in practice? A 4-bit floating-point number spends its four bits on a sign, an exponent, and a mantissa — the same three-part structure as FP16 or FP32, just brutally compressed. The most common layout discussed today is E2M1: one sign bit, two exponent bits, one mantissa bit. Two exponent bits give you a small but real dynamic range; the single mantissa bit gives you almost no precision within any given exponent step. So FP4 does not represent many distinct values — on the order of a dozen after you account for signed pairs and special encodings — but the values it does represent are spread non-uniformly, clustering finely near zero and coarsely at the extremes. That non-uniform spacing is the whole point. Neural network weights and activations are not uniformly distributed. They pile up near zero with a long tail of large-magnitude outliers. A floating-point format that puts its representable values where the data actually lives wastes fewer of its scarce codes than a uniform grid would. The practical meaning: FP4 is a lossy compression of the numbers flowing through your network, chosen so the loss falls where the network is least sensitive to it. Whether that assumption holds for your network is an empirical question, not a given. How is FP4 different from INT4, and when does each format preserve accuracy better? INT4 also uses four bits, but it lays them out as a uniform integer grid scaled by a per-tensor or per-channel scale factor. Every representable value is equally spaced. This is efficient when the data it encodes is roughly uniform within its range, and it collapses when the data has wide dynamic range, because a single scale factor has to cover both the dense cluster near zero and the sparse outliers. Push the scale to capture the outliers and you quantise the dense region too coarsely; clamp the outliers and you lose them. FP4’s exponent bits buy it dynamic range that INT4 lacks. For layers whose activations span several orders of magnitude — attention-heavy blocks, layers immediately after a normalisation step, anything with heavy-tailed activations — FP4 preserves the shape of the distribution better and holds accuracy where INT4 would need aggressive per-channel calibration or would simply drift below the floor. INT4 is not the loser here, though. For weight tensors that are already well-behaved and tightly clustered, INT4 with good calibration is frequently as accurate as FP4 and enjoys far broader, more mature kernel support. The divergence point is the distribution, not the bit width. FP4 vs INT4 vs FP8: a decision surface for edge CV Axis FP8 (E4M3/E5M2) FP4 (E2M1) INT4 Bits per value 8 4 4 Dynamic range Wide Moderate Narrow (scale-dependent) Precision within range Good Very coarse Uniform, coarse Best for Sensitive layers, first/last layers Wide-distribution layers under tight budget Well-clustered weight tensors Calibration burden Low Moderate High (per-channel often required) Hardware breadth (edge, 2026) Broadening Narrow, newest silicon only Broad and mature Typical footprint vs FP16 ~2x smaller ~4x smaller ~4x smaller Ranges above are format properties and typical observed behaviour on CV workloads, not a benchmarked result on any single model. The right column of a real project is filled in by measuring your model, not by reading a table. Which edge hardware targets have native FP4 support, and what happens without it? This is where the “free 4x” claim most often falls apart. FP4 only delivers a latency win if the target hardware has kernels that operate on FP4 tensors directly. NVIDIA’s Blackwell generation introduced hardware FP4 tensor-core support and TensorRT paths to exercise it; NVIDIA’s published material positions FP4 as a first-class datatype on that architecture. That is genuine native support. Most edge silicon in the field today is not Blackwell. A Jetson Orin, an older desktop-class GPU, a mobile NPU, or an Arm-based edge box may have no FP4 execution path at all. On such a target the runtime does one of two things: it refuses the format, or it silently dequantises FP4 weights back up to FP16/INT8 before every matmul. In the second case you keep the memory saving from storing weights at 4 bits but pay a dequantisation tax on the compute — and you may end up slower than a clean INT8 model, because the runtime is doing extra work per layer. The model got smaller and no faster, which is the worst of both outcomes. The rule that survives contact with real deployments: confirm the executor — the specific hardware plus the specific runtime and its kernel library — supports your chosen format before you commit to it. The same model on the Nvidia DGX Spark and where it sits in the edge latency/cost trade-off behaves differently than it does on an older Orin, and the difference is which kernels exist, not which format the exporter accepted. An ML compiler and how model compilation enables cross-platform inference can paper over some of this by lowering to whatever the backend supports, but it cannot conjure an FP4 kernel that the silicon does not have. How much memory and latency does moving to FP4 actually save on a CV model? The memory arithmetic is honest and easy: going from FP16 to FP4 cuts weight storage by roughly 4x, and from FP8 by roughly 2x. For a model whose footprint is dominated by weights, that maps almost directly onto the on-device memory budget. This is the part of the FP4 story that behaves as advertised, and it is often the deciding factor — fitting the model into the device’s memory at all. Latency is the part that does not behave as advertised. On hardware with native FP4 kernels, moving from FP8 to FP4 can approach a further halving of the compute-bound portion of inference, because the tensor cores process more values per cycle. On hardware without those kernels, latency can stay flat or regress, per the dequantisation tax above. And even on capable silicon, many CV models are memory-bandwidth-bound rather than compute-bound, so the latency win from narrower compute is smaller than the raw arithmetic suggests. The right-sizing logic here overlaps with efficiency through right-sized inference: the smallest format that clears the accuracy floor is the efficient choice, not the smallest format available. The operationally relevant framing: FP4 changes model footprint by 2–4x reliably, and may change latency by a similar factor only when the executor supports it natively. On an edge CV workload with a 30ms budget, that distinction is the difference between clearing the budget within the accuracy floor and shipping something that misses one of the two. Which layers tolerate FP4, and which should stay at higher precision? Uniform quantisation — every layer to the same bit width — is the naive move, and it is usually the one that misses the accuracy floor. Sensitivity is not evenly distributed across a network. Layers that generally tolerate FP4 well: Mid-network convolutional and linear layers with well-behaved, near-zero-centred weight distributions. The bulk of a backbone’s feature-extraction stack, where redundancy absorbs quantisation noise. Layers that usually want higher precision (FP8 or FP16): The first layer, which sees raw input statistics that are far from the ideal quantisation distribution. The final classification or detection head, where quantisation error maps directly onto the output the accuracy metric measures. Layers immediately following normalisation, and attention blocks with heavy-tailed activations — the wide-distribution cases where INT4 fails and even FP4 needs care. This mixed-precision strategy is why per-layer sensitivity analysis matters. You quantise aggressively where the network is robust and hold precision where it is fragile. The related explainer, 4-Bit Floating Point (FP4): What It Means for CV Model Precision, goes deeper on the numeric-precision side of this trade-off; the point that carries here is that “the model is FP4” is almost never literally true in a well-tuned deployment. Parts of it are; the sensitive parts are not. How do I validate that FP4 keeps my model above its accuracy floor? Validation is not optional, and it is not “eyeball a few images.” Before FP4 goes to an edge target, run a real evaluation loop: Define the accuracy floor first. Name the minimum acceptable metric — mAP, top-1, recall at a fixed precision — before quantising. The floor is a deployment requirement, not a number you discover after the fact. Establish the FP16 baseline on your held-out evaluation set so you know the ceiling and the gap you are spending. Run per-layer sensitivity analysis — quantise one layer at a time, measure the metric drop, and rank layers by fragility. Build a mixed-precision plan from that ranking: FP4 where the drop is negligible, FP8/FP16 where it is not. Measure on the target executor, not a proxy. Accuracy after quantisation can differ between the exporter’s simulated path and the actual on-device kernels. Latency almost always does. Re-measure latency and memory on the device to confirm you actually gained what the arithmetic promised, and that no dequantisation tax crept in. Skip the sensitivity step and you are gambling that uniform FP4 lands above your floor. Sometimes it does. When it does not, the failure shows up as a quiet accuracy regression in production that nobody attributes to quantisation until much later. Scoping exactly this kind of trade-off is a large part of what a computer vision consultant does when scoping edge deployment, because the answer is always model-and-hardware specific. FP4 is one lever in the wider edge envelope of latency, accuracy, and power that we characterise across our computer vision engineering work. It sits alongside format choice, batching, resolution, and hardware selection — none of which resolves in isolation. FAQ How does 4-bit floating-point work? FP4 packs a sign, exponent, and mantissa into four bits — the common E2M1 layout uses two exponent bits and one mantissa bit. That gives it a small dynamic range with very coarse precision, and it spreads its handful of representable values non-uniformly so they cluster near zero where neural-network weights and activations actually live. In practice it is a lossy compression of the numbers flowing through your network, useful only when the loss falls where your model is insensitive. How is FP4 different from INT4, and when does each format preserve accuracy better? INT4 uses a uniform integer grid with a scale factor; FP4 uses floating-point spacing with exponent bits that buy dynamic range INT4 lacks. FP4 preserves accuracy better for layers with wide, heavy-tailed distributions — attention blocks, post-normalisation layers — where a single INT4 scale factor cannot cover both the dense cluster and the outliers. INT4 with good per-channel calibration is often equally accurate on well-clustered weight tensors and has broader kernel support. Which edge hardware targets have native FP4 support, and what happens without it? NVIDIA’s Blackwell generation added hardware FP4 tensor-core support with TensorRT paths, per NVIDIA’s published material. Most edge silicon in the field — older Jetson Orin, mobile NPUs, Arm edge boxes — has no FP4 execution path. Without native kernels the runtime either rejects the format or silently dequantises FP4 back to FP16/INT8 before each matmul, keeping the memory saving but paying a dequantisation tax that can leave the model slower than a clean INT8 build. How much memory and latency does moving from FP8 or FP16 to FP4 actually save on a CV model? Memory is honest: roughly 4x smaller than FP16 and 2x smaller than FP8 for weight-dominated models. Latency is not — it approaches a further halving of compute-bound inference only on hardware with native FP4 kernels, and can stay flat or regress without them. Because many CV models are memory-bandwidth-bound, the real latency gain is often smaller than the raw arithmetic implies. Which layers of a computer vision model tolerate FP4 quantisation and which should stay at higher precision? Mid-network convolutional and linear layers with near-zero-centred distributions usually tolerate FP4 well, as does the bulk of a redundant feature-extraction backbone. The first layer, the final detection or classification head, post-normalisation layers, and attention blocks with heavy-tailed activations generally want FP8 or FP16. This is why a well-tuned deployment is mixed-precision, not uniformly FP4. How do I validate that FP4 quantisation keeps my model above its accuracy floor before deploying to the edge? Define the minimum acceptable metric first, establish an FP16 baseline, then run per-layer sensitivity analysis to rank layers by fragility and build a mixed-precision plan from it. Measure both accuracy and latency on the actual target executor rather than the exporter’s simulated path, since on-device kernels can behave differently. Skipping the sensitivity step gambles that uniform FP4 clears your floor, and the failure often surfaces only as a quiet production regression later. Before you sign off on this approach FP4 earns its place when a model is too large or too slow for the device by a margin that 8-bit cannot close, and when the target executor has native 4-bit kernels to exercise it. Outside those conditions it is either unnecessary or actively counterproductive. The discipline that separates a working FP4 deployment from a broken one is not the format choice itself — it is characterising per-layer sensitivity and confirming the hardware-plus-runtime executor before committing. FP4 viability is one input to whether a model fits the chosen edge hardware’s latency and memory budget at all, which is exactly the question a production CV readiness assessment exists to answer before code ships to a device you cannot easily patch.
