Post-training quantization (PTQ) takes a model that has already been trained and converts its weights and activations from floating point to a lower-precision format such as 8-bit integers, with no retraining. Its error is shaped by rounding, clipping, the calibration data and the quantization scheme.
LynxBenchAI’s Personal Edition runs the same prepared INT8, FP8 and FP16 artefacts on your own hardware with stock PyTorch, so you can see how each precision behaves there.
For LLMs specifically, see LLM quantization: why memory bandwidth wins and where accuracy breaks. This page stays on the post-training route and on what it takes to verify it.
How Post-Training Quantization Converts a Trained Model Without Retraining
In PTQ the quantization parameters are calibrated offline. The tool processes the trained weights, plus the activations produced by running inference on a sample dataset, and no further training takes place. Uniform quantization happens in two steps: choose the range of real values to represent, clamping anything outside it, then map each value to the nearest integer the bit width allows.
Google’s LiteRT documentation writes the 8-bit mapping as real_value = (int8_value − zero_point) × scale. The scale is the quantizer’s step size. The zero-point is an integer that makes real zero quantize exactly, which keeps zero padding and ReLU from adding error. Weights are constants, so their ranges can be read straight off the tensors. Activations only exist when the model runs, so the converter needs a representative dataset to calibrate them.
NVIDIA’s TensorRT documents the same process as a concrete build. It builds a 32-bit engine, runs it on the calibration set, records a histogram of activation values for each tensor, turns those histograms into a calibration table of scales, and builds the INT8 engine from that table. Quantization-aware training goes the other way and inserts simulated quantize-dequantize operations into training so the network adapts to them. The types of quantization in machine learning have their own page.
Rounding Error and Clipping Error Trade Against Each Other
Range setting trades two errors against each other. Values outside the grid limits are clipped, which causes clipping error. Widening the range to reduce clipping means a larger scale, and the rounding error, which lies in the interval [−s/2, s/2], grows with it. Signal processing calls the same pair overload distortion and granular distortion, and quantizer design has long been about balancing the two. NVIDIA’s TensorRT guide names them discretization error and truncation error.
Inside the range, rounding error never exceeds half a step, and under the high-rate approximation it is close to uniform within each bin, with variance Δ²/12. Clipping error has no such ceiling: it grows with how far an outlier sits beyond the limit. So range setting is a trade-off between clipping and rounding error. It is never a choice of whether to have any.
Error Sources at a Glance
| Error source | What bounds it | What makes it worse | Main lever |
|---|---|---|---|
| Rounding (granular) | Half the step size per value | A wide range set by outliers | Tighter range, finer granularity |
| Clipping (overload) | The distance of clipped values beyond the limit | A range set too tight for the tails | Range-setting method: max, MSE, entropy, percentile |
| Bias shift | Correctable from calibration data | Depth-wise layers with few elements per channel | Bias correction |
| Scheme mismatch | The match between grid and distribution | A symmetric grid on asymmetric data | Asymmetric activations, per-channel weights |
How Much Accuracy Does Post-Training Quantization Lose?
At 8 bits the quantization accuracy loss is small in several published evaluations. In a 2021 white paper from Qualcomm, 8-bit weights and activations (W8A8) stayed within 0.7% of floating-point accuracy for every model tested. Wu et al. (NVIDIA, 2020) report an 8-bit workflow that kept every network studied within 1% of its floating-point baseline, MobileNets and BERT-large included. Krishnamoorthi (2018) found per-channel weights and per-layer activations at 8 bits within 2% across a wide range of CNNs. For LLMs, Kurtic et al. found FP8 effectively lossless and well-tuned INT8 within 1 to 3%.
Those averages hide where the loss concentrates:
- Depth-wise separable layers. Biased error is more pronounced in layers with only a few elements per output channel (usually 9 for a 3×3 kernel), and the main contributor is often clipping. In the same white paper, 4-bit weights cost EfficientNet lite about 4% with per-tensor quantization, and per-channel quantization brought the gap under 1.5%.
- Outliers. SmoothQuant’s authors measured activation outliers about 100× larger than most values. Under per-tensor quantization they leave non-outlier channels only 2 to 3 effective levels.
- Bit width below 8. Traditional PTQ works well up to 8 bits and struggles below that. A 2026 analysis explains why: when the quantization grid is comparable to the width of the low-loss basin, local PTQ objectives can pick a high-loss point even when a low-loss one sits nearby.
- Training history. One 2025 study tied quantization degradation to training dynamics and found it can be acted on with existing tools.
The repairs reported are targeted rather than wholesale. Wu et al. recovered all but BERT to within 1% of FP32 by skipping quantization on a few of the most sensitive layers. The Qualcomm paper kept the few BERT-base activation tensors with extreme ranges in 16 bit. That is the practical meaning of controlled approximation: the error has addresses, and you can leave the worst addresses unquantized. Kurtic et al. left KV cache quantization as an open question outside their weight and activation study.
Does the Calibration Set or the Calibration Method Decide the INT8 Result?
Both matter, and NVIDIA’s PTQ workflow includes a sweep of activation calibration methods. Wu et al. evaluate three methods: max (the largest absolute value seen), entropy (minimising KL divergence, TensorRT’s default) and percentile (for example, clipping the top 1% of magnitudes). Entropy and percentile clip outliers to buy resolution for inliers, which is the trade-off from the previous section applied at calibration time. Their finding is blunt: the best results came from entropy or the 99.99% and 99.999% percentiles, but no single calibration was best for every network. Max calibration produced particularly large drops on Inception v4, EfficientNets and Transformer.
The Torch-TensorRT documentation shows the same choice at the API level. Its calibrator defaults to IInt8EntropyCalibrator2 and describes the MinMax calibrator as better suited to NLP tasks. Range choice can also be per function: Wu et al. clipped GELU outputs to 10 and then got their best BERT result with plain max calibration.
The calibration set does not need to be large. TensorRT reports about 500 images as sufficient for ImageNet classifiers, LiteRT suggests around 100 to 500 samples, and MLPerf Mobile fixes its calibration sets at typically 500 samples. What the set represents matters too. A 2025 training-dynamics study notes that a model calibrated on web data may do better on web-style tasks, and Kurtic et al. found that random-token calibration degraded accuracy in their W4A16 models.
One property is easy to overlook. TensorRT calibration is deterministic for the same inputs, in the same order, on the same device, but the cache is not guaranteed to be bitwise identical across devices, batch sizes or calibration inputs. So a calibration cache is not guaranteed to carry over to another device, batch size or calibration input.
Symmetric or Zero-Point Quantization at the Same Bit Width
Symmetric quantization pins the zero point so the range is symmetric about zero. Asymmetric quantization lets the zero point move with the data’s minimum and maximum. At the same bit width, asymmetric quantization is more expressive because it has an extra offset parameter, but it can add computational overhead. For common accelerators, the Qualcomm white paper recommends symmetric quantizers for weights and asymmetric quantizers for activations. A 2025 evaluation of LLM quantization techniques reached the same split empirically: asymmetric quantization helped activations significantly and weights only slightly, because activations tend to be asymmetric and weights roughly symmetric.
The size of the penalty can be stated. A 2026 paper proves that a symmetric grid inflates quantization noise variance by γ² relative to an asymmetric one, where γ measures how off-centre the distribution is. At γ = 1.2, a mild asymmetry, that is 1.2² = 1.44, or 44% more per-layer output error. The same authors note that the gap matters most when the target is distributional fidelity, and that task benchmarks can absorb it. For ReLU outputs, which are one-tailed, unsigned symmetric quantization is the natural fit.
What the runtime does can differ from what the paper recommends. TensorRT uses symmetric quantization with per-channel weight scales, and when a configured range is asymmetric it uses a larger dynamic range than configured, which can increase rounding error. Check which scheme your target stack executes before you credit a result to the scheme you designed.
Why an INT8 Model Can Come Out Slower than FP16 on a GPU
In TensorRT, the placement of quantize/dequantize (Q/DQ) layers affects both performance and accuracy. NVIDIA’s guidance is to quantize all inputs of weighted operations (convolution, transposed convolution, GEMM) so bandwidth falls and INT8 compute can run. Granularity is constrained as well: integer matrix multiplication needs per-tensor or per-row activation scales and per-tensor or per-column weight scales, and for activations only per-tensor is practical for performance. Per-channel weights are not supported on all commercial hardware.
How scales are computed matters for GPU performance too. In SmoothQuant’s measurements, static quantization runs significantly faster than dynamic quantization because step sizes no longer need computing at runtime, and coarser granularity gives lower latency. LiteRT’s dynamic-range operators still store outputs in floating point, so they gain less than full fixed-point computation. For networks with implicit quantization, TensorRT forces some layers near the output to run in FP32 even when INT8 kernels exist. For LLMs the phase decides which format pays off: prefill is typically compute-bound and decode memory-bound, so weight-only quantization mainly speeds decode while weight-and-activation quantization helps prefill. MLPerf Mobile’s authors add that not every task benefits from INT8, and that NLP needed FP16 to be useful in real deployments. Where a quantization decision meets the rest of the serving path, porting inference across runtime and hardware is the engineering side of the same question.
Worked Example: INT8 Against FP16 in the 26Q3 Results
These are LynxBenchAI scores on DistilBERT inference in the 26Q3 results (higher is better):
| Executor (26Q3) | FP16 | INT8 | FP8 |
|---|---|---|---|
| NVIDIA H100 80GB HBM3 | 2611.7 | 1090.3 | 1362.7 |
| AMD Instinct MI300X | 2816.1 | 946.4 | 1524.8 |
| NVIDIA GeForce RTX 4090 | 1069.7 | 561.2 | 673.1 |
| NVIDIA A100-SXM4-80GB | 1267.4 | 614.0 | 0.0 (no result) |
| AMD Ryzen 5 9600X (CPU) | 2.2 | 2.7 | 1.8 |
On the Ryzen CPU the order flips: INT8 scores above FP16 on DistilBERT, and on MobileNetV3 it scores 13.8 against 2.6. Two cautions apply. Each test is one timed window after a discarded warm-up, and scores compare only within 26Q3. The score also weights throughput by the memory a run actually moves, so a lower-precision run has to earn its speed-up rather than being credited for being cheap. The 26Q3 results do not say why INT8 trails on these GPUs. A mixed INT8/FP16 graph with conversion boundaries fits the kernel and Q/DQ conditions above, but that reading is our judgement, not something the release measures. What the data does show is that the precision label alone does not predict the outcome on a given executor.
Quantized Benchmark Results Need Their Rules Attached
A quantized number means nothing until you know what was prepared, under whose rules, and what threshold it had to clear. MLPerf makes this explicit. MLPerf Mobile chose its minimum-accuracy thresholds based on what post-training quantization can achieve without training data. Submitters may calibrate only on the approved calibration set, and MLPerf Inference prohibits retraining and pruning so that results stay comparable.
LynxBenchAI applies the same discipline to its own catalogue. The benchmarked models are prepared once, before any target device is known, and the identical prepared artefacts go to every vendor’s hardware. Where two vendors’ kernels disagree about what an approximation may do, the stricter constraint is applied to all of them rather than settled vendor by vendor. A run counts only if its output clears a correctness threshold set per test type and precision. A run that misses it is recorded as a zero rather than reported as a fast result. A workload that fails or times out is also recorded as a zero rather than dropped.
That gate is a correctness check for the benchmark. It does not evaluate your quantization scheme. A 26Q3 run executes the catalogue fixed by that release. What it does show is that the bookkeeping around an approximation can be written down rather than assumed. Deciding what a performance benchmark has to measure before its number settles a cost question is a separate decision.
Checks Before a Post-Training Quantized Model Ships
Run these in order.
- Sanity-check the pipeline. Set the quantized model to 32-bit weights and activations, or bypass quantization, and confirm it matches the floating-point model.
- Separate weights from activations. Raise one to a higher bit width while keeping the other low, to see which one drives the drop.
- Sweep activation calibrations. Try max, entropy and the 99.99% and 99.999% percentiles. If none reaches the target, move to partial quantization or QAT.
- Find and exempt sensitive layers. Keeping a few of the most sensitive layers unquantized often recovers accuracy.
- Confirm the target supports your scheme. Per-channel weight quantization in particular is not available on every device.
- Revalidate after removing Q/DQ nodes. Removing them changes the deployed numerics, so accuracy validation has to be rerun against the original acceptance criteria before shipping.
- Measure on the device and precision you will deploy. In the 26Q3 results, the same precision label ranks differently across executors.
The Threshold Is the Decision, Not the Format
“Is INT8 safe for this model?” depends on an application-specific acceptance threshold and the executor it applies to. With both named, PTQ becomes an engineering question with measurable parts: which layers to exempt, which calibration to use, which grid to put on which tensor. The 26Q3 data adds the less comfortable part. Even when the accuracy gate passes, the expected speed-up can fail to appear on a given device, so verifying accuracy and measuring speed are two separate checks. Treating precision as a design parameter rather than a quality compromise means writing both down.
FAQ
What Is INT8 vs FP8?
INT8 maps values to evenly spaced integers through a scale and zero-point. FP8 E4M3, as TensorRT defines it, rounds to the nearest representable floating-point value in the range [−448, 448] after scaling. In Kurtic et al.’s LLM study, FP8 weights and activations were effectively lossless and well-tuned INT8 lost 1 to 3%.
What Is Uniform Quantization?
Uniform quantization uses equally spaced levels: choose a range, clamp values outside it, and round each value to the nearest level. The minimum-distortion quantizer is uniform when the input is uniformly distributed. For other distributions, the minimum-distortion quantizer may not be uniform.