LLM quantization speeds up inference mainly because token-by-token decode is limited by memory bandwidth, not arithmetic: each generated token streams the model’s weights from memory again, so holding them in 8 or 4 bits instead of 16 cuts the bytes moved per token. The gain is conditional. Whether it reaches your workload depends on the quantization scheme, the calibration data, the kernel and the hardware. Accuracy holds well at 8-bit and mostly holds at 4-bit weights on short-context tasks. It breaks with 4-bit activations, and 4-bit methods can lose heavily on long contexts. The useful way to treat quantization is as a controlled approximation whose error can be bounded, measured and reported. It is neither free speed nor model damage.
Why LLM quantization pays off in decode, and which bytes it removes
LLM inference has two phases. Prefill processes the prompt in parallel and is mostly compute-bound. Decode generates one token at a time at small batch sizes and is generally memory-bound. In decode, each large weight matrix is loaded and then used only once for a single token, so the arithmetic units sit underused while memory traffic sets the pace. The standard account puts a floor on a batch-1 decode step of (weight bytes + active KV-cache bytes) divided by peak memory bandwidth.
What you quantize decides which part of that sum shrinks. Li et al. (2024) separate three kinds:
- Weight-only quantization speeds up the memory-bound matrix-vector operators of decode.
- Weight-activation quantization lets the GPU use low-precision Tensor Cores for the compute-bound matrix multiplies of prefill.
- KV-cache quantization relieves memory pressure on long texts and large batches.
Quantizing activations alone does little for decode speed, because the bottleneck there is reading weights. The KV cache can still dominate. The SmoothQuant authors (Xiao et al., 2023) cite a configuration with batch size 512 and context length 2048 where the KV cache totals 3 TB, three times the size of the model weights.
Worked example: what a scale factor costs in bytes
Each quantized group also stores metadata, and that metadata uses bandwidth too. The MixLLM authors give the effective footprint at a group size of 128. Symmetric W8A8 costs 1 + 2/G bytes per weight element. Asymmetric W4A8 costs 0.5 + 2.5/G bytes, because it stores a 4-bit zero point alongside the 16-bit scale. With G = 128 that works out to about 1.016 bytes for W8A8 against about 0.520 bytes for W4A8. That is the saving which pushes up attainable throughput in bandwidth-limited regimes such as small-batch decoding.
Fewer bits does not mean lower latency
The bandwidth saving only matters if the kernel can turn it into time saved. Benazir and Lin (2025) profiled 26 llama.cpp quantization variants on Apple Silicon. A 2.62-bits-per-weight format (IQ2_M) ran at nearly the same latency as a 6.56-bits-per-weight format (Q6_K). Q6_K ran 1.22 times slower than Q8_0 even though it moves less data, because unpacking sub-byte values takes extensive bit manipulation, while INT8 values drop straight into SIMD lanes. Their conclusion is mechanical: when dequantization overhead grows faster than the bandwidth it saves, latency rises even though the tensor got smaller.
Kernel choice matters on NVIDIA hardware too. A 2026 study of batch-1 decode with Qwen-2.5-7B on an L4 measured 62.32 ms per step in bf16. AutoAWQ with Marlin kernels brought that down to 45.24 ms (1.38 times faster). GPTQ with ExLlamaV2’s Ada-tuned int4 kernels brought it to 17.36 ms (3.59 times faster). The authors’ reading is that the lever was the kernel implementation, not the bit width, since Marlin was tuned for Ampere SM80 and not for Ada SM89. A survey by Gholami et al. (2021) makes the general point that quantization speed-ups depend on the hardware, including on-chip memory, bandwidth and cache hierarchy.
So a precision’s speed belongs to the executor (the device plus its software stack), not to the bit width. The split between kernel-level and algorithmic levers is covered in our piece on LLM inference optimization techniques, and the decision about whether to change runtime or hardware without changing the model is covered in porting AI inference to cut cost without a model swap.
The approximation itself, and what bounds its error
Uniform quantization maps a real value r to an integer through Q(r) = Int(r/S) − Z. S is a real-valued scale, Z is an integer zero point, and Int rounds, either to nearest or by truncation. For a range [x_min, x_max] mapped to B bits, the scale is (x_max − x_min)/(2^B − 1). Values outside the representable range are clamped.
That gives two sources of error with opposite pulls. NVIDIA’s TensorRT documentation calls them discretization error, which grows as each quantized step covers a wider range, and truncation error, which comes from clamping values at the limits of the range. Inside the range, rounding error is at most half a step. Under conditions Widrow, Kollár and Liu (1996) set out, it behaves like additive uniform noise with zero mean and mean square q²/12, where q is the step size. Calibration methods that clip outliers accept a large error on a few values in exchange for finer resolution on the many.
Zhang et al. derive the first quantitative error bounds for a deterministic post-training algorithm, OPTQ (the method behind GPTQ). Those bounds depend on properties of the calibration data matrix and on the regularisation parameter. They also note that smaller calibration error usually goes with lower perplexity and higher zero-shot accuracy in LLMs.
Quantization error also differs in kind from training error. Training error comes from data, model capacity and optimisation. Quantization error is a perturbation applied to parameters that have already converged, and it can push the model away from the point it reached in floating point. That perturbation is set by explicit choices of range, rounding, granularity and calibration, so it can be inspected and adjusted in a way generalisation error cannot. It does not stay local, though. In OmniQuant’s layer-wise optimisation, error in one layer can corrupt the supervision for the next and build up in a feedback loop. The broader argument for this framing is in our piece on post-training quantization as controlled approximation rather than model damage.
Same bit width, different error: symmetric, asymmetric, per-channel, group-wise
“INT4” is not one transformation. Quantization is a family of choices about what is quantized, with which calibration data, by which method, and into which target format. The table sets out the main scheme choices.
| Choice | What it changes in the error | What it costs |
|---|---|---|
| Symmetric (x_min = −x_max, zero point 0) | Wastes range on skewed data; symmetric quantization of unsigned activations effectively costs one bit | Cheapest integer arithmetic; NVIDIA recommends scale (symmetric) quantization for weights to maximise performance |
| Asymmetric (non-zero zero point) | Fits ranges that are not centred on zero; needs fewer bits for the same accuracy | Adds a term that depends on the quantized input and cannot be precomputed, which can erase the integer-math advantage |
| Per-tensor | One outlier sets the scale for everything; with outliers around 100 times larger than typical activations, non-outlier channels may get only 2 or 3 effective levels | Simplest kernels; preferred for activations for performance reasons |
| Per-channel (weights) | More accurate than per-tensor for weights | TensorRT recommends per-channel weights with per-tensor activations |
| Group-wise | Higher accuracy than whole-channel scales | More complex GPU kernels; dequantizing fine-grained groups costs about the same as coarse ones |
LLMs add an activation problem. Once models grow beyond about 6.7B parameters, systematic large-magnitude outliers appear in activations and cause large quantization errors. Methods take different routes around them. LLM.int8() keeps outlier dimensions in 16-bit while more than 99.9% of values are multiplied in 8-bit. SmoothQuant moves the difficulty from activations into weights with an equivalent per-channel scaling. AWQ protects about 1% of salient weight channels, identified from activation statistics.
Weight-only post-training methods agree at 4 bits and diverge below. In a benchmark of post-training methods, all baselines performed comparably at 4-bit. They began to separate at 3-bit, and at 2-bit the salience-based AWQ lost its language ability on every LLM tested.
Calibration shapes the outcome more than its size suggests
Calibration chooses the range for each tensor. Wu et al. (NVIDIA, 2020) compare three methods: max (the largest absolute value seen), entropy (minimising KL divergence, which is TensorRT’s default), and percentile clipping. Across their networks, max calibration gave inconsistent quality, 99.9% percentile clipped too aggressively, and no single method was best for every network.
The data matters as well as the method. Williams and Aletras found that downstream accuracy varied with the choice of calibration set, against the common assumption that post-training methods are robust to it. For quantization the spread was smaller than for pruning, for example a 1.6% range for GPTQ on OPT-6.7B. Kurtic et al. (2024) report that random-token calibration degraded W4A16 accuracy, so they calibrated on curated instruction data instead.
One source points the other way. The MixLLM authors found that changing the calibration dataset did not change accuracy for their salience search. Both findings can hold. Sensitivity depends on the method and on how far the bit width is pushed, so robustness shown for one method is not a licence to ignore calibration in another. Since calibration data affects results, Williams and Aletras recommend publishing it for reproducibility. For long-context degradation, the authors of a 2025 long-context study propose long-context and multilingual calibration samples as a mitigation they did not test themselves.
PTQ and QAT put the error in different places
Post-training quantization (PTQ) calibrates a pre-trained model on a small dataset to compute clipping ranges and scales, then quantizes it. Its overhead is very low, it works with little or unlabelled data, and it usually loses more accuracy than QAT at low precision. Nagel et al. (Qualcomm AI Research, 2021) report that PTQ is usually enough for 8-bit with close to floating-point accuracy, and that quantization-aware training (QAT) needs fine-tuning and labelled data but makes lower bit widths competitive. Compensation-based PTQ such as GPTQ also corrects as it goes: the more error one weight column introduces, the more compensation is added to the weights not yet quantized.
QAT moves the error into training. It simulates quantization numerics while keeping weights in floating point (“fake quantizing”), and it uses straight-through estimators to pass gradients through the rounding, so the weights adapt to the quantization noise. The PyTorch team reports that QAT recovered up to 96% of the hellaswag accuracy degradation and 68% of the wikitext perplexity degradation that PTQ caused on Llama3. The price was fine-tuning about 34% slower than ordinary fine-tuning, plus around 2.35 GB of extra memory per GPU with activation checkpointing.
The practical order is escalation. NVIDIA’s recommended int8 workflow is to try PTQ with several calibrators first. If accuracy falls short, leave the most sensitive layers in floating point. If that still fails, run QAT for roughly 10% of the original training schedule, starting from the best calibrated model.
Where accuracy breaks: tolerance by precision and model size
Li et al. (2024) cover 11 model families from 125M to 180B parameters. Their findings, within a 2% loss criterion:
| Precision | Typical behaviour | Main exceptions |
|---|---|---|
| W8, W8A8 | Most families show no significant loss, including on dialogue | OPT-66B loses accuracy even at W8A8 because of extreme activation kurtosis |
| W4, W4A8, KV4 | Negligible loss (under 2%) for most LLMs and tasks | Models under 13B need W8, W8A8 or KV8 to keep multi-step reasoning and self-calibration within 2% |
| W3, KV3 | Noticeable decline in small models across families | LLaMA2-70B at W3 declines only marginally |
| W2, KV2 | Significant loss for most models | The largest models, such as Falcon-180B, hold up better |
| W4A4 | Complete loss of performance for most LLMs | SmoothQuant recovers some dialogue ability but remains well below FP16 |
Model size cuts both ways. Larger models tolerate weight quantization better because their weights have fewer outliers, and they also tolerate KV-cache quantization better, but they tolerate activation quantization worse. At the extreme end, a benchmark of post-training methods found that even the largest model in a family at 2-bit did worse than the smallest at 4-bit.
Other studies sharpen this. A study of five models found 8-bit methods lost about 0.8% while 4-bit methods lost up to 59% on long-context tasks. In long-context retrieval under quantization, accuracy in non-English languages dropped up to five times more than in English. Kurtic et al. (2024), across more than 500,000 evaluations on the Llama-3.1 family, found FP8 effectively lossless, well-tuned INT8 at 1-3% degradation, and INT4 weight-only competitive with 8-bit. These results do not conflict. They cover different models, tasks and context lengths, so one headline number does not travel.
Does passing multiple-choice benchmarks mean a quantized model is safe for code generation?
No. In Li et al.’s evaluation, multiple-choice tasks are scored in perplexity mode: the model rates each candidate answer, and the lowest-perplexity option counts as its choice. It never generates anything. Dutta et al. (Microsoft Research, 2024) found that even when aggregate accuracy moved by 2% or less, the share of individual answers that changed, which they call flips, was 5% or more, reaching 13.6% for some schemes. In their qualitative analysis, the 4-bit model often ignored instructions, made more mistakes and rambled more than the 16-bit baseline. In Li et al., mathematical multi-step reasoning was among the abilities most sensitive to quantization.
Code generation can survive quantization. Kurtic et al. (2024) report 4-bit Llama-3.1 models, quantized with GPTQ on curated calibration data, recovering 98.9% of baseline coding performance. A clean multiple-choice result says little about code. Only a generative test is: pass rates on your own code tasks, plus how many answers flip relative to the unquantized model.
Does quantizing the KV cache carry the same risk profile as quantizing weights?
No. Weight quantization error is fixed once the model has been prepared. KV-cache quantization applies to keys and values produced at runtime, and the cache grows linearly with context length. In Li et al. (2024), most LLMs on long-context tasks of 4K tokens or more were more sensitive to KV-cache quantization than to weight-only quantization at the same bit width. W8KV4 caused more than 10% loss on LongEval even though it was nearly lossless on short-text tasks. The two can also fail in different ways: in small models, KV-cache quantization made outputs more restricted, with the model declining to answer more questions.
The MixLLM authors found 8-bit KV quantization nearly lossless in their setup. MLPerf’s rules treat KV-cache entries like activations for quantization purposes. Evaluation design is treated in more depth in our article on KV cache quantization as a different risk profile from weight quantization.
What should I check first when a published INT4 recipe degrades production quality?
Start by checking whether your traffic looks like the recipe’s evaluation, and only then the recipe itself. In order:
- Context length, language and task type. In the 2025 long-context study, 4-bit retrieval accuracy fell further as inputs grew longer, with drops averaging up to 23% at 128K tokens, and non-English inputs fared worse. A recipe validated on short English prompts says little about long multilingual ones.
- Model family and size. In the same study, one 4-bit method cost Llama-3.1 70B 59% on one benchmark, while the similarly sized Qwen-2.5 72B gained 0.6%.
- Calibration data. Random-token calibration hurt W4A16 accuracy in Kurtic et al. (2024), and different calibration sets give different results.
- Static versus dynamic activation ranges. For W4A8 LLaMA2, static quantization lost much more than dynamic. Switching only the outlier-heavy down-projection layers to dynamic recovered most of the gap.
- Distance from the baseline, not just accuracy. Measuring flips, the share of answers that change from correct to incorrect or the reverse, shows divergence that average accuracy hides.
- Sensitive layers. A few layers often account for most of the loss. Leaving them in floating point trades some speed for accuracy.
Why does INT4 give a bigger throughput gain on LLMs than on image-model training?
LLM decode is the workload INT4 weight-only kernels target. The weights are reloaded for every generated token, so the bytes per weight set the step time. TensorRT’s weight-only INT4 kernels exist for exactly this case: they read 4-bit weights and dequantize them before computing in high precision, which helps when memory bandwidth limits the GEMM.
Training is a different regime. The backward pass has to run in floating point, because accumulating gradients at low precision produces zero gradients or large errors. Even QAT keeps weights in floating point and only simulates quantization. It costs time: 8da4w QAT fine-tuning for Llama3-8B ran about 34% slower than ordinary fine-tuning. For image-model inference, Gholami et al. (2021) cite an additional 50-60% speed-up for INT4 ResNet50 over INT8 on NVIDIA T4 and RTX. In the 26Q3 results, LynxBenchAI’s training workloads run at fp32 and bf16, and low-bit integer formats appear only on the inference side.
Why are two “INT4 with X% loss” reports often not comparable?
Two reports can share a label and differ in almost everything else. Studies of the same network have started from different floating-point baselines, for example 76% versus 77.7% for ResNet50, so a method tested on the stronger baseline looks better by default. Hidden design choices such as symmetric versus asymmetric quantization, per-channel scales, and whether skip connections are quantized can move accuracy by more than 1%, which can be larger than the gap between state-of-the-art methods. Tuning effort matters too: Kurtic et al. (2024) measured 1-3% INT8 degradation where earlier work had reported drops of 10% or more. On top of that come calibration data and the metric itself, where flips can tell a different story from accuracy.
The fix proposed in the literature is a “quantization card” that records these trade-offs alongside the result. The same discipline applies to LLM inference benchmark scores. LynxBenchAI results are compared within one release, so a quantized result from one release is not placed on the same axis as one from another.
Checking that a lower-precision result still clears its correctness bar
A quantization-aware evaluation has to state what was prepared, under whose rules, and what threshold the result had to clear. LynxBenchAI writes those things down for its own catalogue.
LynxBenchAI runs each model at several precisions (fp16, int8 and fp8 for inference; fp32 and bf16 for training; fp64, fp32, fp16 and bf16 for compute workloads), reports each precision separately, and enforces a per-precision correctness threshold before a result counts at all. Its aggregation weights throughput by the memory a run actually moves, so a lower-precision run has to earn its speed-up rather than being credited for being cheap. The models are prepared once, before any target device is known, and the identical prepared artefacts go to every vendor’s hardware. Where two vendors’ kernels disagree about what an approximation may do, the stricter constraint is applied to all of them. A workload that fails, times out or misses its correctness check is recorded as a zero rather than dropped.
MLPerf takes a parallel approach. Its inference rules require a separate accuracy validation run for every submitted performance result, using the same code. Quantization may use only the benchmark owner’s calibration data and must be reproducible and publicly described.
The 26Q3 results show why per-precision reporting matters. On the NVIDIA H100 80GB HBM3, DistilGPT2 inference scores 3797.7 at fp16, 1489.3 at int8 and 1920.2 at fp8. On the AMD Instinct MI300X, the same test scores 3318.5, 1084.3 and 1689.4. On the NVIDIA A100-SXM4-80GB, every fp8 inference test scores 0.0, meaning that precision produced no result on that executor. On the H100 and MI300X, int8 and fp8 scored below fp16 for this test; on the A100, int8 scored below fp16 and fp8 produced no result. Read that as what the scoring credits for this catalogue, not as a verdict on int8 in general.
The catalogue covers five models: MobileNetV3, Flan-T5 Small, DistilGPT2, DistilBERT and GraphGPS. A 26Q3 run executes the catalogue fixed by that release. It does not quantize your model or validate your scheme. LynxBenchAI shows how an executor handles each precision under stated rules. Your own model’s accuracy still needs its own evaluation. The LynxBenchAI Personal Edition is the released edition, so you can measure your own executor at each precision. More on the benchmark is on the LynxBenchAI page.
On its own, the bit width says little about a quantization claim. A claim is only worth trusting if you know which artefact was prepared, what it was calibrated on, which kernel ran it, which context lengths and languages it was tested on, and what correctness bar it had to clear.
