Quantization-aware training (QAT) retrains a model with simulated low-precision arithmetic inside the training loop, so the weights learn to tolerate the rounding they will meet on the device. It is the expensive step, reached for after post-training methods fall short. Post-training quantisation (PTQ) is cheap enough to try first. QAT is the next step when PTQ does not reach the accuracy target.
TechnoLynx brings a trained model inside the device’s accuracy budget through GPU Performance Engineering, applying PTQ first and QAT only where measurements on the target justify it; the first step is to fix the accuracy budget and measure PTQ on the target hardware.
The work covers implementing the model on the Cortex-M (CMSIS-NN), NPU or GPU target, applying structured pruning and INT8 or FP16 quantisation, setting up QAT only where the PTQ loss exceeds the budget, and checking correctness and performance on the hardware before deployment.
Quantization-aware training versus post-training quantisation
PTQ takes a trained FP32 network and converts it to fixed point without the original training pipeline. It needs little or no data, minimal hyperparameter tuning and no end-to-end training. Static PTQ calibrates activation ranges from a small representative set: PyTorch’s guidance puts it at about 100 mini-batches, and Google’s LiteRT converter at roughly 100 to 500 samples.
For GPU serving cost, TechnoLynx offers the Inference Cost-Cut Pack: an audit, then an optimisation sprint.
QAT puts quantisation into training, so the quantisation error becomes part of the loss that the optimiser minimises. That brings the usual costs of training: longer runs, labelled data and a hyperparameter search.
At 8 bits, PTQ is usually enough. Qualcomm’s 2021 white paper reports W8A8 PTQ within 0.7% of floating point for every model it tested. Krishnamoorthi (2018) found 8-bit PTQ within 2% of floating point across a wide range of CNNs, and QAT narrowed that 8-bit gap to 1%. The picture changes at 4-bit weights, and for compact architectures. In the same Qualcomm paper, W4A8 per-tensor PTQ cost MobileNetV2 2.5% and EfficientNet-lite 4.2%, which the authors put down to depthwise-separable convolutions.
Some models become unusable under INT8 PTQ. NVIDIA reports EfficientNet-B0 at 77.4% top-1 in floating point, 33.9% after PTQ and 76.8% after QAT, while ResNet50 quantised fine with PTQ alone. NVIDIA also found that PTQ made its PeopleNet detector unusable, with spurious bounding boxes, while QAT held mAP within 1% of FP32.
Should we do quantisation-aware training from the start?
No. Start with PTQ, and only climb to QAT if the measured loss justifies it. One recommended workflow sets out the ladder step by step. First, run PTQ with several calibration methods. If none reaches the target, apply partial quantisation, which keeps the most sensitive layers in floating point. If that still misses on accuracy or speed, fine-tune with QAT for about 10% of the original schedule, with a learning rate starting at 1% of the original. NVIDIA’s TensorRT recipe says the same: if PTQ accuracy is satisfactory, stop there.
| Rung | What changes | Relative cost | Move up when |
|---|---|---|---|
| PTQ, several calibrators (max, entropy, percentile) | Ranges set from calibration data, no retraining | Lowest | No calibrator meets the accuracy budget |
| Improved PTQ (cross-layer equalisation, bias correction, AdaRound) | Weights and rounding adjusted without fine-tuning | Low | Accuracy still short after these fixes |
| Partial quantisation or mixed precision | Most sensitive layers kept in floating point or at 16-bit | Some performance cost | The speed cost is unacceptable or accuracy is still short |
| QAT fine-tune from the best calibrated model | Weights adapt to simulated quantisation | High: training data, compute, tuning | Last resort, or required for low bit-widths |
The improved-PTQ rung can recover large losses. AIMET reports that cross-layer equalisation plus bias correction took MobileNetV2 from 0.09% to 71.08% at W8A8 against a 71.72% baseline, with no fine-tuning. Partial quantisation brought MobileNet v1 and EfficientNets back to within 1% of FP32 by skipping only a few of the most sensitive layers. One vendor’s guidance also escalates in order, moving from option A to B to C only when accuracy is unacceptable.
In a few cases QAT is the likely end point. Low-bit activations, at 4 bits and below, are where post-training techniques show their limits. TensorRT recommends explicit Q/DQ quantisation for its DLA, which is why one 2026 Jetson study made QAT its production path. Even then, we would run a quick PTQ pass first: it is the fastest way to confirm that the operators compile and the latency target is reachable before committing training time.
The accuracy budget has to be written down before any of this starts, because every escalation decision is measured against it.
What pruning ratio is realistic on a Cortex-M or NPU target?
No single compression setting, pruning ratio included, fits every model. A 2022 study of Cortex-M deployment compressed three architectures to under 10% of their original parameter count before prediction quality degraded. A 2018 study of embedded inference concluded that no single compression setting fits every case, and that the right choice depends on the architecture and the constraints. Our judgement is to choose the ratio at the knee of the measured accuracy-versus-size curve on the target, not at the highest ratio the model will still run at.
The type of pruning also matters. Only structured pruning (removing whole filters, channels or layers) speeds up inference without special sparse hardware or libraries. In the Cortex-M study, structural pruning combined with either post-process quantisation or QAT kept the drop below 5%, measured against the pruned but unquantised model, even under aggressive pruning. Element-wise pruning followed by post-process quantisation lost more than 70% accuracy relative to the pruned but unquantised model. QAT stayed close to zero loss with both kinds of pruning. Iterative pruning costs more than one-shot pruning but generally gives better accuracy.
How quantisation-aware training is done: fake quantisation and the straight-through estimator
Fake quantisation is the mechanism. Inside the training graph, a module quantises a tensor and immediately dequantises it, so the arithmetic stays in floating point but the values carry the quantisation noise that integer inference would add. In NVIDIA’s 2020 formulation, the fake-quantised value is dequantize(quantize(x)), inserted at the inputs of each operation to be quantised. The master weights stay in full precision. Only their fake-quantised copies are used in the forward pass, so small gradient updates are not rounded away.
Rounding has a gradient of zero or undefined almost everywhere, so backpropagation cannot pass through it directly. The straight-through estimator (STE) treats the gradient of the rounding step as 1. Within the representable range, the estimator passes gradients through, and the masked variant zeroes them outside the clipping range. The STE works well empirically, but less so at extreme bit-widths such as 1-bit. One explanation for why QAT recovers accuracy is that training against quantised weights steers the model away from narrow minima, towards flat regions where rounding costs little loss.
In practice, the recipe runs as follows:
- Start from the best calibrated PTQ model, not from random weights. Fine-tuning converges faster and reaches higher accuracy than training a quantised model from scratch.
- Fold batch normalisation and apply cross-layer equalisation where the weights are imbalanced, as in MobileNet-style models.
- Insert the quantisers. In PyTorch eager mode this is
prepare_qatwith a backend qconfig; with NVIDIA’s pytorch-quantization,QuantConv2dreplacesConv2d. For TensorRT targets, put the Q/DQ nodes at the end of each fused block. - Fine-tune for about 10 to 20% of the original epochs.
- Export. Fake-quantisation nodes become ONNX
QuantizeLinearandDequantizeLinearoperators, which TensorRT then optimises.
A model that converges well in simulation has not yet been proven on hardware. Simulated fake quantisation in FP32 does not capture accumulator overflow, rounding modes or operator fusion, so a model that converged under QAT simulation can still degrade on an integer-only target.
Measuring accuracy loss from compression on real device data
The way to measure how much accuracy a compressed model loses is to run it on the target hardware, on real or representative held-out data, and compare it with the full-precision reference. A simulator or a host-side run alone does not establish this. The order of work is set out below: find where the error comes from, measure it on the device, then decide what loss the application can afford.
LynxBenchAI is the measuring tool for the hardware side of this choice: its published results show how candidate targets perform, which informs the board you pick; it measures the hardware, not your model’s accuracy against its reference.
Which layers are most sensitive to INT8, and how do we get a per-layer breakdown?
Run a one-at-a-time sensitivity analysis before choosing a scheme. Quantise a single layer, evaluate the task metric, repeat for every layer, then sort the layers by how much accuracy each one costs. The most sensitive layers are left in higher precision until the accuracy target is met. The effect is often concentrated. Skipping just 2 of the 28 convolution layers in MobileNet v1 gave near-FP32 accuracy in the work PyTorch cites. Qualcomm’s per-quantiser debugging loop and AIMET’s QuantAnalyzer, which enables one layer’s quantiser at a time, are the same method in tool form.
Two cheaper checks are worth running first. Quantise the weights and the activations separately to see which one drives the drop. Activation sensitivity points away from weight-side fixes such as AdaRound. Also run an FP32 sanity check, with the quantisers bypassed, to confirm that the pipeline matches the float model before blaming quantisation. ONNX Runtime offers APIs that match weight and activation tensors between the float and quantised graphs, to find where they diverge most.
Be careful with proxy metrics. PyTorch’s guidance suggests ranking layers by SQNR. However, a 2026 cross-platform study found per-layer SQNR had essentially no rank correlation with the top-1 change (Spearman ρ = −0.030 over 21 layers). We treat SQNR and MSE as hints about where to look, and rank layers by the task metric. That evidence comes from one study, so it is a reason for caution, not a settled rule.
Whether to re-run the analysis per class or per content type depends on how the product is judged. If a regression in one class, genre or site would matter, a global average can hide it. In that case we would repeat the analysis on each subset that has its own acceptance bar. Quantisation effects vary by task and architecture: a 2026 medical imaging study found INT8 PTQ degradation and calibration shifts that depended on the architecture.
Worked example: three INT8 routes on the same Jetson DLA
A 2026 study of a dual-head ResNet-34 attribute classifier on a Jetson Orin NX DLA reports the following, against an FP32 baseline of 94.5%:
| Route | On-target accuracy |
|---|---|
| PTQ, entropy calibration (DLA INT8) | 75.0% |
| PTQ, structurally derived manual ranges (DLA INT8) | 94.0% |
| QAT, 99.99th-percentile calibration (DLA INT8) | 95.0% |
Read the table as a lesson in method more than a ranking. The default calibrator failed, and better PTQ ranges recovered almost everything in minutes of engineering. QAT added the last point and also fixed something accuracy alone did not show. INT8 shifted output scores across the 0.5 decision threshold, and QAT learned to keep scores separated around it. The authors also checked the engine build log to confirm that zero layers fell back to the GPU.
How do we check the quantised model against the full-precision reference on device data?
Run the compiled model on the target, feed it real preprocessed data, and compare with the reference model on the same inputs. Each part of that sentence closes off a known failure:
- On the target, not the host. In ST’s 2026 Edge AI guidance, host-side checks confirm that the model is supported and estimate memory use, but they do not compare outputs or measure performance on the board.
- Real data, not random inputs. With random inputs, similarity metrics can look near-perfect even when an INT8 model has poor real-world accuracy.
- Per target, not once. The 2026 cross-platform study found INT8 predictions disagreeing on about 4% of inputs between targets that used identical scales but different integer kernels, while FP32 agreed exactly. Because the flips cancel out, top-1 accuracy alone does not show this.
- The vendor’s own quantisation path. In the same study, a Qualcomm Hexagon NPU silently ignored externally supplied QDQ scales, and top-1 collapsed from 0.75 to 0.005, with no error raised at compile or run time.
- Full evaluation set, fixed preprocessing. A 1000-image subset overstated top-1 by 9.77 points on average. The choice of resize and crop moved top-1 by 1.07 points, roughly nine times the 0.12-point cost of quantisation itself.
For TensorRT engines, Polygraphy can dump layer outputs, check for NaNs and Infs, and compare against golden values from ONNX Runtime. Because of the calibration shifts noted above, we would also report calibration and score distributions next to accuracy for any threshold-based decision. Agreeing a defensible numerical-parity tolerance between reference and device is a separate question, covered in our guide to validating quantised models for edge deployment and serving efficiency.
How much accuracy loss can the decision afford?
We treat the acceptable loss as a business threshold, not a modelling result. Set it from the consequence of a wrong output before any numbers come in: a missed diagnosis, a bad restocking order, a VMAF floor in a contract. If the quantised model falls below that floor, the honest options are to reduce compression, use mixed precision, or keep the fallback path (a cloud model, a larger edge model), not to move the threshold. Google’s LiteRT guidance says the same thing for PTQ: check that the degradation is within acceptable limits, and consider QAT if it is not.
What release gates should a compressed model clear before the whole fleet gets it?
The first gate runs on the device. Each bundle carries fixed inputs and reference outputs generated in CI on the same target build, and the device compares them within a tolerance before activating the model. This catches broken compilation, delegate fallback, quantisation drift and preprocessing errors. The update handler should also check input shape, quantisation scales and runtime version against the firmware, because matching version strings can hide a preprocessing mismatch.
After that, the release moves through rings: lab devices, a shadow cohort, canaries, then widening percentages. Cohorts are stratified by hardware SKU, runtime version, site and environment, and bake time is counted in inferences per stratum. In shadow mode the candidate runs on the same inputs as the incumbent without driving decisions, which gives a label-free disagreement signal. The metrics and expansion criteria are fixed before rollout starts, and the statistical thresholds are conservative (p < 0.01 has been suggested for industrial edge AI).
Rollback must be triggered by soft degradation, such as accuracy below threshold or a shift in confidence distribution, not only by crashes. A fleet-level gate compares the new version against a holdout cohort, so a single unusual device does not cause a rollback. The device must also be able to revert on its own, because a bad model may be what cuts it off from the server. Deciding when a change triggers revalidation is the subject of our piece on keeping deployed models trustworthy through revalidation, monitoring and rollback. The golden vectors are produced in CI per target, so these gates are part of the MLOps delivery pipeline.
Can INT8 alone meet our memory budget, or do we also need pruning or distillation?
Whether INT8 alone is enough follows from simple arithmetic. Going from 32 to 8 bits cuts tensor storage by a factor of four. If the FP32 weights and activations need more than four times the available memory, INT8 alone cannot fit, and you need something else.
TechnoLynx ports the model to the target device through the AI Porting & Deployment Pack; the first step is to compare the FP32 footprint with the memory available on the device.
Precision strategy that keeps accuracy on the cheaper chip
Below 8 bits, mixed precision is an alternative to uniform low-bit quantisation. Bits are allocated per layer by sensitivity. One 2020 paper used mixed precision to fit MobileNetV1 on an STM32H7 with 2MB of flash and 512kB of RAM, at 68% top-1, 8% higher than earlier 8-bit integer-only results under the same limits. At a 1MB constraint, the same method was up to 7 points below a competing approach that did not stay integer-only. Tooling is the catch. The CMSIS-NN kernels described in 2018 support 8-bit and 16-bit data, and a 2025 microcontroller survey notes that mixed precision is rarely supported because hardware and toolchains are tied to fixed bit allocations.
Fitting the model inside the memory budget
Quantisation reduces bits, not parameters. Pruning and distillation reduce the parameter count. These techniques can be combined: Deep Compression shrank AlexNet by 35x and VGG-16 by 49x without accuracy loss, using pruning, trained quantisation and Huffman coding. Distillation can also be folded into quantised training, with the student network trained against a teacher while its weights are quantised. Which lever to pull depends on the constraint. The 2018 embedded study found quantisation cut storage most, while pruning gave faster inference, and on that CPU quantisation actually increased latency and energy because of dequantisation overhead.
Confirming the compiled model runs on the actual chip
Measure the speedup on the target, do not assume it. The same INT8 model ran 1.83x to 2.11x faster on Arm cores with dot-product instructions and 1.65x to 1.76x slower on an x86 core without AVX-512 VNNI and a Cortex-A53 without dot-product instructions. Vendor NPUs define their own numerics, and the cross-platform study recommends treating vendor-native quantisation as mandatory. Our view is that the right early milestone is a working build on the real silicon with a golden-vector check passing, before anyone commits weeks of QAT effort to a model that may never compile cleanly.
Who can run the PTQ-to-QAT escalation on our target hardware?
TechnoLynx takes a trained model and brings it inside a device’s memory, latency and accuracy budget, on the target that will ship. The result is an accuracy-versus-size curve from the real hardware and a model validated there before deployment. When the hard part is moving the model to a new chip or runtime, the AI Porting & Deployment Pack is the matching starting point.
This is not the right engagement if your model already meets its budget with plain PTQ.
FAQ
Does quantization reduce model size?
Yes. Moving tensors from 32-bit to 8-bit storage cuts their memory by a factor of four. It does not remove parameters, which is what pruning and distillation do.
What is a QAT LLM?
It is a large language model fine-tuned with fake quantisation so that it holds its quality at low precision. NVIDIA reports that QAT fine-tuning for under 1% of the original pre-training time is often enough to restore LLM quality.
