Quantisation-aware training (QAT, also searched as quantized aware training) fine-tunes a pretrained model with fake-quantisation operations in its graph. The weights adapt to INT8 rounding and recover accuracy that post-training quantisation (PTQ) loses. The gap can be large. NVIDIA reports EfficientNet-B0 at 77.4% Top-1 in floating point, 33.9% after PTQ and 76.8% after QAT, while ResNet50 quantises well enough with PTQ alone. The sensible order is PTQ first, then partial quantisation of the sensitive layers, then QAT, which costs about 10% of the original training schedule on an annealed learning rate. Whichever route wins, measure the INT8 speed figure at the same batch size as the FP16 run and revalidate accuracy.

PTQ and QAT do different things to the same INT8 model

PTQ needs no retraining and no labelled data. It calibrates scales on a small sample and casts the trained model to INT8. QAT needs fine-tuning and labelled training data, and in return it allows lower bit widths with competitive accuracy. In QAT the model is retrained with quantised parameters, so it can learn to correct the bias that rounding introduces. The Qualcomm white paper by Nagel and colleagues (2021) states that PTQ is usually enough to reach 8-bit quantisation with close to floating-point accuracy. That is why PTQ is the right first step for INT8 work.

In PyTorch’s own QAT flow, the converted model is the same quantised model the PTQ flow would produce with the same settings, with better-trained weights inside it, so it drops in as a replacement and reuses the same kernels. QAT does not change the deployment path. It changes the weights you deploy.

When is quantisation-aware training worth the extra training cost over PTQ?

QAT is worth its cost when every PTQ calibration you try misses the accuracy target and partial quantisation either still misses it or gives up too much speed. The workflow Wu et al. (2020) recommend is concrete. Quantise all compute-heavy layers and try max, entropy and the 99.99% and 99.999% percentile calibrations. If none of these is good enough, run a one-at-a-time sensitivity analysis, then leave the most sensitive layers in floating point until accuracy is acceptable. Only when that costs too much performance, or still falls short, do you move to QAT.

Some architectures need QAT and some do not. Even with per-channel quantisation, MobileNet does not reach baseline accuracy under INT8 PTQ. In NVIDIA’s EfficientNet-B0 example, PTQ collapses accuracy and QAT brings it back to within 0.6 points of floating point. The Jetson study discussed later attributes the PTQ drop on standard EfficientNet-B0 to its Squeeze-Excite blocks and SiLU activations.

Bit width changes the picture. In the LLM-QAT study (Liu et al., ACL Findings 2024), PTQ was enough at 8 bits, but quantising weights, activations and the KV cache together below 8 bits made QAT necessary. Google’s 2018 whitepaper found the same pattern for CNNs: QAT’s improvements were far larger at 4 bits than at 8. Run-to-run noise matters as well. Across the networks Wu et al. tested, QAT kept every model within 1% of FP32. Where PTQ came out marginally ahead, the difference was at the level of random-initialisation variance.

Fake quantisation and the straight-through estimator, step by step

Fake quantisation quantises a tensor and immediately dequantises it. The values carry INT8 rounding error, but the computation stays in floating point. Wu et al. define it as x̂ = dequantize(quantize(x, b, s), b, s), inserted at the inputs of each operation to be quantised. PyTorch’s FakeQuantize module does the same thing: it simulates the quantise and dequantise operations during training, using a scale and zero point, with switches that turn fake quantisation and statistics collection on or off independently.

The backward pass has a problem to solve. The gradient of a rounding step is zero almost everywhere. The straight-through estimator (STE) treats the derivative of the fake-quantisation function as 1 inside the representable range, so gradients pass through the fake-quantisation step. The weights are kept and updated in floating point, which stops small updates from underflowing, and are quantised again on every forward pass. Why STE works is not clear. It is reliable in practice except at extreme bit widths such as 1-bit binary.

Wu et al. offer an intuition for why the trick helps. PTQ can push a weight sitting in a narrow minimum onto a rounding point where loss is much higher. Wu et al. argue that training against the quantised weights favours wider minima, where the quantisation points cost less.

The QAT failure modes worth checking first

• Batch-norm folding. Batch norm is folded into the preceding weights for inference. On MobileNet v1 and v2, the 2018 whitepaper reports that naive folding during training is less stable than batch-norm correction with freezing, and that quantised inference must match the training forward pass.
• Exponential moving averages of weights. Floating-point weights in QAT converge towards quantisation decision boundaries, so small EMA differences can flip quantised values and cost accuracy.
• Weight oscillation. Nagel et al. (2022) show that quantised weights can oscillate between two grid points, which corrupts batch-norm statistics at inference. The effect is strongest at 4 bits or fewer in MobileNet- and EfficientNet-style networks.
• A slower INT8 engine. QAT can recover accuracy and still produce an engine that runs slower than FP16 on a given TensorRT version, as the Jetson case below shows.

Running quantisation-aware training in PyTorch and deploying it with TensorRT

The PyTorch-to-TensorRT path runs through these steps. Each step depends on the one before it:

1. Prepare. In torchao, prepare inserts fake-quantise operations into the linear layers. QATConfig takes a PTQ config and works out the matching fake-quantisation settings. NVIDIA’s pytorch_quantization toolkit does the same job for CNNs, swapping torch.nn.Conv2d for QuantConv2d.
2. Calibrate. Feed calibration data, which is usually a subset of training or validation data, to record activation ranges. Then compute scales with max, histogram or entropy methods.
3. Check PTQ first. If the calibrated model is accurate enough, export it and stop, because that is the PTQ recipe.
4. Fine-tune. Start from the best calibrated model and fine-tune for about 10% of the original schedule. Begin the learning rate at 1% of the original initial value and decay it on a cosine schedule. The Torch-TensorRT notebook keeps the calibrated ranges frozen during this phase.
5. Choose granularity. Use per-channel scales for weights and per-tensor scales for activations. NVIDIA’s TensorRT documentation reports this configuration gives the best accuracy.
6. Export Q/DQ. Each fake-quantise operation exports to ONNX as a QuantizeLinear/DequantizeLinear pair. PyTorch 1.8.0 and later can export these with per-channel scales.
7. Build the engine. TensorRT expects a Q/DQ pair on each input of a quantisable layer. It fuses those pairs into quantised layers and moves Q nodes backward and DQ nodes forward, so more of the graph runs at low precision. With Torch-TensorRT, compile the model with enabled_precisions set to torch.int8, then re-test its accuracy on the compiled module.

Q/DQ placement is a real trade-off, not plumbing. Aggressive quantisation cuts latency and costs accuracy. The default guidance is to quantise the inputs of convolutions and GEMMs and leave their outputs in higher precision when an activation such as SiLU follows.

What QAT costs in data, compute and memory

The usual budget is about one tenth of the original training steps. BERT is the exception in Wu et al., where the full SQuAD fine-tuning schedule was repeated because it lasts only two epochs. Each step also costs more. PyTorch measured QAT fine-tuning (8-bit dynamic activations, 4-bit weights) as about 34% slower than regular full fine-tuning on Llama3, with a significant memory increase because weights are cloned before fake quantisation. Activation checkpointing mitigates most of that memory. Dremov et al. (2025) report no significant slowdown, but only after making sure torch.compile handled the quantisation operators without slow fallbacks. We read those two results as compatible: the overhead is real by default and mostly recoverable with compilation work.

The 10% rule is a starting point, not a universal optimum. Dremov et al. find that the compute-optimal QAT fraction grows with the total compute budget. They explicitly challenge 10% as a universal optimum and report that low-bit QAT is the most sensitive to getting the fraction wrong. LLM-QAT puts the cost against PTQ in absolute terms: 0.7 to 1.9 days on 8 A100 GPUs for the 7B to 30B models, against 0.1 days for PTQ. The authors argue the training cost is small once spread over millions of inference requests.

Data access is a constraint too. QAT is not possible when training data is too sensitive or unavailable. LLM-QAT works around this for generative models by distilling on text the model generates itself. Calibration alone needs far less: TensorRT’s documentation cites about 500 images as enough for ImageNet classifiers.

Does the BitNet model answer our customer prompts as well as the INT8 model?

A defensible answer comes from running BitNet b1.58 and the incumbent INT8 model on the same representative prompt set, scored with one shared metric, against a quality-drop threshold agreed before anyone sees a score.

TechnoLynx can run this comparison as an engagement. TechnoLynx agrees the prompt sample, the scoring rubric and the pass/fail threshold with your team. TechnoLynx then builds a harness that runs BitNet b1.58 through bitnet.cpp alongside your quantised model, with both served under matched conditions on the target hardware. The decision work sits in TechnoLynx’s LLM Selection Pack, and the first step is fixing the threshold and prompt set together.

Microsoft’s BitNet b1.58 2B4T report describes performance comparable to full-precision open-weight models of similar size, using less memory than INT4 PTQ versions of Qwen2.5 1.5B. Nielsen and Schneider-Kamp (2024), summarising the original BitNet b1.58 work, note that from about 3B parameters, 1.58-bit models trained from scratch match 16-bit ones. Those results compare against public baselines, not your INT8 model on your customers’ prompts. Public LLM benchmarks and leaderboards rank models on shared tasks, which is a different question from whether your product regresses.

Three design choices decide whether the harness tells the truth:

• The threshold is set first. A threshold chosen after seeing the scores turns the result into whatever someone wanted it to be.
• The scoring method is fixed in advance. Switching between human and automated scoring partway through breaks the comparison.
• The serving stacks are matched. Differences in tokenisation, sampling defaults or kernels between the two stacks would shift quality or speed for reasons unrelated to ternary weights.

Report score deltas with their run-to-run variation. Wu et al. found that some PTQ-versus-QAT INT8 differences were no larger than retraining noise. The decision rule follows from the threshold. Approve the switch if the ternary model stays within it across representative categories. Reject it, or tune further, if any critical category falls outside. Consider a staged rollout when results are mixed. Even a passing set of prompts will miss some real edge cases, so ongoing regression monitoring belongs in the plan.

If the switch also means choosing new hardware, LynxBenchAI is the tool for that part. Its published results show how each device runs the public PyTorch stack a team’s own models run on, so they are broadly indicative for custom PyTorch models too. It does not run or score your own workload.

What SRAM and accuracy thresholds would justify switching shipped firmware to ternary?

Replace INT8 with ternary weights in production firmware only if a side-by-side pilot shows two things: ternary meets a predefined accuracy-retention threshold, and it does so at a substantially lower SRAM footprint within your firmware’s budget. Keep a documented fallback to the INT8 kernels in case ternary underperforms during integration.

LynxBenchAI publishes results showing how each device runs the public PyTorch stack, which helps when the switch also means choosing host or accelerator hardware; the first step is to check its published results for the devices you are considering, and they do not run your firmware.

TechnoLynx scopes that go/no-go decision with you. TechnoLynx defines the SRAM budget, the accuracy threshold and the integration constraints, and designs the INT8-versus-ternary pilot so that checkpoints and tracking fit your existing MLOps pipeline. TechnoLynx ports and optimises ternary kernels on the target MCU or accelerator alongside the INT8 path, measures both against the gates, and builds the mechanism that reverts to INT8.

The thresholds have to come from your use case. A wake-word false accept on a consumer device and a missed alarm on safety equipment do not share a tolerance, and a generic benchmark threshold suits neither. Two parts of the published ternary results deserve a careful look. Activations in BitNet-style layers stay at 8 bits, so the saving is in weights only. Capacity is the other: in the small networks they tested, Nielsen and Schneider-Kamp (2024) found that 1.58-bit models needed hidden layers about twice as wide to match 16-bit performance.

Worked example: what doubling the hidden size does to the weight saving

Assume a dense layer of width h, so h² weights, stored at 8 bits in INT8. Assume ternary weights at their information content of 1.58 bits each. At equal width, ternary uses 1.58 / 8 ≈ 20% of the INT8 weight memory, a saving of about 80%. Now widen the layer to 2h to recover capacity: it has 4h² weights, which is 4 × 1.58 ≈ 6.3 bits for every original INT8 weight, so the saving falls to about 21%. If your kernel packs each ternary weight into a 2-bit field, the widened layer takes 4 × 2 = 8 bits per original weight and the saving disappears. The wider layer’s activation buffers also double, and those stay at 8 bits. These numbers follow from the assumptions above, not from a measurement.

The worked example shows why SRAM savings on paper may not survive kernel-level reality. The fallback plan matters for a similar reason. Ternary layers can in principle use multiplication-free kernels built from additions and subtractions, but whether such kernels are ready on your MCU is something the pilot has to establish. TechnoLynx would keep the INT8 path compiled and switchable until the ternary build has passed on real devices. If the pilot fails either gate, keep INT8 and treat ternary as a candidate to revisit when its kernels mature.

Reporting INT8 accuracy and speed without conflating them

Report accuracy and speed as two separately measured deltas against the same FP16 baseline. Then add an accuracy-matched point: the fastest INT8 or mixed-precision configuration that stays within the agreed accuracy tolerance. A raw INT8 speed-up from a model that lost accuracy is not comparable to anything.

TechnoLynx offers the Inference Cost-Cut Pack for teams whose quantisation work is meant to lower inference cost. The reporting discipline below is what makes such a saving credible to a customer.

The accuracy-matched point comes straight out of the PTQ workflow. Leaving sensitive layers in floating point trades some speed for accuracy. The honest comparison is therefore between FP16 and the configuration that reaches FP16-level accuracy, not the most aggressive one. NVIDIA’s TensorRT benchmarking guide sets the other ground rules. Measure the quantised model at the same batch size as the FP16 run. Calibrate on representative data and revalidate accuracy, because an engine built on random calibration data is a performance experiment, not a deployable result.

Report item	What to state	Why it matters
Accuracy delta	INT8 versus FP16 on the same held-out set, with run-to-run variation	Some INT8 differences are no larger than retraining noise
Accuracy-matched point	Fastest configuration within the agreed tolerance	Separates the speed gain from any accuracy lost
Speed metric	Throughput and median latency	These are the comparison metrics NVIDIA’s benchmarking guide names
Batch size	Identical for INT8 and FP16	Batch size moves both latency and throughput
Clocks	Locked, with GPU status monitored	Runs of the same engine can differ by tens of percent with floating clocks or throttling
Inputs	Representative values, not zeros	Zero inputs draw less power and flatter the result
Data transfers	Stated as included or excluded	trtexec excludes host-device copies by default
Runtime version	Exact TensorRT version	Q/DQ fusion quality differs between releases

Profile before claiming a speed-up, because the precision that sounds faster is not always the one that is faster on your device and runtime. A study of EfficientNet-B0 on a Jetson Orin Nano gives a published example. FP16 matched FP32 at 3.19 ms per image including transfers. QAT recovered INT8 accuracy, but on TensorRT 8.5.2 the engine ran slower than FP16 because of insufficient Q/DQ fusion, and no INT8 route beat FP16 on both speed and accuracy. The authors’ rule is to check first whether FP16 meets the latency target. trtexec --dumpProfile shows which layers are slow before you decide where precision changes would help. Workloads differ too: MLPerf Mobile found that NLP tasks needed FP16 in real deployments, while vision tasks use INT8 heavily. Profiling of this kind is part of TechnoLynx’s GPU performance engineering work.

A controlled keyword-spotting pilot on one STM32H7

A controlled pilot can answer the INT8-versus-ternary question for a keyword-spotting model. Train one architecture in three arms (INT8 PTQ, INT8 QAT and BitNet b1.58 ternary) on the same data splits, then measure accuracy, peak SRAM, flash and latency on the same STM32H7 board. The memory gate is tight. Arm’s Hello Edge study (Zhang et al.) notes that microcontrollers typically have tens to a few hundred KB, and the weights, activations and I/O must all fit. The same study reports 95.4% accuracy for a DS-CNN, making it a reasonable reference architecture.

The small-network ternary results cited here include text classification and graph tasks, not wake words. On text classification, 1.58-bit WideMLP variants reached 98.4% to 99.0% of full-precision performance. We would report false-accept and false-reject rates at one fixed operating point for every arm, because top-1 accuracy alone hides the trade-off a wake-word product actually ships.

The ternary arm needs a full training pipeline, not a conversion. BitLinear keeps 16-bit shadow weights, quantises them to {−1, 0, 1} on each forward pass and updates them through STE, so it needs labelled data like any QAT. Budget for a sweep as well as a single run. In small-model experiments, lower learning rates worked best, AbsMean versus AbsMedian scaling is left as a hyperparameter, and hidden layers needed about doubling. Budget the sweep as several training runs, not a conversion step. Two neighbouring questions are separate pieces of work: keeping rare-class recall intact under INT8 on a particular MCU or NPU toolchain, and comparing ternary training with INT4 QAT under a matched budget.

Who can run an INT8 versus ternary pilot on our hardware?

TechnoLynx designs and runs quantisation pilots on the hardware you ship. TechnoLynx agrees the accuracy threshold and SRAM budget with you, designs the INT8-versus-ternary pilot, optimises both paths on the target hardware, and measures them against those gates. A specialist is worth bringing in when the kernel porting and fair-measurement work is what is missing. When choosing a partner, ask how they will fix thresholds before testing and keep the serving stacks matched. TechnoLynx’s Inference Cost-Cut Pack is the engagement for this, and it starts with an audit.

An engagement like this does not suit every team. If PTQ already meets your accuracy target, the INT8 work is close to done and you need a profiling check, not a pilot. If the decision is mainly about which off-the-shelf model to adopt, or about domain sign-off such as safety certification of a device, the engineering pilot covers only part of it, and the certification belongs with the specialists who own that process. A team that only needs a hardware comparison can start from published LynxBenchAI results before commissioning any engineering.

FAQ

Does quantisation reduce inference time?

Usually, though not always. Google’s 2018 whitepaper reports 2x to 3x faster quantised inference on CPUs and almost 10x on Qualcomm DSPs. The Jetson study shows a QAT INT8 engine running slower than FP16 on TensorRT 8.5.2.

How many bytes does INT8 use?

An INT8 value takes one byte, a quarter of the four bytes of FP32. Wu et al. note that this smaller word size gives memory-bound operations up to a 4x speed-up over FP32.

What are ternary AI models?

Ternary models restrict the weights of their linear layers to −1, 0 or 1, as BitNet b1.58 does, usually with 8-bit activations. They are trained with quantisation in the loop rather than converted afterwards.