INT8 quantization converts a model’s 32-bit floating-point parameters and activations to 8-bit integers, using a scale and a zero point. That cuts the bytes moved per operation by a factor of four and lets NVIDIA GPUs run convolutions and matrix multiplies on their faster 8-bit Tensor Cores. The catch is that none of the accuracy, latency or cost gains carry over automatically from a paper or a vendor chart to your model on your hardware. You have to measure them. Check accuracy and confidence against the FP32 (or FP16) baseline on your own data, measure speed on the target device after confirming the INT8 kernels actually run, and work out cost per request from those two results together.
How does INT8 quantization cut the memory and cost of serving a model?
Which savings you get depends on whether the workload is memory-bandwidth bound or compute bound. A workload limited by memory bandwidth, such as autoregressive decoding at small batch sizes, gains mostly from quantising weights, because its cost is bytes moved rather than arithmetic. A compute-bound workload (prefill, vision backbones, high-concurrency batched serving) gains from quantising weights and activations together, so the engine can issue real integer matrix multiplies on Tensor Cores.
The gains depend heavily on the hardware: on-chip memory, bandwidth and cache hierarchy all change how much a quantised layer speeds up. Krishnamoorthi’s 2018 whitepaper reports a fourfold model-size reduction from 8-bit weights alone, 2x to 3x speedups on CPUs, and up to 10x on DSPs with fixed-point SIMD. Meta’s 2021 account of its production recommendation models cites up to 4x memory savings and a 2x compute boost on Intel Broadwell and Skylake CPUs.
The DEEP-GAP study of ResNet models on T4 and L4 GPUs found a 2x to 3x smaller memory footprint and the lowest P99 latency of the three precisions tested. In the DEEP-GAP study, the speedup rose quickly between batch sizes 1 and 32 and then flattened as compute saturated. So a single-request latency test understates what INT8 does to serving capacity, and a large-batch test overstates what a latency-bound service will see.
A GPU billed by the hour that serves more requests in that hour costs less per request, and a smaller memory footprint fits larger batches on the same card. For LLM serving, a 2024 quantisation study found that W8A8 formats (8-bit weights and activations) gave the highest throughput under asynchronous continuous batching. The same study found W4A16 (4-bit weight-only) quantisation the most cost-efficient choice for synchronous deployments.
Quantisation is one of the serving-path levers worth pulling before anyone proposes replacing the model. It changes how the model is served rather than what it is, and profiling, not habit, decides whether it is enough. TechnoLynx audits production AI infrastructure and SaaS stacks for these improvements before recommending model replacement.
Is FP8 worth planning for on A100, or is the real choice FP16 versus INT8?
On an A100 the realistic choice is FP16 versus INT8. The A100’s Tensor Cores support FP16, BF16, TF32, FP64, INT8, INT4 and binary, but not FP8. FP8 Tensor Cores and the Transformer Engine arrive with H100, and Databricks states plainly that 8-bit floating-point operations run on H100 and not on A100.
TechnoLynx starts by agreeing the quality metrics and latency targets the tender must meet.
NVIDIA A100 and H100 GPUs do INT8 math twice as fast as FP16, and the A100 whitepaper lists 624 INT8 TOPS against 312 FP16 TFLOPS. How you reach that gain still matters. Databricks found that INT8 weight-only quantisation, which leaves activations and therefore the arithmetic in FP16, improved time per output token by about 30% below batch size 8 and gave little or no speedup above it. Activations carry more outliers than weights.
If a requirement genuinely needs FP8-level efficiency, that is a hardware decision pointing at H100, not a quantisation setting to look for on A100.
Measuring the drop on your own chatbot traffic
The size of the quality drop depends on the model, the task and the traffic, so measure it on your own model and real queries rather than reading it off a vendor chart. In the 2024 LLM study, well-tuned W8A8 INT8 lost only 1–3% per task on average. That result used GPTQ with dynamic activation quantisation or SmoothQuant, adding SmoothQuant at 70B and better calibration data for larger models. The same paper reports that proper tuning cut a 10-point drop reported elsewhere for a 405B INT8 model to 0.7 points.
Aggregate scores can also hide changed answers. Dutta et al. found that compressed models with near-identical accuracy still flipped answers from correct to incorrect and back. For a public-facing query assistant, score answer correctness and the tone and safety of responses on a sample of real queries, and track how often answers flip between FP16 and INT8, rather than relying on perplexity. Measure latency and throughput in the same run on the same cluster, so every quality change is paired with the speed it bought.
Two limits apply. A sample from one time window may miss seasonal query patterns. Quantisation-aware training may recover more quality than post-training quantisation, but it needs retraining, which a tender timeline may not allow.
Which is better for production inference, INT8 or FP8?
Neither wins everywhere. The hardware decides first, and the model’s outlier behaviour decides second.
Qualcomm’s 2023 analysis found FP8 multiply-accumulate units 50% to 180% less efficient than INT8 in dedicated hardware, depending on accumulator size. It concluded that very few models cannot be reproduced accurately in INT8, given post-training quantisation, W8A16 fallback for a few layers, and QAT where needed. The deciding factor in that analysis is outliers: distributions with large outliers favour FP8-E4 or FP8-E5, while well-behaved, Gaussian-shaped ones favour INT8.
For post-training quantisation, other evidence leans the other way. A 2023 study across 75 architectures found FP8 E4M3 met a 1% relative-loss criterion on 92.64% of workloads, against 65.87% for INT8. INT8 struggled on EfficientNet, MobileNetV3 and ViT. The NVIDIA, Arm and Intel FP8 formats paper (2022) likewise reports BERT keeping its accuracy under FP8 post-training quantisation, where INT8 lost significant accuracy. For LLMs, the 2024 study found W8A8 FP8 effectively lossless at every model scale.
The two camps disagree less than they appear to. The Qualcomm paper deliberately left out INT8-only techniques such as AdaRound and bias correction, which lowers its INT8 PTQ numbers. The 75-architecture study left quantisation-aware training out of scope. INT8 is the more portable target, and in dedicated hardware the more efficient one, when you can afford careful calibration or QAT.
| Target | Low-precision formats with native acceleration (vendor documentation) | Realistic comparison to run |
|---|---|---|
| NVIDIA A100 | FP16, BF16, TF32, INT8, INT4, binary; no FP8 | FP16 vs INT8 |
| NVIDIA H100 | FP8 (E4M3, E5M2), FP16, BF16, TF32, INT8 | FP16 vs INT8 vs FP8 |
| Jetson Orin NX | GPU plus two DLA cores; INT8 engines on the DLA | FP16 vs INT8, on GPU and DLA separately |
How much accuracy does INT8 quantisation lose, and how do you check it against the FP32 baseline?
The only honest answer is the difference your test harness measures. Run the FP32 baseline and the TensorRT FP16 or INT8 engine on the same test set, compute the task metrics for both, and compare the difference with a tolerance agreed before the test. Published results give a range to expect, not a number to quote. A 2025 study of YOLO12 detectors found FP16 TensorRT engines lost negligible accuracy, while static INT8 engines ran 1.5x to 3.3x faster than FP32 and lost about 3–7 points of mAP50-95, with smaller models hit hardest.
TechnoLynx builds this equivalence harness on the target device, for example a Jetson Orin. TechnoLynx begins by agreeing the acceptable accuracy tolerance and the safety-critical edge cases. TechnoLynx then builds and calibrates the TensorRT engine and reports the accuracy and latency differences against the baseline as a pass or fail.
The harness, step by step
- Keep preprocessing identical across FP32 inference, calibration and deployment: resize, padding, channel order, normalisation and tensor layout.
- Compute task metrics for both models: mAP, per-class precision and recall, and confusion on the hard examples.
- Before ground-truth evaluation, check behavioural consistency between the FP32 and INT8 outputs: detection count, confidence shift and bounding-box IoU.
- Gate the result against the pre-agreed tolerance. Meta’s production example is a normalised-entropy difference of no more than 0.05%, with a 100 ms p99 latency target.
- Localise failures with layer-by-layer mean squared error and cosine similarity between the reference and INT8 outputs, then move the layers with high error back to FP16.
- Confirm the fast path. Read the TensorRT build log and check that INT8 kernels were selected, because a quantised model running on FP16 kernels is slower than the baseline it replaced.
Meta ran its search on small calibration and evaluation sets first, then rechecked on the full evaluation set and drew more samples when the two disagreed. It also re-validated numerics after each vendor software release, because kernel implementations change between releases. As a reference point, one published FP32-versus-INT8 consistency check on a detector reported a mean IoU of about 0.897 for matched detections. Its confidence shift was centred near zero, with a standard deviation of roughly 0.06 to 0.08.
Coverage is the harder question. Calibration data should cover the conditions the deployed application must handle, such as day and night, varied camera angles and boundary cases. Noise deserves particular attention. In the YOLO12 study, static INT8 engines were consistently more sensitive to Gaussian noise than FP32 and FP16, with relative mAP drops above 60% for the nano model under medium noise. Adding degraded images to calibration did not reliably fix this across model sizes.
Equivalence is whatever your thresholds and coverage define, so write both down before the run. A fixed benchmark catalogue cannot answer this question for you, because it does not run your detector on your data.
Calibration choices can break INT8 on their own
Calibration is the process of choosing each tensor’s clipping range. On a Jetson Orin NX DLA, one study saw standard TensorRT entropy calibration drop a classifier to 75% accuracy, against a 94.5% FP32 baseline. Structurally derived manual dynamic ranges recovered 94.0%. It recommends entropy calibration for CNNs and min-max for NLP, with per-tensor activations and per-channel weights for the best accuracy. Calibrating on a public corpus when production serves different inputs sets the scales to the wrong distribution. The better practice is to calibrate on production samples and keep adding them until the per-layer scales stop moving.
Because calibration data shapes the result, fix and version the calibration set so that a rerun reproduces the same engine. And never let a latency gain cover for an unacceptable loss on a safety-critical edge case.
When is quantisation-aware training worth the extra validation effort?
Post-training quantisation (PTQ) calibrates a trained model on a small representative dataset to set clipping ranges and scales, with no retraining. Quantisation-aware training (QAT) adds fake-quantisation operations to training so the network learns to tolerate the rounding. PTQ’s overhead is very low, often negligible, and it usually costs some accuracy compared with QAT, especially at low bit widths.
For many models PTQ is enough. NVIDIA’s 2020 integer-quantisation workflow kept 8-bit accuracy within 1% of FP32 on every network studied, including MobileNets and BERT-large, and ResNet50 quantises with PTQ alone. EfficientNet-B0 is a counterexample: NVIDIA reports 33.9% top-1 accuracy with PTQ, 76.8% with QAT and 77.4% for the floating-point baseline. The price of QAT is retraining. NVIDIA’s recipe fine-tunes a pretrained model for around 10% of the original training schedule, while a 2021 survey notes that low-bit QAT can need several hundred epochs.
The order the published evidence supports is:
- PTQ with representative calibration data and per-channel weights.
- Per-layer diagnosis, with FP16 or W8A16 fallback for the sensitive layers.
- QAT only when the accuracy gap survives both.
In the Orin NX study, better calibration came within 1 percentage point of QAT with no training time at all. So a failed PTQ run is not yet evidence that you need QAT. Because QAT fine-tunes the weights, validate it against the original FP32 baseline, not the PTQ engine it replaces. More on that handover: validating compressed models from PTQ to QAT.
Why do confidence scores shift after INT8 quantisation, and should thresholds be recalibrated?
Quantisation adds noise at the logit level, and low-confidence predictions are the first to change. Calibration can therefore degrade even when accuracy barely changes. A study of quantised LLMs found higher calibration error after quantisation in 41 of 48 measurements. The models were under-confident on correct answers and over-confident on wrong ones.
LynxBenchAI publishes how each candidate device runs the public PyTorch stack your models are built on, so the first step is to check its results for the devices on your shortlist before per-target recalibration starts.
TechnoLynx runs the INT8 and FP16 quantisation on each target, diagnoses the confidence shift and implements per-target recalibration. That includes a field protocol comparing the quantised model’s detections and confidence against the reference model under production conditions. For choosing between candidate devices before that work starts, LynxBenchAI publishes how each device runs the public PyTorch stack your models are built on. Its results are broadly indicative for custom PyTorch models, but it does not run or measure your own model.
Diagnosing the shift on an Orin NX
Recalibrate per hardware target. Temperature scaling, a single-parameter variant of Platt scaling, is a simple first fix. Guo et al. (ICML 2017) found it surprisingly effective on most datasets. The calibration method also interacts with fixed thresholds. In the Orin NX study, 99.99th-percentile calibration kept outputs calibrated around the decision boundary, which made it the right choice for fixed-threshold deployment.
To tell a quantisation artefact from a shift between development and field data, run the FP32 reference and the quantised engine on the same field data. Suppose the reference holds its calibration and only the quantised engine drifts, in the same way across targets. That points to a quantisation artefact, which per-target recalibration addresses. If the reference drifts too, the data has moved, and recalibration would only hide the problem.
Pairing throughput numbers with Dice or IoU
Every INT8 throughput figure shown to a customer should sit next to a task metric measured against the FP32 baseline on the same images, such as Dice or IoU for segmentation. Quantisation research reports segmentation results the same way, for example mIoU for DeepLabV3 on Pascal VOC and HRNet on Cityscapes. Matching Dice or IoU does not prove that confidence scores survived, so a calibration metric belongs in the same table.
Which clinical guidance to cite is a question for your regulatory affairs team. TechnoLynx supplies the engineering evidence, not the regulatory interpretation.
What evidence should an INT8 quantization report give procurement and quality reviewers?
Give reviewers measured differences on their own queries, not quantisation theory. Put latency, cost per request and quality for each tested configuration on one page, and keep the full method behind it for audit.
TechnoLynx builds that report as part of the Inference Cost-Cut Pack. TechnoLynx starts by agreeing the workload and the quality, latency and cost thresholds with procurement. TechnoLynx then benchmarks INT8, FP8 where the hardware supports it, and continuous-batching configurations on the target hardware.
For a non-technical board, the trade fits in one sentence: lower precision gives up numerical detail in exchange for throughput and speed. The report then shows how much quality that trade cost on this workload. FP8 needs accelerators that support it, such as H100, while INT8 runs on A100, H100, many custom accelerators and CPUs.
| Report item | Why reviewers need it |
|---|---|
| Hardware, software versions, precision, batch size | Vendor kernels change between releases, so each number must name the setup that produced it |
| Quality change against baseline on your own queries, with flip rate | Equal accuracy can hide changed answers |
| Calibration error before and after quantisation | Quantised LLMs showed higher calibration error in 41 of 48 measurements |
| Median and P99 latency | Median shows average behaviour; P99 shows tail behaviour |
| Throughput and cost per request | Cost per request is GPU cost per hour divided by requests served per hour |
| Kernel dispatch confirmation | A quantised model on FP16 kernels runs slower than the baseline |
| Pass or fail against pre-agreed business gates | Gates come from business impact, not a leaderboard metric |
Continuous batching is a serving technique rather than a quantisation choice. The same discipline shows whether it hurts answers: score batched and unbatched outputs on the same query set with the same quality metric. The 2024 LLM study found W8A8 formats strongest under asynchronous continuous batching, so we would test the two choices together rather than in sequence. A one-page summary necessarily drops edge cases, so keep the full benchmark method, scripts and raw results in the audit trail.
Whether a planned INT8 setting keeps every contracted channel inside its quality SLA has to be checked channel by channel. Measure a metric such as VMAF before and after rollout, because a healthy average can hide one channel in breach. After rollout, per-channel automated regression alerts catch what the pre-release test missed.
How do we prove reduced precision hasn’t changed a ranking, a tone curve or caption timing?
Run the reference precision and the reduced-precision build on the same representative inputs, score the difference with the metric the domain cares about, and compare it with a threshold set before the run.
TechnoLynx first sets the equivalence metric and acceptance threshold with the domain team. TechnoLynx then builds the reduced-precision pipeline and the parity test on the target hardware, under the Inference Cost-Cut Pack where serving cost is the driver.
Ranked hit lists in virtual screening
For mixed-precision screening, the measure that matters is rank agreement: overlap of the top-N set and rank correlation against the FP32 or FP64 reference, repeated across reruns to show reproducibility. In CNN classifiers, quantisation flipped the least confident predictions first. So inspect candidates near the cutoff individually.
Tone-mapping models versus a LUT
For an INT8 or fixed-point tone-mapping model, compare output against FP32 on representative HDR content using objective metrics such as PSNR and ΔE, with a threshold agreed in advance. Objective metrics miss some perceptual changes, so near the threshold a MOS panel or expert review still decides. The comparison can run automatically as a step in the compilation pipeline rather than by eye.
Caption timing before it goes on air
For caption alignment, measure timestamp drift between FP32 and INT8 outputs on real newsroom footage. Report the distribution and the worst case against your contracted drift limit, not just the mean. How many hours is enough depends on how varied the content is, so cover every content type that goes to air rather than a fixed small sample. The go/no-go report should state the threshold, the footage covered, the worst observed drift and the verdict.
Accuracy, throughput and power trade-offs for pathology tiling and Jetson pose models
The trade-off curve for a whole-slide image pipeline or a 6D pose network has to be measured on that pipeline, at each candidate precision, on each candidate device. Published numbers come from other models, data and pipelines, and cannot certify a clinical or grasp-accuracy requirement.
TechnoLynx produces that curve through GPU Performance Engineering. TechnoLynx defines the accuracy, throughput and (for pose) power metrics and thresholds. TechnoLynx then builds the TensorRT pipelines or quantised models on A100, L4 or Jetson Orin NX, and runs controlled benchmarks across each combination of precision and hardware. LynxBenchAI’s published results can narrow the hardware shortlist, but its fixed model catalogue does not run your pathology or pose pipeline.
Whole-slide image tiling
The detection results above suggest FP16 will land close to FP32 and INT8 will depend on calibration and model, but treat that as a hypothesis for your harness to test, not a forecast. Results from one pathology pipeline do not certify another. An A100 versus L4 comparison therefore needs the same tiling pipeline, the same slides and the same metric on both. Clinical classification outcomes may need more statistical rigour than an engineering benchmark provides.
6D pose on Orin NX
NVIDIA notes that QAT almost always gives better accuracy than PTQ. Regression and keypoint heads are a sensible place to try it first, though the margin is model-specific and needs confirming. Power draw only means something under sustained, representative load such as continuous picking, measured over a long run rather than a short burst. Calibration data and software versions both move the accuracy difference, so record them with every result.
Who can prove our INT8 model is safe to ship, and should we do it in-house?
TechnoLynx validates quantised models on your own workload and hardware. TechnoLynx profiles the serving path, quantises with PTQ or QAT, and measures accuracy, confidence, latency and cost against your baseline. You get a one-page evidence report with the full method behind it. The first step is a profiling pass on one production model to find where quantisation would pay, delivered through the Inference Cost-Cut Pack.
Whether to build this in-house depends on two things: whether your team already runs a calibrated evaluation harness, and whether someone can read TensorRT build logs and per-layer error. When choosing any partner, ask to see measurements on your hardware, kernel-dispatch evidence and per-layer diagnostics, not a format recommendation.
The engagement covers the engineering evidence; regulatory sign-off stays with your regulatory and quality functions. Where the only open question is which device to buy, LynxBenchAI’s published results are a lighter starting point.
FAQ
How many bytes does an INT8 value take?
One byte: INT8 is an 8-bit integer. TensorRT uses a signed range of −128 to 127 for INT8 activations.
What is asymmetric quantisation?
Asymmetric quantisation maps real values to integers with a scale and an integer zero point, so the integer range can sit off-centre to match a skewed distribution. Symmetric quantisation drops the zero point. In TensorRT, dequantisation is simply a multiplication by the scale.
