Matching overall accuracy does not prove an INT8 model is as good as its FP32 original. The two models can post the same score while disagreeing on many individual inputs and losing ground on particular classes. To test for parity, run both precisions on the same images, build confusion matrices per class and per object-size bin, and judge the difference with a paired statistical test against a tolerance you fixed before the run. Run that test on the original validation set and on freshly labelled real-world samples. Before you blame precision for any gap, check the quantisation pipeline itself for calibration and tooling bugs.

Why isn’t matching accuracy enough to sign off an INT8 model?

Accuracy is the ratio of correct predictions. It says nothing about whether they are the same answers. Quantisation shifts the model’s decision boundary slightly, so inputs that sit close to the boundary can cross it. In one study of compressed language models, correct-to-wrong and wrong-to-correct changes were roughly equal in number.

Dutta et al. (Microsoft Research, 2024) found that compressed language models usually stayed within 2% of baseline accuracy, while the share of individual answers that changed could reach 5% or more. In one case, 8-bit Bitsandbytes quantisation of Llama2-70b came within 0.55% of baseline accuracy and still flipped 4.6% of answers. Vision models show the same thing. A 2022 study of quantised models recorded 216 disagreements on DenseNet-121 with no change in accuracy at all.

The divergence also depends on hardware. A 2026 cross-platform measurement study held one ONNX file and its quantisation scales fixed and ran it on several targets. FP32 predictions were bit-identical on every pair of targets (1000/1000). INT8 predictions agreed on only 958 to 965 of 1000 inputs whenever two targets used different integer kernels, and top-1 accuracy stayed the same. A standard accuracy report would not show any of that.

Averages also hide per-class damage. A 2023 benchmark of post-training quantisation (PTQ) found that average accuracy held up when the calibration set changed, but accuracy on individual categories did not. In object detection, large objects lost much more accuracy than small and medium ones. Overall accuracy fell modestly from 0.9115 under FP16 to 0.8879 under INT8, while recall on the thin peduncle class fell from 0.75 to 0.6786. The comparison in that study was FP16 against INT8, not FP32, but the lesson holds: the class you care about can move much more than the headline number.

How do we tell a calibration or tooling bug from genuine INT8 precision loss?

Compare the INT8 model with an FP32 or FP16 reference tensor by tensor, not only at the final output. Divergence concentrated in a few tensors marks those tensors to leave unquantised or to recalibrate with another method. Small, spread-out deviation that still costs accuracy points to calibration coverage or a real precision limit.

TechnoLynx covers INT8 quantisation done to lower inference cost through the Inference Cost-Cut Pack.

ONNX Runtime calls this process quantisation debugging: compare the weights and activations of the original graph with those of the quantised graph, find where they differ most, and then either leave those tensors unquantised or change the calibration method. It provides Python APIs that pair each float32 tensor with its quantised counterpart. Its documentation also warns that graph optimisation during quantisation can change the graph so much that tensors become hard to match. In the TensorRT ecosystem, Polygraphy can save the inputs and outputs from one run and compare a later run against them with absolute and relative tolerances. That works across frameworks, platforms and TensorRT versions. Either way you need access to intermediate tensors, not only the final detections, and on some deployed runtimes that access has to be planned.

Failure patterns that look like precision loss but are not

• Calibration method. For a ResNet-34 classifier on a Jetson Orin NX DLA, PTQ entropy calibration gave 75% accuracy against an FP32 baseline of 94.5%. Manual dynamic ranges derived from the network structure brought accuracy back to 94.0%. The authors found 99.99th-percentile calibration worked best for their model class. NVIDIA reports EfficientNet-B0 at 33.9% top-1 with PTQ against 76.8% with quantisation-aware training (QAT) and 77.4% in floating point.
• Silent non-quantisation. In a 2026 study of bitsandbytes INT4 quantisation, the library left the output projection at full precision by default, whatever the skip-list argument said. The quant-regress CI tool documents a similar trap: architectures built on transformers.Conv1D would have left the transformer body in FP32 and reported a 0.00-point drop and a PASS. A parity result that looks too clean deserves a check that quantisation actually happened.
• Preprocessing drift. In the 2026 cross-platform study, changing the resize and crop choice moved top-1 by −1.07 points, about nine times the −0.12-point cost of quantisation itself.
• Non-reproducible calibration. NVIDIA states that TensorRT calibration is deterministic only for the same input in the same order on the same device, and that the calibration cache is generally not portable across TensorRT releases.

Is the calibration set representative of real conditions?

PTQ assumes the calibration set and the test data come from the same distribution. Activation ranges have to be measured on data that represents the real inputs. Accuracy on individual categories is sensitive to noise and bias in the calibration set, even when the average holds. Calibration coverage should therefore be checked per category, not only by average accuracy.

More images help only up to a point. YolactEdge reported 24.4 mAP with no calibration images, 29.6 with five, and a plateau after that. On CIFAR-10 and ImageNet classifiers, the 2023 PTQ benchmark found that going beyond 32 calibration samples did little to reduce per-class variance. Adding degraded images to calibration is not a reliable fix either. A 2025 YOLO robustness study found that a 50/50 mix of clean and degraded calibration data performed about the same as clean data for most models, even though static INT8 engines were consistently more sensitive to noise than FP32.

Measure accuracy again after the engine is built. Do not assume it matches the source PyTorch checkpoint. Confirming that the software and hardware paths are comparable is a prerequisite for the calibration audit, not a replacement for it.

Symptom in the parity run	Likely cause	First check
Large divergence confined to a few layers	Quantisation or calibration of those tensors	Layer-wise tensor comparison against FP32
Zero or near-zero difference that looks too clean	Layers not actually quantised	Inspect the loaded model’s layer precisions
Gap appears only on one device	Integer kernel difference between targets	Rerun with the same artifact and fixed scales on both targets
Gap tracks one noise condition	Calibration coverage, noise sensitivity	Stratified rerun by condition
Gap shrinks when preprocessing is pinned	Preprocessing, not precision	Fix and report the resize/crop pipeline

How do you compare FP32 and INT8 confusion matrices class by class and size bin by size bin?

Run both models on exactly the same test inputs. Build a confusion matrix for each model, and next to it a paired 2x2 table comparing the two models’ predictions.

TechnoLynx starts by defining the defect-class and size-bin strata needed for a statistically meaningful result. TechnoLynx then reports, per class and size bin on the target edge runtime, where INT8 diverges from FP32.

You need two different tables. A standard confusion matrix lists one model’s true and false positives and negatives. A McNemar table is a 2x2 table that compares the predictions of two models with each other. Its diagonal cells count inputs where both models were right or both were wrong. The off-diagonal cells, where only one model was right, show how the two models actually differ. If you report the confusion-matrix entries as fractions of each true class, every entry behaves like a sensitivity, which makes FP32 and INT8 easy to compare cell by cell.

Worked procedure for a per-stratum comparison

1. Freeze one checkpoint, one preprocessing pipeline and one input set for both precisions, so that precision is the only variable.
2. Split the test set by class and by object size, since accuracy drops differ across both.
3. Run FP32 and INT8 on identical inputs and log every prediction against its input.
4. For each stratum, build both confusion matrices and the paired 2x2 table of discordant inputs.
5. List the inputs that FP32 caught and INT8 missed, and look at what they have in common.
6. Test each stratum against the margin set in advance (see the next section).

In a 2020 study of American Sign Language recognition, the INT8 version of one model mislabelled 181 Y’s as I’s and 410 H’s as G’s. Those letter pairs look alike in ASL, and the INT8 version of a second model made neither error. In the 2022 quantisation study, disagreements had closer top-1 and top-2 probabilities than other inputs, and that margin was the best of the uncertainty metrics tested at separating disagreements from normal inputs. A 2026 explainability study found that images FP32 classified correctly but INT8 misclassified usually came with a shift in the image regions the model relied on, rather than the same regions with a different label. In the orchard study, the degraded class was the peduncle, where fine detail matters. Object size does not always work in the expected direction, though: the 2023 benchmark found large objects more vulnerable than small ones. Bin by size and measure. Do not guess.

The breakdown has limits. Each bin needs enough examples for the matrix to mean anything. A class with no measured loss can still degrade on production cases the test set does not contain. Deploying different precision policies for different classes also adds operational complexity.

What statistical test lets you say INT8 and FP32 are equivalent, not just similar?

Use two tools for two questions. McNemar’s test on the discordant pairs asks whether the two models perform differently. An equivalence test with bounds fixed before you see the data, such as the two one-sided tests (TOST) procedure, asks whether any difference is small enough to ignore.

TechnoLynx designs and runs this protocol on a team’s own model and target hardware. The first step is framing the tolerance margin and sampling plan, before any comparison is run.

The distinction matters because a non-significant McNemar result is not evidence of equivalence. It may only mean the sample was too small to detect a difference. TOST provides evidence of equivalence only by rejecting two null hypotheses, that the difference is below −c and that it is above c. Without that interval the procedure cannot succeed. Choosing the interval is not a statistical question. Domain experts or regulatory guidance should set it, and it should be fixed independently of the data. For an inspection line, the margin should come from the operational cost of a miss, such as an acceptable change in missed-defect rate per class.

McNemar’s statistic is χ² = (B − C)² / (B + C), where B and C are the two discordant cells. Edwards’ continuity-corrected version, χ² = (

B − C

− 1)² / (B + C), is the one most often used today. The chi-squared approximation assumes reasonably large off-diagonal counts. The corrected version gives almost the same p-values as an exact binomial test once both B and C exceed 50. For a rare class with only a handful of discordant pairs, the exact binomial version is the safer reading.

Pairing is not a formality. A 2026 study of bitsandbytes quantisation reported that an unpaired Fisher’s test found no difference between FP16 and INT8 in any of three models. McNemar’s paired test on the same trials found a real INT8 penalty in two of them (p = 0.018 and p = 0.004). When you test many classes, adjust the significance threshold for multiple comparisons, for example with a Bonferroni correction. Report effect sizes as well, because a large sample can make a practically meaningless difference statistically significant.

Worked example: zero discordant pairs, and what that does and does not prove

A 2026 paper in Computers, Materials & Continua compared FP32 and INT8 versions of a 1D-CNN intrusion detector on 40,000 clean test samples. Both discordant cells were zero, so the two models made identical predictions (p = 1.0). The authors call this “measurement-precision-limited equivalence” rather than identity: any difference is bounded to roughly ±0.05 percentage points by 95% Wilson intervals. On 1,000 adversarial samples, the same pair showed 13 inputs correct in FP32 and wrong in INT8 and none the other way, a significant difference (p < 0.001). The authors note the limits themselves: smaller differences can hide behind sample size, and real-world distribution shift can expose sensitivity that clean data does not.

How many samples per class make a defensible parity report?

Size each stratum for the precision you need. Do not size the whole test set and assume the classes will follow. A 2012 sample-size study for classification models found that 75 to 100 test cases are usually needed to test a good but imperfect classifier. To get a confidence interval no wider than 0.1 at an expected 90% sensitivity, it needed 140 test samples, and comparing classifiers generally took hundreds of independent cases. Paired designs need fewer samples than unpaired ones. For scale, Recht et al. (ICML 2019) note that 10,000 test points give a Clopper-Pearson 95% interval of at most ±1% on overall accuracy.

For small proportions, one study suggests keeping the relative margin of error, the half-width divided by the proportion, between 0.1 and 0.5. As a worked definition: for a class whose miss rate is 1%, a relative margin of 0.5 means a half-width of 0.005. A margin of 0.05 would be fine for a proportion near 0.1 and meaningless for one near 0.001. Among the intervals compared in that 2021 study, the Wilson interval reached the target performance with the smallest sample.

Is the original validation set still valid ground truth?

The original validation set is necessary but not sufficient. It gives you the baseline equivalence claim. Fresh, labelled samples from the deployment environment show whether that claim holds in the real world. Recht et al. (ICML 2019) rebuilt ImageNet’s test set by following the original collection process closely and measured accuracy drops of 11% to 14%. Northcutt et al. (2021) estimated label errors averaging at least 3.3% across ten common test sets, and at least 6% in the ImageNet validation set. In the 2022 quantisation study, test data with distribution shift produced more FP32-versus-INT8 disagreements than in-distribution data in 82% of cases. The ASL study kept a separate set of 720 noisy-background images specifically for generalisation testing.

Fresh labelling has its own annotation noise, which needs controlling. Three lighting buckets are a sample of the deployment’s variability, not all of it.

Which classes should stay in FP16 or FP32 after an INT8 parity test?

Keep at higher precision any class or stratum that fails its pre-set margin. Options include keeping the layers that hurt the at-risk class in floating point, or moving to QAT.

NVIDIA’s 2020 integer-quantisation study notes that a few quantised layers often cause most of the accuracy loss. It proposes a one-at-a-time sensitivity analysis: quantise one layer at a time, then leave the most sensitive layers in floating point until accuracy is acceptable. Its recommended workflow runs PTQ with max, entropy and percentile calibration, then partial quantisation, then QAT for about 10% of the original training schedule. Measure sensitivity on the at-risk class itself. The 2026 cross-platform study found per-layer SQNR, over 21 layers, had essentially no rank correlation with the top-1 delta, so it did not support ranking layers by SQNR. In the YolactEdge study, converting the prediction head to INT8 always cost a large amount of segmentation accuracy, and the head and FPN were kept in FP16.

Tooling supports this. By default TensorRT picks tensor types for speed, which can lose accuracy where a layer needs higher precision. Convolution, deconvolution and fully connected layers can take INT8 input and produce FP16 or FP32 output to preserve accuracy. Polygraphy’s debug precision tool marks subsets of layers at higher precision, one iteration at a time, to find which ones must stay there.

Profile the speed you are actually buying before you accept any accuracy cost. On a Jetson Orin Nano Super, a 2026 fruit-detection study found INT8 gave almost no latency benefit over FP16 (6.073 to 6.254 ms against 6.157 to 6.181 ms), while mAP@50:95 dropped by 0.0213 to 0.0393 from FP32. On that device, FP16 was the better trade.

Parity result for a class or stratum	Action
Fails, and the layer audit shows a pipeline fault	Fix calibration or tooling
Fails, and sensitivity analysis isolates a few layers	Keep those layers in FP16/FP32
Fails broadly across layers	Try QAT before giving up on INT8

Re-running the parity test on every target and release

An INT8 parity result applies to one engine on one target, not to the model in general. The 2026 cross-platform study recommends re-validating INT8 per target rather than once. NVIDIA warns that calibration caches are generally not portable across TensorRT releases.

LynxBenchAI publishes results showing how each device runs the public PyTorch stack, so a team choosing targets first checks those results for the devices it plans to ship on, then re-validates INT8 on each one it keeps.

TechnoLynx implements and optimises INT8 conversion for Jetson and other edge GPU targets. That includes the TensorRT calibration workflow, a comparison of calibration methods for minority-class accuracy, and the per-class FP32-versus-INT8 gate weighted by the cost of a miss.

One repeatable version of this test is a CI regression suite that reruns the comparison whenever a model is re-quantised.

Who can run an FP32-to-INT8 parity validation on our own model?

TechnoLynx runs this validation for a team’s own detector or classifier, on the hardware it will ship on. TechnoLynx designs the per-class and per-size strata, sets the pre-agreed equivalence margin and sample plan, audits the quantisation pipeline for bugs, and reports which classes can safely run in INT8. Whether to build this in-house depends on whether your team has both the runtime instrumentation and the statistics to design the test before the data arrives. The work sits within TechnoLynx’s computer vision R&D practice. When the hardware itself is still undecided, LynxBenchAI’s published results show how each device runs the public PyTorch stack. They are broadly indicative for custom PyTorch models, but they do not measure your own model.

Some teams do not need this engagement. If you only want a headline accuracy number, or your team already runs ONNX Runtime debugging and Polygraphy comparisons with a statistician involved, building it in-house is realistic.

FAQ

What do confusion matrices show?

In a confusion table normalised by true class, the diagonal entries are the sensitivities and the other entries are the specificities for each particular misclassification.