Model quantisation to INT8 can give an embedded accelerator the latency and power headroom it needs, but average mAP is the wrong acceptance test for it. Compression losses concentrate on a small subset of inputs, and in one small-object detection study that subset was the smallest objects. To protect those cases, evaluate per class and per object size, use a one-at-a-time sensitivity analysis to find the few sensitive layers and keep them in floating point, and choose post-training quantisation (PTQ) or quantisation-aware training (QAT) based on the gap you measure. After that, design the calibration set deliberately, re-validate on the target silicon and put a fallback in place.

TechnoLynx runs quantisation and compilation validation for embedded accelerators as core compute work. The work starts by framing the accuracy tolerance and the layers and classes most at risk, so every INT8 number that follows has an honest reference.

How do we stop a small-object recall drop being averaged away in mAP?

Report the metric broken down the same way the risk is, by class and by object size, and compare each slice against the FP32 model on identical inputs. The COCO evaluation protocol already does this for size. It reports AP and AR separately for small objects (area under 32² pixels), medium objects (32² to 96²) and large objects (over 96²). About 41% of COCO objects fall in the small bucket.

The size of the effect can be large. In a Leiden University (LIACS) thesis on small-object video detection, PTQ to INT8 cut overall AP by 10.5%, from 0.181 to 0.162. AP on extremely small objects fell by 35%, from 0.034 to 0.022. A ten percent headline drop looks tolerable on its own. A one-third loss on the smallest targets is not tolerable.

Hooker et al. (2020) describe the general pattern: overall accuracy hides disproportionately high errors on a small subset of examples, and compression hits the under-represented long tail of the data distribution hardest. It does not follow that the small bucket is always the casualty. In a 2023 benchmark of PTQ reliability, large objects lost much more accuracy than small and medium ones. Which slice breaks depends on the model, so it has to be measured rather than assumed.

Two more traps are worth checking. Dutta et al. (2024) showed that a compressed model can match the baseline’s accuracy while individual answers flip from correct to incorrect and back, and they proposed flip counts and KL divergence as distance metrics alongside accuracy. Separately, INT8 can shift absolute output scores, which moves borderline samples across a fixed decision threshold even when ranking quality looks unchanged.

Which metric catches which INT8 failure

Metric	Catches	Misses
Aggregate mAP (AP@[.5:.95])	Broad regressions across the whole model	Losses concentrated in one class or size slice
AP and AR per size bucket (COCO thresholds or your own)	Collapse in small or distant object recall	Class-specific losses inside a bucket
Per-class recall and confusion matrix against FP32	Rare-class degradation and class swaps	Instance-level changes that cancel out in totals
Flip rate between FP32 and INT8 outputs	Behaviour changes hidden by equal accuracy	Which flips actually matter to the application
Recall at the deployed confidence threshold	Score shifts that push borderline cases across the threshold	Ranking quality across other thresholds

Downstream components need the same scrutiny. In the Leiden thesis, the tracker thresholds were calibrated on the FP32 confidence distribution and then reused unchanged for the INT8 models, which may have disadvantaged them.

Should a safety-critical perception model use post-training quantisation or quantisation-aware training?

Start with PTQ and move to QAT only when the per-slice gap you measured is outside tolerance and cheaper fixes have failed. Being safety-critical does not rule out PTQ. What it changes is the evidence the decision must rest on. Qualcomm’s white paper by Nagel et al. (2021) finds that PTQ is sufficient in most cases to reach 8-bit quantisation with close to floating-point accuracy, while QAT needs fine-tuning and labelled training data but enables lower bit widths. Krishnamoorthi’s 2018 whitepaper reported PTQ with per-channel weights within 2% of floating point on classification networks, with QAT closing that gap to 1%.

When QAT does help, the gains can be small but real. In a mixed-precision study on PointPillars, QAT with the three most sensitive layers kept in FP16 scored 0.63% mAP above the equivalent PTQ model and finished only 0.17% below FP32. QAT is not a guaranteed rescue on the hardest objects, though. In the Leiden thesis, QAT fine-tuning recovered none of the PTQ accuracy loss on extremely small objects, and AP fell further, to 0.106. The author points to a confound: automatic mixed precision was enabled for the FP32 baseline but disabled for QAT. We read that result as a warning that QAT has to be verified slice by slice, not as proof that QAT fails on tiny objects. QAT also costs compute and training data that are not always available.

Transformer-based BEV models at ultra-low bit widths tilt the decision towards QAT. In the QD-BEV study (2023) of BEVFormer-style multi-view 3D detection, PTQ below 8 bits caused a significant accuracy drop, W4A4 gave close to zero mAP, the backbone and encoder were the most sensitive stages, and progressive stage-wise QAT beat standard QAT by up to 5% mAP. TechnoLynx scopes BEV quantisation work around an agreed mAP retention budget for the target accelerator. It then chooses between QAT and PTQ, and picks the FP16 layer set, from a sensitivity analysis of that architecture rather than from CNN detector results.

Which layers should stay in FP16 to protect rare or safety-critical classes?

Run a one-at-a-time sensitivity sweep. Quantise a single layer, evaluate the model, rank the layers by how much each one hurts, and leave the worst offenders in floating point until accuracy is back inside tolerance. One integer-quantisation study found that MobileNet v1, EfficientNets and Transformer needed only a few of their most sensitive layers left unquantised to get back within 1% of FP32. BERT was the exception: no single layer stood out, so the authors turned to QAT. Score the sweep on the metric you care about. The PointPillars study ranked layers by AP40, and for a pedestrian model we would rank by small-object recall rather than overall mAP.

Often just a few layers carry most of the accuracy loss. In the PointPillars work, moving just one sensitive layer to FP16 lifted PTQ mAP to 48.24%, above the all-INT8 PTQ model, and three FP16 layers reached 63.84% without any retraining. Each extra FP16 layer adds latency, because data has to be converted between FP16 and INT8 at every boundary. First and last layers are often left at higher precision because they are more sensitive, sensitivity also varies with layer type and position, and in transformers, keeping the FFN output projections, residual summation inputs and attention output projections in FP16 brought accuracy back close to its original level after global W8A8.

Tools can automate the search. NVIDIA’s Polygraphy debug tooling iteratively marks subsets of layers for higher precision to find which ones a TensorRT network needs, and HAWQ picks per-layer precision from each layer’s Hessian spectrum. Cheap proxies deserve scepticism. A 2026 cross-platform study found that per-layer SQNR had essentially no rank correlation with the accuracy loss (Spearman ρ = −0.030 over 21 layers), while a 2025 microcontroller survey recommends per-layer SQNR to guide mixed sub-byte precision. The two sources disagree, and we would trust a measured task metric over either proxy. Check that the target can actually run mixed precision before you plan around it. Most edge hardware requires uniform bit widths across operations.

For a pedestrian detector on an automotive SoC such as the R-Car V4H, start the sweep at the high-resolution feature levels. In the Leiden thesis, P3 and P4 are the levels where objects under 32 pixels mostly appear, and the author argues quantisation noise there is most damaging to them. That makes the neck and the small-object detection layers our first FP16 candidates. Pair them with a calibration set stratified by pedestrian size, distance and occlusion, and check that the target compiler supports that layer split before validating recall by size bucket on it.

For small, distant traffic signs, start from per-channel weight quantisation, the recommended baseline scheme. Krishnamoorthi and NVIDIA’s TensorRT documentation both recommend per-channel weights with per-tensor activations. Add QAT on the detection head where profiling shows broad sensitivity, and track a small-sign recall slice separately from mAP. TechnoLynx starts this work by agreeing the size thresholds against your sign taxonomy. It then profiles layer-wise error in the head and backbone before choosing between mixed precision and QAT.

How many validation cases do we need to compare FP32 and INT8 outputs credibly?

The case count comes from a power calculation on the metric and margin you must defend, counted where the risk sits: near the detection floor and at the earliest, hardest stage. Counting across the whole validation set is not enough. TechnoLynx scopes this kind of re-validation with the client’s QA owners. It defines the FP32-versus-INT8 objective (overlap metrics such as Dice or IoU, plus recall stratified by defect size), sizes any near-threshold augmentation with a power calculation, and hands over the validation evidence package that QA or the regulator then reviews.

Re-establishing the sensitivity floor after quantisation

Reuse the pilot study’s dataset and statistical method if the dataset already holds enough cases near and below the old floor. The method should carry over unchanged. Only the comparison changes, from FP32 against ground truth to INT8 against FP32 on the same cases. Because both models see identical inputs, a paired design applies. McNemar’s test on detected-versus-missed outcomes, or a non-inferiority test against a margin fixed in advance, fits the question better than comparing two independent accuracy figures. The number of cases needed depends on the detection rate near the floor, the margin you will accept, the significance level and the power you want.

Check whether the original challenge set holds enough samples right at the threshold to probe a floor that may have moved. If it does not, augment it with seeded or real near-threshold defects. Size the augmentation by the power calculation, and only use seeded defects whose morphology matches real early-stage defects, because otherwise the new floor will not generalise. Confirm with the regulator or QA before the run that they will accept a reused dataset and method. Even with that agreement, plan a field validation, since a lab-measured detection rate still has to hold under real deployment conditions.

Measuring the delta on the earliest, hardest instances

Break out precision and recall by size or life stage for both models, so that the change on the smallest annotations is visible on its own. Hooker et al. found that compression amplified the false-negative rate more than the false-positive rate. When a small-defect slice comes back degraded, look at the calibration set first. If it held only common cases, the quantisation ranges were never shown the rare ones.

What calibration set stops INT8 scales fitting only the common cases?

Calibration chooses the clipping range for each tensor. The usual methods are max (the largest absolute value seen), entropy (minimising KL divergence, TensorRT’s default) and percentile clipping, and no single method is best for every network. For rare cases, the method can matter more than the number of samples.

Two studies push in opposite directions, and the difference between them is instructive. In an aerial person-tracking detector, a moving-average calibrator washed out the rare large values from a heavy-tailed box-size head. That cut AP25 to 0.361 against 0.694, while true min-max calibration brought INT8 to within 0.008 AP25 of the float model. In a DETR comparison across two vendors, the opposite happened: a per-tensor min-max scale was set by the single largest activation, so one outlier image coarsened every other value, and an EMA calibrator that damped the outlier did better. Our reading is that the right calibrator depends on whether the rare extremes are signal (large boxes the model must output) or noise (one odd frame). Inspect which kind you have before choosing.

Size and composition have measurable effects, though not always the expected ones. A 2023 PTQ reliability benchmark found that larger calibration sets gave more stable results: going from 1 to 32 samples cut the standard deviation on one CIFAR-10 class from 1.22 to 0.58. The same benchmark found that average accuracy barely moved with calibration composition, noise or class imbalance, while individual categories were sensitive to all three. The PointPillars study, in contrast, got 23.82% higher mAP calibrating on four frames than on 4,096. In that study, the effect of calibration size was linear and consistent across easy, moderate and hard samples (Pearson correlation above 0.998), so tiny calibration sets did not penalise the hard cases specifically. That is one model’s result, not a rule.

Simply mixing in degraded images did not broadly improve robustness in one YOLO12 study either. Calibrating YOLO12 models with a 50/50 mix of clean and degraded images performed much like clean calibration for most model sizes and degradations. A sensible default is to sample uniformly across all sequences, so that every object size and scene type is represented, as the Leiden thesis did with 500 frames, and then to check the rare slices explicitly. That author notes the calibration set may still have under-represented the tail of extremely small objects. Whether one calibration set can serve every station, store or sensor is something to test across sites, not assume.

Accelerator targets can make the choice of calibrator matter even more. On NVIDIA’s DLA, standard entropy calibration produced 75% accuracy against a 94.5% FP32 baseline, while structurally derived manual dynamic ranges recovered 94.0% with no calibration data.

The same logic holds for a model whose job is ranking the tail, such as a risk score served on a T4 GPU. Use per-channel INT8 with the tail-sensitive layers left in FP16 or FP32, calibrate on data that actually contains tail cases, and accept the result only if tail discrimination holds against the full-precision baseline.

What runtime fallback should sit behind a quantised model in production?

Run the full-precision model in shadow and continuously measure its disagreement with the quantised model, weighted towards the regions where errors are hazardous. Switch traffic to full precision automatically when that disagreement crosses a threshold agreed in advance. TechnoLynx treats this as production AI reliability work, distinct from model accuracy. The deliverables are the regression suites, the disagreement and drift monitors and the release gates that catch a quantised model degrading before customers do.

The disagreement metric matters more than the plumbing. Flip counts and KL divergence between the two models’ outputs reveal behaviour changes that matching accuracy figures hide. Weight those metrics by hazard: safety-monitoring research shows that a perception model’s errors are only dangerous in some state configurations, such as particular speeds and distances to other vehicles, so hazard analysis can locate the regions worth watching most closely. For a margin or collateral engine, the tail of the distribution plays that role. Revisit the thresholds over time, because covariate shift degrades model performance as the operating environment changes.

The costs need to be planned, not discovered. Shadowing a full-precision model adds its compute to that path, and that cost has to be budgeted. On edge hardware, we would compare sampled frames against FP32 off-device and log low-confidence cases instead. Fallback speed depends on how quickly the serving infrastructure can reroute traffic, and alert thresholds should be calibrated against historical false alarms so the fallback does not fire constantly. Redundant architectures have one specific trap. Two heterogeneous compute units running the same INT8 model should not be assumed to agree on every input: they do under FP32 but may not under INT8.

Can an existing embedded GPU card run the quantised model in real time without visible artefacts?

It may be able to, but you have to measure it. If the FP32 model only reaches the target frame rate on a desktop-class RTX GPU, INT8 or FP16 on an embedded card can still fall short, because the card’s compute ceiling, not the number format, may be the limit. TechnoLynx starts this kind of job by confirming the real-time target and measuring the gap between the FP32 model on desktop hardware and the embedded card. It then quantises and compiles the model for the card and runs a per-class, per-region regression harness against the FP32 baseline.

Nobody can promise a frame rate before that measurement, because the speed-up from quantisation is not guaranteed. In one study on a consumer RTX 3050, quantisation changed the numerics but delivered no real speed-up, because hardware support and kernel implementation determine the gains. On a Jetson Orin Nano Super, INT8 gave a modest additional energy reduction over FP16 but no throughput improvement. In the other direction, the PointPillars study saw a 2.538× latency reduction against FP32 on an RTX 4070 Ti with one FP16 layer in an otherwise INT8 model. The spread across these results is the reason to measure on your own card.

In ReTiDe, an INT8 video denoiser compiled for FPGAs (2025), errors introduced by PTQ degraded denoising most in smooth background regions, and QAT fine-tuning brought quality nearly level with FP32. The same work found that quantisation error became the dominant factor under low-noise conditions. So the footage that most needs checking is clean, low-noise material where editors and viewers will notice. Training-serving skew can also mimic or mask quantisation artefacts, which is why the FP32 baseline has to run through the same production pipeline as the INT8 model. If the card still misses the target after quantisation, the options are a lighter architecture or different edge hardware.

The same FP32-versus-INT8 comparison also serves as the diagnostic for what to protect. Run per-class confusion matrices for both models on identical inputs, then combine them with the one-at-a-time layer sweep to see which rare-class heads should stay in FP16 or FP32.

How much power, latency and thermal headroom does each quantisation level cost on the target accelerator?

Only measurement on the target answers this. Simulator and desktop-GPU numbers are not substitutes, and thermal and power behaviour in particular can only be confirmed on the real device. Good practice is to build the TensorRT engine directly on the hardware it will be evaluated on and to re-evaluate detection accuracy after the engine is built, rather than assuming it matches the PyTorch checkpoint. TechnoLynx’s AI Porting & Deployment Pack covers this step. It starts by pinning down the target’s memory, compute, power and thermal budget and the accuracy and recall bands that must survive compression, then quantises, compiles and benchmarks the candidate model on the device itself. Before a port exists, LynxBenchAI shows in its published results how candidate devices run the public PyTorch stack your models are built on, which is broadly indicative for narrowing a hardware shortlist. Those results do not measure your quantised model on your exact silicon variant.

Does the architecture fit a Cortex-M33-class MCU at all?

Check that some architecture class fits the memory and compute budget before you spend more effort on quantisation. If none does, switch family, for example to a depthwise-separable CNN or a TCN. Mixed precision can be what makes a model fit. On an STM32H7 with 512 KB of RAM, a hardware-aware mixed-precision MobileNetV2 reached 68.4% accuracy, beating every 8-bit model that fitted the same memory budget. Toolchains limit what is possible here: frameworks such as TFLite Micro allocate a fixed-size static tensor arena, which rarely accommodates mixed-precision layers.

Benchmarking on the ADAS SoC before the compile chain freezes

Benchmark the vendor-native quantisation path on the real SoC against a recall, latency and power test set agreed before the run. If the tolerance band is set after the numbers arrive, the sign-off has nothing to stand on. The vendor path matters because accelerators own their quantisation. In the 2026 portability study, one vendor NPU silently ignored externally supplied quantisation scales, so accuracy fell from 0.75 to 0.005 while the model compiled and ran without error, and another vendor’s compiler refused the same graph outright.

Drone and field boards: measure the envelope, do not infer it

Which board is better depends on precision and power mode, so the comparison has to be run per configuration. In AgriJetsonBench (2026), the preferred Jetson changed with workload, precision and operating mode. For precision-agriculture vision, the authors conclude that deployment decisions need accuracy, latency, tail latency, board power, energy per inference and thermal stability reported together. A budget-based estimate can be checked against a device measurement. On a Raspberry Pi Zero 2W, one tracking pipeline measured 31.4 ms per frame, within 6% of its 29.6 ms budget.

Fleet devices: re-test after every model update

Yes: re-test every update on the target before it goes to the fleet. INT8 behaviour depends on the target and its calibrator as well as on the model. The 2026 portability study’s recommendation is to re-validate INT8 on each target rather than once. For a retail or field fleet, that means an automated on-device suite run under representative thermal conditions as a release gate.

A model quantisation workflow that holds accuracy inside an agreed tolerance

Fix the tolerance in the domain’s own terms before starting, whether that is pedestrian recall, shrink rate, clinical sensitivity or tail-risk ranking. Then escalate from PTQ to mixed precision to QAT only as far as the measured gap requires, and accept the result only on the compiled engine, evaluated on data that matches production conditions.

TechnoLynx runs this workflow through its AI Porting & Deployment Pack, which quantises, compiles and benchmarks the model on the target device and validates it against the FP32 reference, and the first step is agreeing the accuracy and recall bands that must survive compression.

Escalation procedure, in order

1. Agree the acceptance metrics (per class, per size bucket, at the deployed threshold), the tolerance against FP32 and the held-out test set before generating any INT8 numbers.
2. Apply PTQ to all compute-heavy layers (convolution, linear, matrix multiplication) and try max, entropy and 99.99% or 99.999% percentile calibration.
3. If no calibration meets the target, run a sensitivity analysis and leave the most sensitive layers in floating point.
4. If that costs too much performance or still misses the target, start QAT from the best calibrated model and fine-tune for about 10% of the original training schedule, with an annealing learning rate starting at 1% of the original.
5. Build the engine on the target hardware and re-evaluate accuracy on that engine, not on the checkpoint.
6. Run the equivalence test against FP32 on production-representative data, stratified by condition (lighting, angle, site, class), with the pass criterion fixed in advance.
7. Lock the calibration set and precision configuration into the release pipeline, so that every future model update goes through the same harness automatically.

The paper behind steps 2 to 4 reports that those steps kept accuracy within 1% of floating point on every network studied, including hard cases such as MobileNets and BERT-large. Techniques tested mainly on ImageNet and COCO leave open how they generalise to specialised domains such as medical imaging or autonomous driving, so the tolerance check has to run on your hardest real subclasses as well as on benchmark data.

Keep preprocessing fixed throughout, because it can swamp the quantisation effect. In the 2026 portability study, changing the resize and crop method moved top-1 accuracy by 1.07 points, roughly nine times the 0.12-point cost of quantisation itself. For claims of statistical equivalence, a two-one-sided test or a non-inferiority test against a pre-agreed margin replaces “close enough by eye”. Where a low-confidence result routes to a human, as with a cashier confirm step, set that confidence threshold from the INT8 model’s score distribution rather than reusing the FP32 one.

TensorRT has its own failure modes. QAT models export to ONNX with each fake-quantisation op as a QuantizeLinear and DequantizeLinear pair. NVIDIA recommends quantising the inputs of weighted operations but, by default, not their outputs, since a following activation may need higher-precision input. TensorRT 10.x has deprecated implicit INT8 quantisation, so INT8 engines need explicit Q/DQ nodes. In one DLA deployment, leaving the skip-connection quantiser out made residual additions fall back to FP16 and cascaded 36 layers onto the GPU, and that team checked every build log for zero GPU fallback layers. Apply the same rule on Jetson: build and evaluate the engine on the hardware you ship. Our piece on validating quantised models for edge deployment and serving efficiency covers the latency and cost side of the same evidence.

Memory is the other half of fitting the device. Accelerators such as NVIDIA’s DLA support only a restricted set of operators, so unsupported ones have to be replaced with native equivalents before quantisation. Add pruning or distillation only if quantisation alone misses the budget, and measure RAM headroom with the full firmware loaded, not with the model on its own.

What if the model does not fit the power and latency budget even after INT8?

Try targeted mixed precision before abandoning quantisation. After that, the options are a smaller backbone or a distilled student, then pruning, then a different target. TechnoLynx’s Inference Cost-Cut Pack covers model quantisation to INT8 or FP16, by post-training quantisation or quantisation-aware training, for a model that has to fit a tighter budget.

Speech separation within a hearing-aid power and latency envelope

In a study of RNN speech enhancement on an MCU, uniform INT8 degraded PESQ by 0.3 on average. A mixed scheme with INT8 recurrent layers and FP16 for the rest cut the loss to 0.06 while saving 1.4 to 1.7× memory, without QAT. The recurrent layers there held up to 84% of the parameters, and their bounded activations suited INT8.

A recipe for an ADAS SoC that keeps small-object recall

Limit FP16 to the layers the sensitivity sweep flags, because each additional FP16 layer adds conversion latency. Then measure on the SoC rather than estimating.

A compression path when the model does not fit as-is

Distillation can make the smaller model survive INT8 far better. In a 2026 vulnerable-road-user study, a YOLOv8-L teacher lost 23.4% mAP under INT8 while a distilled YOLOv8-S student lost only 5.6%. Distillation has a ceiling, though: the student could not match the teacher on rare classes, whatever the training method.

Running more pipeline stages on the MCU

The most durable INT8 gains on edge devices come from per-channel weight quantisation, bias correction, cross-layer equalisation and quantisation-aware fine-tuning, according to a 2025 survey of microcontroller quantisation. Validate each added stage’s accuracy against the full pipeline rather than in isolation.

Fixed-point INT16 DSPs without GPU hardware

Fixed-point formats offer less dynamic range than floating point and need careful calibration to avoid overflow or underflow. A 16-bit integer grid has far more levels than INT8, so the first question on a DSP is whether INT16 alone meets accuracy before anything more aggressive is tried.

Who can quantise and validate our perception model for an embedded accelerator?

TechnoLynx quantises, compiles and validates perception and edge models for the embedded accelerator you have already chosen, and produces the per-class and per-size evidence a release gate needs. When choosing a partner, ask to see per-bucket results, the calibration design and an evaluation of the on-target engine, not an aggregate mAP. TechnoLynx runs this work through the AI Porting & Deployment Pack.

The engagement hands over validation evidence for regulator or QA sign-off, not the sign-off itself. The realistic alternatives are the silicon vendor’s own tools and application engineers, which matter more than usual because vendor NPUs own quantisation and can silently ignore externally supplied scales. A team with in-house expertise can also use NVIDIA’s TensorRT Model Optimizer, which handles INT8 calibration, QAT and PTQ and exports to ONNX.

FAQ

How many bytes does an INT8 value use?

One byte. Signed INT8 covers −128 to 127 and unsigned INT8 covers 0 to 255.

What is the difference between symmetric and asymmetric quantisation?

Symmetric quantisation assumes values are centred on zero and fixes the zero-point at zero, which simplifies the hardware. Asymmetric schemes add a zero-point that shifts the integer grid to match the distribution, which reduces bias for data that is not centred.

Is dynamic quantisation better than static quantisation for embedded deployment?

Dynamic quantisation computes scales at runtime and often gains accuracy at extra compute cost. Static quantisation fixes the ranges during calibration, so inference is efficient at the cost of some accuracy.