A TensorRT engine only loads on the device type and TensorRT version it was built with, and NVIDIA’s hardware compatibility mode is not supported on JetPack. In practice, moving a model to a Jetson target means rebuilding the engine on that target.
TechnoLynx runs this rebuild-and-validate pass for each Jetson target, starting with GPU performance profiling of the current engine on the device, so decisions come from your model, not a generic benchmark.
One engine, one Jetson, one TensorRT release
A serialised plan records the major, minor, patch and build version of the TensorRT library that created it, plus the compute capability of the GPU. If either differs at load time, the plan will not deserialise. TensorRT can build engines that later TensorRT versions can load, but NVIDIA notes that these version-compatible engines can be slower than engines built for one specific runtime.
The calibration side is a little more forgiving. NVIDIA’s TensorRT documentation says an INT8 calibration cache can move between devices when calibration happens before layer fusion (the IInt8EntropyCalibrator2 or IInt8MinMaxCalibrator path). That lets you build the table on a discrete GPU and reuse it on an embedded platform. The same documentation says the cache is generally not portable across TensorRT releases. Calibration also depends on input size: the AgriJetsonBench study on Jetson Orin (arXiv, 2026) did not reuse a 640-pixel calibration cache at any other resolution.
ONNX Runtime’s cached TensorRT engines are likewise not portable. Its documentation says cached engine and profile files must be deleted whenever the model, the ONNX Runtime version, the TensorRT version or the hardware changes. A misaligned BSP, driver, CUDA or TensorRT stack fails less visibly. TensorRT can then still compile and load, but falls back to slower kernels, with latency spikes and small output differences.
Taken together, a Jetson engine is a build output tied to the GPU, TensorRT version, precision and shape profiles. It is not a file you copy from a workstation GPU.
What do we do when TensorRT cannot map some of our model’s operators?
Four fixes recur for an unsupported operator: re-export or fold the graph so the operator goes away, let that node fall back to another runtime, rewrite it with supported operators, or write a plugin. The check has to be repeated for each toolchain, because the set of unsupported operators differs from one runtime to the next, and so do the substitution strategies.
TechnoLynx gives you a per-target port plan: it inventories the model’s custom operators, especially any custom head, against each toolchain’s supported-operator list. For each one TechnoLynx chooses between native export, a custom plugin or a CPU fallback, and picks INT8 or FP16 per toolchain based on accuracy risk.
When an engine build fails, the cause is usually one of two things: GPU memory runs out, or the ONNX parser meets a node it cannot map to a built-in kernel. NVIDIA’s opset guide recommends running Polygraphy constant folding before you debug parser errors, because this clears many shape and constant-subgraph problems. If an operator really is unsupported, the guide points to simplifying the graph with ONNX-GraphSurgeon, changing the opset with the Version Converter, or adding a custom plugin. A TensorRT build-failure guide on markaicode adds that re-exporting at a newer opset often helps, because TensorRT 10 and 11 cover more opset 17 and 18 nodes natively.
If the model stays in PyTorch, Torch-TensorRT splits the graph and runs any operator without a converter in eager PyTorch between TensorRT subgraphs. Its dry-run mode shows where those boundaries fall before you commit to a full compile. The Torch-TensorRT documentation advises leaving an infrequent fallback alone if it runs correctly. It suggests a custom converter when the operator can be built from TensorRT layers, and a plugin for custom CUDA or Triton kernels. ONNX Runtime works the same way at the graph level: it sends only supported subgraphs to TensorRT, and NVIDIA’s guidance is to register the CUDA execution provider as well, so the remaining nodes still run on the GPU.
Fallback has two costs. The first is speed. A 2026 cross-platform INT8 study (arXiv) found the Jetson DLA handled ResNet-50 cleanly, but DETR fragmented into 326 DLA layers and 404 GPU-fallback layers, running about 30 times slower than the same model in FP16 on the integrated GPU. The second is that fallback can be silent. The same study recorded a TensorRT execution provider that was listed as active but was actually running on the CPU, with p50 latency dropping from 11.83 ms to 0.41 ms once that was fixed. For that reason, “the engine built” is the start of the audit, not the end.
Custom heads, such as path-planning outputs, unusual decoders or bespoke post-processing inside the graph, are where we would expect unsupported operators and accuracy regressions to cluster. Support also changes between TensorRT versions, so an operator audit has to be repeated. Data-dependent-shape operators such as NonMaxSuppression are one example: ONNX Runtime says TensorRT runs them by default from 10.7, while 10.0 to 10.5 had a known performance problem with them.
Hailo and Qualcomm: the accelerator owns the quantisation
Vendor NPU toolchains add a separate risk: they own the quantisation. The 2026 cross-platform study found that Qualcomm’s Hexagon HTP silently ignored externally supplied QDQ scales. Top-1 accuracy fell from 0.75 to 0.005 even though the model compiled, profiled and ran without any error. ONNX Runtime’s QNN documentation adds that QNN operators typically support uint8 and uint16, even though the quantisation utilities also offer int8 and int16. On Hailo, the Dataflow Compiler converts and quantises the model into a hardware-specific HEF on an x86 build machine. Ultralytics reports INT8 HEF retention against PyTorch of about 91% to 100% for YOLOv8n and YOLO11n across four tasks, on validation splits such as COCO8-pose and COCO128-seg, and advises improving calibration data first when accuracy drops.
The Ultralytics figures cover standard YOLO models, not custom heads. Until an operator audit has been run, treat that as an unknown, not a reassurance. If the open question is which Jetson module to buy rather than how to port to it, LynxBenchAI’s published results show how each device runs the public PyTorch stack. That is a broad guide for custom PyTorch models, but it does not measure your own model or application.
A detector-and-tracker pipeline on Jetson that holds up in production
A detector and tracker on Jetson starts from a known pipeline pattern. Backbone size and quantisation effort should then follow the FP32-to-INT8 difference per class and per distance band on your own data. A model-zoo table cannot make that decision for you.
TechnoLynx would begin by confirming the latency and power budget, the safety-relevant classes and the distance ranges that matter. TechnoLynx would then export to ONNX, build FP16 and INT8 engines on the target Jetson, integrate the tracker and measure accuracy differences by class and distance next to on-device latency and power.
The known pattern is ONNX export followed by an engine build, either through native TensorRT or through ONNX Runtime’s TensorRT execution provider. One published Jetson tutorial uses the TensorRT execution provider with INT8 calibration data on JetPack 6. It pins CUDA 12.2, TensorRT 8.6 and ONNX Runtime 1.17 to match. Engine and timing caches matter on-device: ONNX Runtime measured session creation for one UNet model at 384 seconds with no cache and 9 seconds with an engine cache. A pipeline someone else has shipped is a pattern to start from. It does not guarantee your model, your tracker or your Jetson SKU will behave the same way.
How much accuracy does INT8 lose against FP16 or FP32 on a Jetson?
On published YOLO evidence, FP16 costs almost nothing and INT8 costs several mAP points, and smaller models lose more. An arXiv study of YOLO12 (2025) found FP16 TensorRT engines had negligible accuracy loss compared with FP32. Static INT8 TensorRT engines were about 1.5 to 3.3 times faster but dropped about 3 to 7 absolute points of mAP50-95. A study in Electronics (2025) on Jetson Orin Nano measured YOLOv8s at 0.8608 mAP50-95 in FP32, 0.8622 in FP16 and 0.7968 in INT8, with 27.0, 37.9 and 41.2 FPS respectively. That is a 6.5-point INT8 loss against FP16 (0.8622 − 0.7968) for a 3.3 FPS gain.
| Precision on Jetson | What the published evidence shows | When we would choose it |
|---|---|---|
| FP16 | Negligible mAP loss against FP32 for YOLO12 and YOLOv8 | First choice when it meets the latency and power budget |
| INT8 (PTQ) | About 3–7 mAP50-95 points lost on YOLO12; 6.4 points against FP32 for YOLOv8s on Orin Nano | When FP16 misses the budget and per-class retention passes on your data |
| INT8, sensitive layers in FP16 | Running YOLOv5’s last three convolutions in FP16 raised DLA mAP from 35.9 to 37.3, with throughput falling from 410 to 252 FPS | When the accuracy loss sits in a few output layers |
| INT8 on transformer detectors | DETR INT8 lost about 44% of mAP on Jetson in the 2026 cross-platform study | Only after a dedicated measurement shows it holds |
Score quantisation on the full validation set, not a subset. The 2026 cross-platform study found that a 1,000-image subset overstated top-1 accuracy by 9.77 percentage points on average, and reversed the sign of three quantisation deltas. Our separate page on validating quantised models for edge deployment and serving efficiency covers how to build that accuracy, latency and cost evidence package. Report the difference per class and per distance band, not as one mAP figure. AgriJetsonBench, for example, accepted an INT8 detector only when mAP50-95, mAP50 and AR100 each fell by no more than 0.03 against the FP16 engine on the same board, and no important class AP collapsed.
Post-training quantisation or quantisation-aware training for Jetson?
Start with post-training quantisation (PTQ), and move to quantisation-aware training (QAT) only when no calibration method and no partial-precision setting reaches the target. NVIDIA researchers (2020) recommend this order: PTQ trying max, entropy and 99.99% or 99.999% percentile calibration, then partial quantisation that keeps the most sensitive layers in floating point, then QAT fine-tuning for about 10% of the original training schedule. Some architectures cannot skip that last step. NVIDIA reports EfficientNet-B0 reaching 33.9% top-1 with PTQ and 76.8% with QAT, against a 77.4% floating-point baseline.
On Jetson, the choice also depends on the target engine. NVIDIA’s YOLOv5 Orin example notes that DLA did not support QAT through TensorRT at the time of writing, so the QAT model had to be converted to a PTQ model before inference. TensorRT also lets you set layer execution and output precision directly. That is the practical tool for partial quantisation once a sensitivity analysis has found which layers cause the loss.
Which images belong in the INT8 calibration set?
The calibration set should be representative of the scenes the camera will actually see. NVIDIA’s TensorRT documentation says the amount of data needed depends on the application, and that about 500 images were enough for ImageNet classification networks. Ultralytics’ Hailo guidance warns that out-of-domain calibration images can lower quantised accuracy.
In one study, adding synthetic degradations to calibration did not make INT8 engines more robust. The 2025 YOLO12 study calibrated with a 50/50 mix of clean and synthetically degraded images and found no consistent robustness gain over clean calibration, apart from the largest model under noise. The same study found Static INT8 engines more sensitive to Gaussian noise than FP32 or FP16. If your sensors produce noisy frames, that degradation has to be part of the validation set, not only the calibration set. A rare class that barely appears in the calibration set can lose accuracy without the overall metric showing it.
Should we fine-tune a smaller backbone instead of fighting quantisation?
Not by default, because a smaller backbone does not necessarily quantise better. The YOLO12 study found smaller models more affected by INT8. On Orin Nano, YOLOv8n dropped from 0.8422 to 0.7190 mAP50-95 under INT8, a 12.3-point loss, while YOLOv8s lost 6.4. A smaller model can also lose recall on small or distant objects, which is exactly what collision avoidance depends on.
The decision logic we would follow: if the current backbone meets the latency budget in FP16, ship FP16. If it needs INT8, try PTQ and then QAT, and check retention by class and distance. Only if critical classes or ranges still fail would TechnoLynx retrain a smaller or quantisation-friendly backbone and measure again. Repeat the result for each board and power mode. AgriJetsonBench found the preferred Jetson platform changed with precision and operating mode.
How do we catch output changes from the TensorRT conversion before they reach the field?
You catch them with an automated parity check against golden reference outputs. It compares the TensorRT engine’s outputs with the PyTorch reference. It compares per-class metrics against an agreed tolerance band and blocks every export that fails.
TechnoLynx fixes the validation set, the pre- and post-processing parity and the per-class tolerances with the team before any export work starts. TechnoLynx then builds the PyTorch-versus-TensorRT comparison harness and wires it into CI/CD as a gating stage, with defined escalation and rollback for an export that fails.
Conversion does introduce errors. A study of failures in ONNX model converters (Jajal et al., 2023, revised 2024) found that the node-conversion stage accounted for about 75% of defects. It also found that 33% of reported failures involved semantically incorrect models. Some failures raise no error at all. The 2026 cross-platform study recorded a zero-copy output-buffer aliasing bug that collapsed top-1 to 0.0014 with no error raised. It also found that INT8 predictions differed on about 4% of inputs. Those flips were net-neutral, so a standard accuracy report would not show them.
A parity gate in five steps
- Freeze a labelled validation set and generate golden reference outputs from the original PyTorch model, the first stage of the deployment flow described in a 2026 study of PyTorch-to-NPU conversion.
- Compare tensors with Polygraphy, for example
polygraphy run model.onnx --trt --onnxrt. Loosen--atoland--rtolfor reduced precision, because the default tolerances suit FP32. - When outputs diverge, compare per-layer outputs with
--trt-outputsand--onnx-outputsto find where the error first appears. - Score the engine’s predictions per class against agreed tolerance bands, and pass the export only if every class stays inside its band.
- Store the result alongside the engine’s metadata: JetPack/L4T, CUDA and TensorRT versions, nvpmodel mode and clock policy, as AgriJetsonBench did for reproducibility.
NVIDIA says a built engine should give bit-accurate outputs across runs on the same GPU. When accuracy differs between two builds of the same model, a poor tactic choice may be the cause, which the Editable Timing Cache lets you inspect and correct.
There are two limits to this gate. Per-class tolerances are a domain decision, set by the people who carry the cost of a missed detection. TechnoLynx engineers the gate to enforce tolerances agreed with the team. And lab parity does not prove field behaviour. The Electronics (2025) authors note that their indoor test bed could not fully replicate outdoor lighting or environmental interference. For backend parity, such as CUDA against ROCm or a ported kernel against its original, see proving kernel equivalence after porting or precision changes.
Will a Jetson module throttle after hours of continuous inference?
It can. NVIDIA’s Jetson Linux developer guide says MAXN is not the maximum performance mode, because hardware throttling starts when total module power exceeds the TDP budget. On Orin NX and Orin Nano, average and instantaneous power limits on the VDD_IN channel trigger hardware clock throttling of the GPU and CPU. NVIDIA also advises against running heavy workloads in MAXN for long periods, unless the module is flashed with the more conservative -maxn thermal configuration. Super Mode raises the requirements further: on Orin NX it needs a carrier board with an HV rail and a thermal design that supports 40 W.
At +40 °C in a climate chamber, a synthetic stress test throttled one Jetson. A YOLOv8s TensorRT FP16 workload in the same conditions sustained 108.8 FPS at 19.1 W, with chip temperatures well below the threshold. That result applies to that model, enclosure and power mode, and to nothing else.
Validating sustained throughput means soaking the actual engine for hours in the intended nvpmodel mode, inside the real enclosure, at the worst ambient temperature you expect. During the soak, read the over-current event counters NVIDIA exposes under /sys/class/hwmon/hwmon<x>/oc*_event_cnt. AgriJetsonBench treated an external inline power logger as the primary energy reference and tegrastats as a diagnostic channel, because internal rail coverage differs between boards and modes. The same study only counted a run as valid if there were no failed inferences, unexpected reboots, OOM events or over-temperature shutdowns. State the timing boundary too, whether pure engine or full pipeline, because the two numbers measure different things. For turning that telemetry into a power mode and clock policy you can defend, see from thermal telemetry to trustworthy edge-inference configuration.
What happens when NVIDIA retires our JetPack version?
When a JetPack branch reaches end of life, NVIDIA stops shipping new releases for it. NVIDIA scheduled JetPack 5 to reach end of life in Q3 2026 and recommended that Orin platforms on it move to JetPack 6 or the upcoming JetPack 7.2. NVIDIA’s Jetson FAQ lists JetPack 6 as sustaining for Orin, and JetPack 7 as supporting Jetson Thor, with Orin support planned for 2026.
A reproducible build path means freezing the complete image, not the phrase “latest NVIDIA JetPack”. One Jetson lifecycle guide lists what that record should name: the module, board configuration, L4T release, kernel, bootloader, CUDA, cuDNN, TensorRT, drivers, container runtime, root filesystem, model artefacts and security configuration. The same guide says to record the production module SKU and not just the kit name, which matters if development happened on a Jetson AGX Orin developer kit. Decade Defense describes building engines inside containers that mirror the target BSP and refusing to load an engine whose recorded CUDA, cuDNN or TensorRT versions differ from the runtime. Failing at load time is better than running half-optimised.
A branch migration is a re-qualification event, not a package update. If the TensorRT version changes with the branch, the calibration cache generally has to be regenerated. The engine has to be rebuilt on the new stack. The lifecycle guide adds that the update path should be tested from every deployed baseline, not only from the newest lab image, with inference, thermals and power validated again after each branch change. If the parity gate and the thermal soak described above are already scripted, a JetPack migration becomes a scheduled rerun with a pass or fail result, instead of a rebuild nobody can reproduce. For sourcing, validation and fleet management across a running deployment, see deploying and maintaining the Jetson Orin NX in production.
