Say the same model and the same input frame give different outputs. Candidate causes sit in distinct layers of the stack: precision, the kernels (tactics) TensorRT selects for each architecture, batching, and, in video pipelines, NVENC settings that vary with GPU architecture. Before you blame any of them, compare the gap with the run-to-run noise on each SKU, measured separately for each one. The work then runs in a fixed order: profile, change one factor at a time, pin the settings that turn out to matter, and accept the throughput that pinning costs. Latency and other GPU performance gaps between SKUs follow the same rule, because TensorRT picks tactics for speed and those tactics also change the numerics.

Why the same frame gives different bounding boxes on A10G, L4 and T4

Precision paths: TF32 on Ampere and Ada, plain FP32 on Turing

The T4 is a Turing part, while the A10G is Ampere and the L4 is Ada. TensorRT allows TF32 Tensor Cores by default: TF32 rounds FP32 multiplicands to FP16 precision, multiplies them exactly and accumulates in FP32, and TensorRT always falls back to FP32 on platforms without TF32 support. TF32 is an Ampere-and-later feature. A “FP32” engine therefore really is FP32 on the T4, but it can quietly run convolutions and matrix multiplies in TF32 on the A10G and L4.

The FP16, INT8 and TF32 builder flags permit lower precision rather than force it. TensorRT still picks a higher-precision kernel for a layer when that kernel is faster or when no low-precision version exists. By default TensorRT runs each layer in whatever precision performs best, and NVIDIA’s documentation warns this can cost accuracy.

There is a second layer of architecture dependence underneath. The internal accumulation behaviour of Tensor Core matrix-multiply units is officially undocumented, yet it produces observable differences across Volta, Ampere, Ada and Hopper. In one 2026 arXiv study of LLM inference, A100 and L40S agreed with each other on every TF32 test case and both disagreed with H100, because Ampere and Ada issue the same TF32 instruction and Hopper issues a different one. That study tested A100, L40S and H100, not A10G or L4. Whether the same pairing holds on your fleet is something to test, not assume.

For debugging only, setting NVIDIA_TF32_OVERRIDE=0 when building an engine disables TF32 even where the builder flag allows it. If the A10G/L4 gap against the T4 shrinks to the noise floor once TF32 is off, you have found your dominant cause.

Kernel and tactic selection per architecture

TensorRT picks each layer’s implementation by timing the candidates. When two candidates time almost the same, system noise can decide which one wins. Different implementations use different accumulation orders, and sometimes different precisions, so two builds of the same model typically do not give bit-identical results. Across architectures, engines are tied to the compute capability they were built for, and tactics chosen at one clock speed are not optimal at another.

Most cuDNN routines give bit-identical results across runs on GPUs of the same architecture, but no cuDNN routine guarantees bitwise reproducibility across architectures. If part of the pipeline still runs in PyTorch, torch.backends.cudnn.benchmark can pick different convolution algorithms on later runs, even on the same machine, because of benchmarking noise.

In the arXiv study’s LLM GEMM kernels, a CUTLASS-style performance-first rule that sizes the split-K factor by SM count chose different split factors on A100, L40S and H100 (108, 142 and 114 SMs) for the same shape. Their outputs differed across architectures on four of six shapes. In that study, the split factor was the GEMM tuning knob that moved the bits.

Batching and padding change the numbers too

PyTorch documents that the first element of a batched matrix multiply is not guaranteed to be bitwise identical to the same product computed on its own. Thinking Machines Lab describes common matmul kernels as deterministic from run to run but not batch-invariant: change the batch size and each element can get a different result. A serving layer whose batch size follows load therefore adds variation that looks like randomness to any single request.

TensorRT adds a shape dimension to this. It builds separate kernel implementations for each min/opt/max shape range and tunes hardest for the opt shape. A pipeline that pads frames to a different resolution, or uses a different optimisation profile on one SKU for throughput, gets kernels tuned for a different shape there. Hold batch size and the padded input shape fixed on every SKU (trtexec takes these through --shapes) before you read any remaining gap as a precision or tactic effect.

How much run-to-run variance is normal on the same GPU SKU?

For a fixed TensorRT engine on a fixed GPU, output variance should be zero. NVIDIA states that once an engine is built for a specific GPU, it should give bit-accurate outputs across multiple runs. If repeated runs of one serialised engine on one GPU disagree, check your harness first, such as preprocessing or batch composition.

Measure a wider noise floor in three layers, in order, on each SKU:

1. Same engine, repeated runs. This should be bit-identical. It confirms the harness is clean.
2. Rebuilt engines, same GPU. Build several engines from the same ONNX file and compare them, because builder timing noise can change tactics between builds. If your deployment rebuilds engines per image or per node, this spread is your real floor.
3. Batch composition, same GPU. Run the target frames inside the batch mixes production actually sees. In the 2026 arXiv study, 6–25% of BF16 LLM output streams differed across seeds on the same GPU purely from batch-composition changes.

Measure the floor for every SKU on its own, never once for the fleet. A 2025 arXiv study of LLM inference found A100s showed slightly higher hardware-induced variability than L40S under identical settings. That study did not test T4, L4 or A10G, so measure each one. Then compare the cross-SKU delta distribution (IoU shift, confidence delta per matched box) against those per-SKU distributions. Treat a delta inside the floor as normal variance. A delta outside it gets assigned to whichever single-factor test removes it.

In published LLM studies, the size of the gap depends critically on precision. That study found that across 12 configurations on A100 and L40S, BF16 greedy decoding produced up to 9% standard deviation in accuracy on one reasoning benchmark, while FP32 was close to zero. Those are LLM greedy-decoding results, where a near-tie between tokens turns tiny rounding into a different answer. Detection heads have their own thresholds (NMS, confidence cut-offs), so the principle carries over but the percentages do not.

Latency needs its own floor. TensorRT engines need a warm-up pass to load into GPU memory and trigger kernel selection, and both the baseline and the candidate should get the same number of warm-up iterations, timed with CUDA events. Torch-TensorRT’s own tuning protocol runs three warm-ups and reports the median of ten CUDA-event timings. That is a sensible minimum before you read any SKU-to-SKU latency difference.

In the LynxBenchAI 26Q3 results, the NVIDIA L4 scores 209.1 for inference overall and 534.9 for MobileNetV3 inference at FP16, with each test run as a single timed window after a discarded warm-up. LynxBenchAI’s published results show how each device runs the public PyTorch stack, so they are broadly indicative for custom PyTorch models. They cannot measure the output or latency gap in your own detector and pipeline.

How do I profile a TensorRT stack layer by layer before blaming the GPU?

Profile per layer and compare outputs per layer first, and only then say which SKU, setting or stage causes a gap. “The A10G is slower” or “the L4 drifts” are conclusions, and the layer profile is what earns them.

For timing, trtexec benchmarks an ONNX model under TensorRT with explicit control of precision, shapes and iteration count. Adding --dumpProfile gives a per-layer performance profile, and building with --profilingVerbosity=detailed plus --dumpLayerInfo shows per-layer engine detail. Diff those layer reports between SKUs. A latency gap that sits in a handful of layers, or comes with different per-layer precision, has a cause you can name. Device-level differences matter too: tactics chosen at one clock speed are not optimal at another, and in the 2026 arXiv study, peak specifications did not predict the measured ordering of the H100 and L40S. When the gap sits inside specific kernels, the next step is kernel-level profiling of the bottleneck rather than more end-to-end timing.

For outputs, Polygraphy compares TensorRT against ONNX Runtime with absolute and relative tolerances, and when network outputs do not match, it can compare per-layer outputs to find where the error enters. Its default tolerances suit FP32 and need relaxing for reduced precision. We would compare each SKU against the same ONNX Runtime reference, not SKU against SKU, because three pairwise diffs give no fixed point from which to tell which GPU moved.

Inference side or encoder side: isolating NVENC presets and SDK versions

If the pipeline encodes with NVENC, separate the stages before you blame either one. Capture the frames the model emits before they reach the encoder. If those already differ beyond the noise floor, look on the inference side. If they match and the encoded output does not, look on the encoder side. Any neural enhancement step (sharpening, denoising) belongs on the inference side of that split, so compare its raw output frames first.

On the encoder side, a preset is only the coarse level of control. Each NVENC tuning info has seven presets, P1 to P7, and a preset sets all the relevant encoding parameters at once. Rate control runs in NVENC firmware, and 1-pass and 2-pass modes spread bits differently across a frame. NVIDIA’s preset migration guide states that the newer preset and multipass values vary with GPU architecture, and Video Codec SDK 13.0 introduced Blackwell enhancements that apply across presets.

Split-frame encoding is worth checking on its own. In the default auto mode it switches on when a GPU has two or more NVENCs and the frame is tall enough (2112 pixels or more for HEVC, 2048 for AV1), and it trades quality for speed. Check how many NVENC engines each SKU has, then set the split mode, tuning info, rate control and multipass explicitly so that defaults cannot differ. Read the presets and SDK versions each of T4, L4 and A10G supports from the driver and SDK on each image rather than assuming them.

Should we pin a TensorRT engine and precision mode per GPU SKU?

Yes, once a test has shown that auto-selection moves outputs beyond the noise floor. Pin per SKU, never with a single engine for all of them. Engines are device-specific by default and fail to deserialise on a different compute capability. Hardware compatibility mode makes an engine portable by excluding architecture-specific tactics, at a cost in throughput or latency. Treat it as a portability feature, not a numerical-parity one.

NVIDIA documents the pinning tools. Build with the editable timing cache, keep the logs (which list the available and selected tactics per layer), then update the cache with chosen tactics so later builds use them. NVIDIA suggests reusing a timing cache only on the same hardware and software configuration, meaning the same CUDA/cuDNN/TensorRT versions, device model and clock frequency. For precision, --layerPrecisions constrains individual layers when precision constraints are set to obey or prefer. Under prefer, TensorRT can still use a faster implementation and issue a warning, so read the build log.

The arXiv study measured a pinned GEMM configuration at 1.4% slower than the best autotuned one at the median and 21.9% at worst. Thinking Machines Lab reported losing about 20% against cuBLAS for batch-invariant matmuls. Both come from LLM workloads, so measure the trade-off on your own detector. Bit-identical results across architectures were reached in that study by keeping every linear-layer accumulation off the Tensor Cores. Given that cost, we would aim for a bounded tolerance per SKU rather than full parity. The wider question of non-determinism on shared and rented GPUs, including whether to pin customers to one SKU, is covered separately.

Attribution table for a cross-SKU gap

Cause	How it shows	Isolating test	What to pin
TF32 vs FP32 path	TF32 available only on A10G/L4 (Ampere and later)	Rebuild with NVIDIA_TF32_OVERRIDE=0 (debug only)	TF32 flag off, or accept TF32 tolerance
Per-layer precision choice	Different layer precisions in --dumpLayerInfo	Force layer precisions with constraints set to obey	--layerPrecisions per SKU
Tactic selection	Gap changes between rebuilds on the same GPU	Compare several builds per SKU against reference	Editable timing cache per SKU
Batching and padding	Gap moves with load or batch mix	Fixed batch size and shapes via --shapes	Fixed opt shape and padding
NVENC settings or SDK	Pre-encode frames match, encoded output does not	Explicit preset, tuning, rate control, split mode	Explicit encoder config per architecture
Library or driver version	Results not guaranteed across releases	Same SKU, old vs new image	Versions pinned in the image

Keeping the explanation valid after the next image update

A gap you have explained today can return after a library or software update. PyTorch states that bitwise-identical results are not guaranteed across releases or platforms, and NVIDIA suggests reusing a timing cache only on the same CUDA/cuDNN/TensorRT combination. If T4 nodes run an older image than the L4 nodes, you are measuring two variables at once.

So the deliverable is not just the explanation. It is a fixed frame set with per-SKU noise floors and tolerances that runs again whenever a driver, SDK or engine build changes, ideally wired into the regression and shadow-traffic harness used to validate serving changes. The question left for your team is which tolerance matters downstream: a 0.5% confidence shift that never crosses a threshold can be ignored, while a smaller one that flips an alert decision cannot.

Sources

1. Algorithm Selection and Reproducible Builds — NVIDIA TensorRT, docs.nvidia.com
2. Numerical accuracy — PyTorch documentation, docs.pytorch.org
3. NVENC Preset Migration Guide (Video Codec SDK 13.0), docs.nvidia.com
4. Accelerating the Mitigation of LLM Inference Nondeterminism Across GPU Architectures (arXiv), arxiv.org
5. Improving Model Accuracy — NVIDIA TensorRT, docs.nvidia.com
6. Engine Compatibility — NVIDIA TensorRT, docs.nvidia.com
7. Odds and Ends — NVIDIA cuDNN Backend (Reproducibility/Determinism section), docs.nvidia.com
8. Reproducibility — PyTorch documentation, docs.pytorch.org
9. Defeating Nondeterminism in LLM Inference — Thinking Machines Lab, thinkingmachines.ai
10. Performance Tuning Guide — Torch-TensorRT, docs.pytorch.org
11. Command-Line Programs (trtexec) — NVIDIA TensorRT, docs.nvidia.com
12. Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference, arxiv.org
13. Polygraphy example: Comparing Frameworks (NVIDIA/TensorRT GitHub), github.com
14. NVENC Video Encoder API Programming Guide (13.0), docs.nvidia.com
15. NVIDIA Video Codec SDK 13.0 Powered by NVIDIA Blackwell, developer.nvidia.com
16. Support Matrix — NVIDIA TensorRT, docs.nvidia.com