Our approach to CUDA port regression testing for a SLAM and obstacle-detection stack runs three checks on every run. The gate we recommend replays a fixed held-out dataset through the full pipeline, compares the ported build’s outputs with the CUDA reference within a numeric tolerance, and tests per-frame latency against a band derived from measured run-to-run jitter. A gate that fails on any deviation at all will block correct builds, because a sound port still produces floating-point differences and the hardware still produces timing noise.

TechnoLynx builds this kind of gate as a Production AI Monitoring Harness engagement. That means one automated harness that ties dataset replay, accuracy scoring and per-frame latency capture together, runs as a CI step ahead of on-vehicle campaigns, and is checked against known-good and known-regressed builds. The first step is agreeing the accuracy metric, the latency budget and the acceptable jitter with your team, then measuring that jitter on the target hardware.

Moving a perception stack off CUDA is a portability change, and like any portability work it should be tied to a trigger you can measure: a deployment-target constraint, a lock-in cost or a performance ceiling. Whether to port at all, and whether SYCL or Vulkan compute can hold real-time performance, is a separate feasibility decision. The gate described here assumes that decision has been made and asks a narrower question: did this build keep what made the stack work?

Regression testing a perception pipeline after a CUDA port: replay, compare, time

Testing a ported computer vision pipeline is differential testing: the same program and the same input run on two platforms, and the results are compared. A 2024 study of NVIDIA and AMD GPU numerics put the case for doing this continuously, since code ported between GPU platforms needs ongoing testing to confirm numerical robustness and to find where the platforms differ. We would not rely on a one-off validation at the end of the port, since later changes to kernels, compilers or drivers can reopen the gap.

The useful structure has two levels of validation. A porting study of the CReSS typhoon-simulation code checked each kernel against data dumped from the original CPU application, then validated the integrated application separately. For a SLAM stack, the kernel level catches a broken reduction or a mistranslated intrinsic before it is buried in a trajectory. The application level catches what only shows up when the full pipeline runs over time.

A model that performs in training is a different artefact from one that performs on the deployment target, and the regression suite that acts as a release gate is a reliability artefact in its own right, separate from accuracy.

Should the suite measure accuracy, per-frame latency, or both?

The suite should measure both, in the same pass. SLAMBench pairs the two: a framework that quantifies quality of result together with instrumented execution time and energy, which together capture the trade-offs for real-time vision platforms. Measuring accuracy and latency in separate jobs invites the case where each passes on a different build configuration.

How frames are fed matters. In real-time operation a slower platform drops input frames, so two builds compared at camera rate are not processing the same data. SLAMBench’s answer is a process-every-frame mode, where the next pre-recorded frame is loaded only once the previous one is done, so every platform processes exactly the same frames in the same order. A port gate should replay this way. The accuracy comparison then measures numerics rather than speed, and the per-frame latency recorded in the same pass is compared against the fixed per-frame budget.

Choosing the SLAM accuracy metric

Absolute trajectory error (ATE) can be computed at every frame against ground truth, which makes it a natural per-frame accuracy signal for a replay harness. Zhang and Scaramuzza (IROS 2018) set out its trade-off: ATE gives one easily compared number, but it is sensitive to when an error happens, so a rotation error early in a trajectory inflates ATE more than the same error near the end. Relative error is less sensitive to timing and produces a distribution over sub-trajectories (median, percentiles), at the cost of being harder to compute and to rank.

We would gate on ATE as the headline number and track relative-error percentiles alongside it, since a port bug that causes local drift can hide inside an acceptable ATE. The alignment step has to be fixed inside the harness. In the same tutorial, translation ATE differed by roughly 150% depending on how many states were used for trajectory alignment, so changing the alignment between runs would look like a regression. The obstacle-detection side needs its own detection metric, scored on the same replayed frames; the gate logic is the same.

Why does the ported build give different numbers, and how much should the gate tolerate?

The ported build differs because the same mathematics is not executed as the same sequence of operations. NVIDIA’s floating-point guide is explicit on this: individual IEEE 754 operations give identical results on any compliant hardware, but fused multiply-add, parallel reordering, unexpectedly high intermediate precision and math functions that the standard does not require to be correctly rounded all change the final value. Even the grouping of an addition, (A + B) + C versus A + (B + C), changes the rounded result. Changing the number of threads per block reorganises a parallel reduction, which amounts to moving the parentheses in a long chain of additions. The guide’s conclusion is the one a gate needs: a difference is not automatically evidence that the new result is wrong.

Across vendors the effect is measurable. That 2024 study ran 652,600 randomly generated test instances on NVIDIA and AMD GPUs and found discrepancies in 0.98% of FP64 runs, 1.10% of HIPIFY-converted FP64 runs and 9.00% of FP32 runs. The causes they traced were math library calls, FP32 versus FP64 precision and the HIPIFY conversion itself. They also warn that -ffast-math and even -O2 are not equivalent between nvcc and hipcc, so matching flags does not mean matching optimisations. One of their cases is directly relevant to iterative SLAM code: a tiny difference in an fmod result grew with each loop iteration until the final outputs differed significantly.

SLAMBench shows a benign version of the same effect. Maximum ATE ranged from 2.01 cm on the ODROID OpenCL build to 2.07 cm on the other platforms and implementations, and the authors attribute the small differences to a reduce kernel performing non-associative summations. That is the shape of a healthy port: different bits, same trajectory quality.

So the tolerance belongs on the metric the vehicle depends on, not on bitwise equality. In the CReSS study, the integrated GPU application counted as valid when both of its error metrics fell below 10⁻⁴, and it came in at 1.0×10⁻⁵ and 5.6×10⁻⁵. Their kernel-level check was not a purely automatic pass/fail test. Elements within tolerance were accepted; elements outside it went into a diagnosis loop to rule out implementation errors before anything was classed as a genuine numerical discrepancy. We would copy that pattern: a tight element-wise tolerance at kernel level as a diagnostic, and a looser application-level tolerance on ATE and detection metrics as the actual gate. Sign-off on tolerances and divergence goes deeper than one CI page can cover; our guide to validating ported CUDA kernels, including tolerance, divergence and sign-off covers it.

One lever helps on the reference side. CUDA Core Compute Libraries 3.1 lets CUB reductions choose a determinism level: not_guaranteed, run_to_run (the default, identical results on the same GPU) or gpu_to_gpu, which uses a reproducible accumulator and, according to NVIDIA, costs 20% to 30% more execution time on large problems on an H200. A reproducible reference makes the comparison cleaner. It configures reductions in CUB, the CUDA library, so the comparison with the ported build still needs the tolerance.

Which baseline should the port be compared against, and when should the suite run?

The baseline is the CUDA build’s output on the reference hardware, frozen together with the versioned held-out dataset, plus kernel input and output dumps captured from real runtime states. The CReSS team dumped kernel inputs and reference outputs from actual execution because synthetic inputs were not enough for meaningful validation of kernels fed by long, time-dependent processes. A SLAM pipeline’s map and pose state are just as path-dependent, so the same reasoning applies.

Snapshot dumps have a hard limit. In the CReSS study, a branch not executed at the dump point but executed at an intermediate timestep slipped past kernel-level validation and appeared only after integration. Application-level validation is the complement that catches errors outside the captured snapshots. For a gate, that means the full-pipeline replay is never optional, however good the kernel coverage looks.

Timing follows from that. Each ported kernel is checked against its dump as soon as it is transformed, before integration. The full replay runs on every CI build from the first integrated port onwards, and a passing gate is the condition for each on-vehicle campaign. Small, self-contained failing cases are worth keeping as a permanent suite: the 2024 GPU numerics study notes they are easy to analyse, can be handed to a vendor and can be reused when testing newer systems.

How do you set a threshold that separates a real regression from GPU jitter?

The threshold comes from the measured spread of passing runs on the same device, not from a fixed number. Daly and colleagues (ICPE 2020) found a threshold-based system insufficient for finding performance changes in a timely manner; in their CI, worst-case run-to-run variation was under 10% on most tests and 20% or more on sensitive ones. Hardware identity does not remove the problem either. An SC22 study in HPC clusters measured 8% average performance variation (22% at most) across GPUs of the same architecture and SKU, with outliers up to 1.5 times slower. We would therefore build a band for each specific board rather than for a model number.

Orin is not quiet. One forum report describes iterations on a Jetson ranging from about 0.2 ms to more than 3.4 ms with a mean around 0.53 ms. The reply attributes much of this to memory-intensive work, where DRAM, its controllers and the cores interact in ways that cannot start from an identical state each run. Where a perception pipeline is memory-heavy, we would expect the latency band to be wider than a compute microbenchmark suggests.

The two-standard-deviation band

A methodology for measuring GPU frequency-switching latency gives a usable model. It compares iteration execution times with the mean plus or minus two standard deviations, chosen over a confidence interval because it reflects the variability of individual executions rather than the precision of the mean. If times are roughly normal, about 95% of samples fall inside that band. The distinction matters for a gate. A confidence interval on the mean narrows as runs accumulate, so a gate built on one would eventually flag ordinary single runs; a band of individual-run spread does not.

The 95% figure has a cost that holds under the same normal assumption: even with nothing wrong, about one run in twenty lands outside a two-sigma band. A gate that blocks on a single excursion will block correct builds regularly, so a single excursion should trigger a rerun, and only a repeated or sustained excursion should fail. Outliers from driver management or CPU-side interruptions also need filtering before the band is computed; the same study used DBSCAN clustering to remove them. Kalibera and Jones (ISMM 2013) add the sizing discipline: put repetitions where the uncertainty actually arises (between builds, executions or iterations) and report differences as effect-size confidence intervals.

Which statistic to band is a judgement. We would band both the mean per-frame latency and a tail percentile, since a port can keep the mean and still breach the per-frame budget on its worst frames; p99 is the familiar choice for tail latency.

Change-point detection for slow drift

A per-run band misses regressions that arrive a few percent at a time. Change-point detection works on the history instead: Daly and colleagues’ E-Divisive means system finds statistically significant shifts in noisy CI results. It needs several data points, so it cannot judge the most recent run, and their regressions typically surfaced three to five days late. Hunter (ICPE 2023) adds a 5% relative-change floor so that changes too small to reproduce are not reported, and found that 14 daily points were too few while about 30 cut false positives.

Both tools have the same weak spot. E-Divisive handles independent noise well but not correlated noise, and Hunter’s authors report that results from untuned hardware still swamped it. After one team moved its benchmarks to the public cloud, Hunter went from about three change points a month, all false positives, to none. Change-point detection works as a second layer that flags trends, with the per-run band as the blocking layer. Neither saves a gate whose timing runs are not controlled.

The threshold trade-off is real in both directions. Too tight, and the gate blocks good builds; a 2020 study of embedded-system CI notes that intermittently failing tests are costly to troubleshoot and lead staff to distrust results. Too loose, and a real regression reaches the vehicle. The same study’s q-score, the fraction of a test’s verdicts that change state over a moving window, gives a direct measure of whether the gate itself has become flaky. Setting and signing off thresholds across board and model revisions is covered in our piece on benchmarking across hardware and model revisions, with isolation and threshold-setting.

Orin throttling: proving a timing run is valid before judging it

A latency result measured while the SoC was throttling says nothing about the build. NVIDIA’s Jetson AGX Orin thermal design guide describes two stages. When a sensor reaches the thermal management threshold, software polls the temperature and caps CPU and GPU clocks according to a control algorithm. If that cannot hold the temperature, hardware throttling engages as a last resort, significantly reduces performance and cannot be altered. Short power spikes that push junction temperature over the threshold cause temporary clock reductions on their own. On top of that, the sensor uncertainty is ±3 °C.

The Jetson Linux Developer Guide adds two details that matter for CI. Hardware throttling caps CPU and GPU clocks at three levels, light, medium and heavy (clock throttled percentages of 50, 75 and 87.5 in its table), and the host OS is not notified, so the drop has to be detected by sampling counters. Even MAXN does not guarantee stable clocks, because hardware throttling engages when total module power exceeds the TDP budget.

We would pin clocks before anything else. According to an NVIDIA developer forum guide, sudo nvpmodel -m 0 selects MAXN and persists across reboots, while sudo jetson_clocks locks CPU, GPU and memory clocks but resets after every reboot, so it must be reapplied. A correct configuration shows current frequency equal to maximum frequency for CPU, GPU and EMC. tegrastats reports memory and processor usage during the run, which gives the harness a log to check against. The frequency-switching methodology above checked throttle reasons every five passes and skipped frequency pairs hit by power throttling. Another forum report measured about 40 µs maximum jitter on isolated cores over a 12-hour cyclictest.

Before a timing run counts

• Power mode confirmed as the one the band was built under.
• jetson_clocks applied in this boot, and current frequency equal to maximum for CPU, GPU and EMC.
• Warm-up frames run and discarded before timing starts.
• Temperature and processor usage logged with tegrastats throughout the replay.
• No clock drop below the pinned value at any point in the log.
• No other workload sharing the board during the replay.

A run that fails any item is invalid, not failed: the gate reruns it and never records it in the band’s history.

Running the gate as a blocking CI step on embedded hardware

We would run the gate on a self-hosted CI runner on the board itself, taking one job at a time, so timing reflects the target hardware. Public guidance on hosting CI runners on Orin is thin, so the procedure below is our engineering judgement rather than a documented standard. Each gated build runs these steps in order:

1. Set the power mode, pin clocks and verify frequencies; abort as invalid if verification fails.
2. Run a canary workload whose latency is known, and treat the whole run as suspect if the canary moves, as Daly and colleagues do with their canary tests.
3. Run the kernel-level dump comparisons for changed kernels.
4. Replay the held-out dataset in process-every-frame mode, recording outputs and per-frame latency together.
5. Score ATE, relative error and detection metrics against the frozen CUDA reference and its tolerance.
6. Compare latency statistics with the device’s jitter band, filtering outliers first.
7. Check the thermal and clock log, then emit pass, fail or invalid, and block promotion to on-vehicle testing on anything but pass.
8. Append passing results to the history used for the band and for change-point detection.

We would give the ported build’s own target board the same treatment with its vendor’s clock and telemetry controls, and its own band. A band measured on Orin says nothing about another vendor’s jitter.

Gate verdicts at a glance

Observation on a build	Verdict	Action
Clocks not pinned, throttle detected or canary moved	Invalid	Rerun; exclude from history
Accuracy within tolerance, latency inside the band	Pass	Promote; add to history
Kernel elements outside tolerance	Diagnose	Check implementation and translation before classing as numerical
Application accuracy outside tolerance after diagnosis	Fail	Block on-vehicle campaign
One latency excursion outside two sigma	Rerun	Expected about one run in twenty under normality
Repeated latency excursion, or a change point above the relative floor	Fail	Block and bisect commits
Latency inside the band but tail above the per-frame budget	Fail	The budget is absolute; the band only models noise

Who can build a regression gate for our ported SLAM stack?

TechnoLynx builds jitter-aware regression gates for ported perception stacks: the replay harness, accuracy and latency scoring, the empirical jitter band on your target boards, and the blocking CI step ahead of on-vehicle campaigns. Building in-house makes sense when your team already owns embedded performance measurement and statistics. A specialist earns its place when the port team is busy with kernels and no one owns the gate. When choosing a partner, ask whether they will validate the gate against known-good and known-regressed builds and hand over thresholds you can recalibrate yourselves. TechnoLynx delivers that validation as part of the Production AI Monitoring Harness, within its AI engineering for manufacturing and automotive. LynxBenchAI measures accelerator hardware, and its published results are the tool for comparing candidate targets when the open question is which accelerator to port to.

The engagement has limits. It builds the regression harness and release gate, not safety certification or regulatory sign-off. LynxBenchAI measures hardware; it does not run your pipeline or your dataset. Teams that already have stable timing infrastructure may need only a change-point layer, and Hunter is an open-source command-line tool designed to slot into build pipelines.

When the band has to be rebuilt

A jitter band describes one combination of board, dataset, model and software stack, and it goes stale whenever any of them changes. A new BSP, a different power mode, a retrained detector or an extended dataset each need fresh characterisation runs before the gate’s verdicts mean anything again. The held-out dataset needs the same care. If it stops representing the scenes the pipeline now meets, the gate keeps passing while measuring the wrong thing.

Thermal behaviour varies across test sessions, ambient conditions and enclosures, and clock throttling under power spikes can widen jitter between sessions in ways no single characterisation captures. A gate that logs clocks and temperature on every run will at least show when that has happened, instead of reporting it as a regression.