A thermal power validation protocol worth trusting does three things. It runs the real workload at the hottest ambient the unit will see, it keeps running until the enclosure stops heating up, and it logs clocks, power, temperature and throttle flags for the whole run. A short benchmark finishes before the chassis has warmed through, so it reports the best case, not the field case. Pin the software stack, and rerun the same protocol every time the model, firmware or power mode changes. The pass test is easy to state: sustained throughput and inference latency still meet the requirement with headroom left.

What does a bench test need to reproduce to count as worst-case in-vehicle conditions?

A worst-case bench reproduces four things at once. These are the hottest air at the mounting location, the sun load reaching the unit, the airflow and mounting the installation really gives it, and the most power-hungry operating mode the product runs. Every degree of extra ambient is a degree less headroom between the die and its limit.

Sun load is more than extra ambient. MIL-STD-810H Method 505.7 notes that solar radiation causes directional heating and thermal gradients, and that how much heat is absorbed depends on surface properties such as colour and roughness. Its Cycle A1, representing the most extreme hot deserts, peaks at 1120 W/m² and 49 °C, though the two peaks do not happen at the same time of day. The method also warns that equipment fully exposed to the sun can go above 60 °C when the surrounding air is only 35 to 40 °C. When the component of concern sits behind a windscreen or other glazing, the method calls for a full-spectrum light source rather than plain heating.

Mounting and airflow change how hot the unit gets. Method 505.7 asks for the test item to be mounted as it will be deployed, because its orientation to the source changes the heating a lot. It specifies an airspeed of 1.5 to 3.0 m/s, and no less than 0.25 m/s where the deployed item is shielded from wind. The airspeed must be measured and recorded, or the test cannot be repeated.

Vehicle electronics have their own climatic reference. ISO 16750-4:2023 specifies a high-temperature operation test of 96 hours at Tmax, using IEC 60068-2-2 Test Bd for heat-dissipating devices, and the device must stay fully functional (status class A). Its temperature cycling for small, lightweight devices runs 30 cycles. Each cycle holds 90 minutes at Tmin and 110 minutes at Tmax. A hot-soak variant adds a tabulated increase to Tmax, and the device must not be operating while it is above Tmax. AEC-Q100 grades set operating temperature ranges for integrated circuits: Grade 2, for example, covers −40 °C to +105 °C ambient. A part’s grade tells you the temperature range the part is qualified for. It does not tell you what an inference module can sustain inside your enclosure under load.

Will our bench numbers actually predict what the OEM measures in-vehicle?

Bench estimates should be verified against measurements on a real installation as one becomes available. The ZVEI Robustness Validation handbook builds requirements from a Mission Profile, which represents all relevant conditions the module will meet over its life. The handbook is explicit that a Mission Profile is not a test description: it is the basis for design, test engineering and simulation. It also asks for the assumptions behind it to be checked by measurement in the real application as installations become available, including temperatures at the module package and of the components inside it.

The handbook defines the OEM-relevant temperature as the air 1 cm from the module package at its mounting location. We would put a sensor at that point on both the bench and the vehicle, so the two traces can be compared directly. Where turning a field load into a lab load is too uncertain, the handbook recommends using proven standards.

Simulation estimates conditions before physical parts are built. It is a bridge to the bench measurement, not a replacement for it, and it is only as good as the simulation model. A repeatable bench does not guarantee a first-time pass either. That depends on how much margin the design has relative to the limits.

Writing the soak protocol once and rerunning it on every change

A documented soak protocol reproduces the worst-case ambient and mounting, runs a sustained load and logs telemetry throughout. With it, the hardware and software teams agree the thermal margin once and then rerun the same test cheaply after every firmware, model or hardware revision. TechnoLynx builds this kind of protocol for edge accelerators as part of its GPU Performance Engineering work. The work starts by capturing the duty cycle and the pass/fail thermal and power criteria both teams need, then moves on to the rig, the telemetry logging and the regression rerun.

The soak procedure, step by step

1. Write down the duty cycle and the environment, including the hottest air the computer will sit in (cabinet temperature and sunlight included).
2. Fix the configuration: on Jetson, the power mode set with nvpmodel stays in place until it is changed, including across power cycles.
3. Mount the unit as deployed, and measure and record the airflow.
4. Run the real application, or a stress tool such as stress-ng, until the chassis reaches a steady temperature. For a fanless enclosure this can take well over half an hour.
5. Log clocks, power and temperature for the whole run, with tegrastats on Jetson.
6. Read the throttle evidence: on NVIDIA GPUs, nvidia-smi’s Clocks Event Reasons report what is pulling clocks down, and Jetson Orin exposes over-current event counters in sysfs, which record power-limit throttling events.
7. Allow for the real ambient. Use a climatic chamber where you can, or account for the gap between the test room and the hottest day at the installation.
8. Compare against the requirement: does the sustained speed still meet the frame rate, cycle time or latency the application needs?

A 2026 climate-chamber study of a Jetson AGX Orin in an outdoor enclosure found that a synthetic CPU and GPU stress test substantially overestimated the thermal and power demand of a YOLOv8s inference workload running in TensorRT FP16. A synthetic load suits a conservative margin check, while reported latency and throughput should come from the real model.

Proving thermal margin for a sustained duty cycle such as 150 kW charging

Margin here means the gap between the worst-case steady-state temperature and the throttle threshold, measured while the target workload runs. NVIDIA defines total module power as the average power dissipated while the target workload runs under worst-case conditions in steady state. Its Orin NX and Nano thermal guide gives a worked example. It assumes 55 °C ambient and targets 95 °C at the SoC, which leaves 4 °C for sensor inaccuracy and workload-driven fluctuation. At 25 W, the required heat sink is (95 − 55) / 25 = 1.60 °C/W. That arithmetic sets the heat-sink requirement before the enclosure is built. The bench soak then checks that budget on real parts at the maximum environmental temperature. For battery or system pre-conditioning, we would start the soak from the hottest state the unit can realistically be in when the duty cycle begins, not from a cold lab bench.

Can we automate the thermal and power re-check on every model update?

Yes, provided the protocol is built for it from the start. That means scripted conditions, a fixed workload set, the same telemetry fields, and an automated comparison of each run against a stored baseline trace. We would split it into two tiers: a shorter steady-state check of power draw and latency on every model update, and the full soak on release candidates or whenever power draw goes up. A protocol validated for one duty cycle does not carry over unchanged to a materially different charging rate, SoC or session length. Those changes need a fresh validation.

How do we document and sign off the protocol once?

Agree the acceptance criteria with the user of the result before the first run. The ZVEI handbook describes the user and the supplier setting requirements and acceptance criteria together, based on a defined Mission Profile. It prefers testing to failure, or to an agreed level of degradation, over test-to-pass plans, because a pass/fail plan says little about failure modes and failure points. For the record itself, Method 505.7 asks for chamber temperatures, test item temperature against time, and wind speed. MLPerf Power asks submitters to report cooling solutions, power management techniques and environmental conditions alongside the power figures. A repeatable validation protocol like this acts as a regression suite for thermal and power behaviour. It is not a safety certification or a regulatory sign-off. Turning the run records into material an auditor or procurement panel will accept is covered in building a defensible evidence pack for audit and procurement review.

One measurement standard for a Jetson, a Coral board and a server GPU

Comparisons across hardware classes need every device measured under the same rules. MLPerf Power measures full system power: for edge devices, the whole system-on-chip plus any peripherals that cannot be powered down. For edge inference it requires a SPEC-certified power meter. It sets a minimum measurement duration of 60 seconds and loops shorter workloads until they reach it. That 60 seconds is a minimum for averaging power. It is not thermal steady state; the soak runs until the chassis reaches a steady temperature.

For power-limited edge deployments, we would report performance per watt at a fixed latency target as the main procurement metric. Raw throughput is the better headline only when power is not the binding constraint. A lower-TDP processor that runs at full speed all day can deliver more than a faster one that spends the afternoon throttled. For shortlisting devices before you own them, LynxBenchAI publishes results showing how each device runs the public PyTorch stack your own models build on. That makes them broadly indicative for custom PyTorch models, though they do not measure your model or your enclosure.

Back-to-back duty cycles, such as qualifying followed by a race

When sessions run back to back, the second one starts from whatever heat the first one left behind. The soak should therefore run the whole sequence without a cool-down in between. Method 505.7 likewise repeats continuous 24-hour cycles, at least three and up to seven, until the peak response temperature stabilises. For a street circuit in direct sun, the bench equivalent is Method 505.7’s solar loading and airflow, run through the full session sequence. Standardise the bench on the worst case in the mission profile, and use real highway or track runs to check that profile’s assumptions.

How long must a thermal soak run before throughput and latency figures count as steady-state?

Until the temperature stops rising, not for a fixed time. For a fanless enclosure that can take well over half an hour. Method 505.7 treats its 24-hour solar cycles as stable when the peak response temperature repeats within ±2 °C. Figures recorded before the temperature plateaus describe a system that is still warming up. If a fault shows up in the field after four hours (a frame drop on a medical console under surgical drapes, for example), the bench run should be at least that long. It also has to reproduce the ambient air and the console’s orientation, since both change the unit’s temperature.

How do we pin JetPack, driver and TensorRT versions so field units match the qualified unit?

Treat the qualified software stack as part of the hardware. Flash every field unit from one versioned baseline image that carries the exact JetPack, driver, TensorRT and power-mode configuration used in qualification. TechnoLynx sets this up as a single qualified baseline image with JetPack, driver and TensorRT locked to fixed versions. Around it sit a defined set of tegrastats or jtop telemetry fields and soak tests at the hottest expected ambient, which measure the worst-case latency distribution against the cold-start figure.

Pinning matters because TensorRT engines are tied to their build. NVIDIA’s TensorRT support matrix states that, unless built in hardware compatibility mode, engines do not carry across GPU architectures, and serialised engines never carry across platforms such as Linux and Windows. Compatibility across major versions is limited to two documented combinations: TensorRT 10 version-compatible engines on a TensorRT 11 runtime, and TensorRT 8 version-compatible engines on a TensorRT 10 runtime. The same document advises deserialising during a controlled startup, requiring a non-null engine and checking the TensorRT logger before accepting traffic. Build that startup check into the image and you have a cheap guard against a unit quietly running a different engine.

Power mode belongs in the image too. NVIDIA notes that MAXN is not the maximum performance mode, because hardware throttling engages once total module power exceeds the TDP budget. It also advises against running heavy workloads in MAXN for long periods, unless the module is flashed with the -maxn configuration and its more conservative thermal settings. A model that performs in a development environment is a different artefact from one that performs on the deployment target, and closing that gap is engineering work.

Nominally identical boards still vary from one to another. An EdgeSys’21 study of 13 nominally identical Jetson AGX boards found variability between boards in both runtime and power for the same ML inference workloads under the same conditions. NVIDIA’s AGX Orin thermal guide tells designers to build in margin for variation from module to module. We would qualify on several units, not one, and size the margin to the worst unit.

Bounding worst-case latency at the hottest ambient, not from a cold start

A cold-start benchmark finishes before the chassis has warmed through, so it reports the best case. Orin NX and Nano modules hold their specified clocks up to 99 °C at the SoC, software clock reductions start there, and shutdown is at 105 °C. Power fluctuations that push junction temperature briefly above the threshold also cause temporary clock reductions. If software throttling cannot hold the temperature, hardware throttling engages as a last resort. It cuts performance significantly, and its settings cannot be changed. Allow for the temperature sensor’s accuracy of ±3 °C when deciding how much headroom to keep.

To set the bound, soak in a chamber at the hottest expected ambient. Record latency percentiles over the steady-state window and compare them with the cold-start run. Take the worst observed value plus a stated margin as the bound. In the field, log the same telemetry fields at a low rate, so a temperature creeping towards the trip point raises a flag before a missed event does. Budget the logging overhead and storage on the device. A bound set in controlled soak testing can still be broken by a combination of ambient, enclosure and workload that the test matrix did not cover. How accuracy and throughput change once throttling sets in is covered in characterizing accuracy and throughput under thermal throttling.

When thermal throttling pushes inference latency past the target

Whether a GPU stays inside its thermal and power envelope through a multi-hour live workload is settled by a soak that copies the real rack’s ambient and airflow for the full duration. A vendor’s thermal specification cannot answer it. TechnoLynx scopes this as an extended soak of the real-time ASR model on the candidate GPU node. The soak reproduces the outside-broadcast truck’s rack ambient and airflow for a full broadcast length, logging temperature, power draw and clocks against the caption-delay target.

A short burst test reports the best case, because short benchmarks finish inside the turbo window and throttling shows up once the chassis has warmed through. NVIDIA’s thermal figures rest on stated conditions. NVIDIA defines total module power for the AGX Orin under worst-case conditions, with the target workload, in steady state. Its Jetson Orin heat-sink example assumes a 55 °C ambient. Check those assumptions against the truck’s rack before using any vendor number.

Throttling onset shows up in the GPU’s clock event reasons. On NVIDIA discrete GPUs, software thermal slowdown cuts clocks once the GPU is hotter than its Max Operating Temp, and hardware thermal slowdown cuts core clocks by a factor of two or more. Where the T.Limit sensor is supported, nvidia-smi reports the remaining margin to the maximum operating temperature as a countdown, and at 0 °C or below the GPU may start adjusting its clocks for thermal reasons. Halving the clocks on a streaming ASR model is the kind of event that pushes caption delay past its target. The soak exists to show whether that happens.

If the soak shows stable temperature and power and no latency breach, the GPU is validated for that rack. If throttling pushes latency past the target, there are three ways out: improve the cooling to match the vendor’s assumptions, choose a lower-TDP or better-cooled variant, or reduce the model’s compute so it draws less sustained power. The result belongs to the truck, season and enclosure it was measured in, and how throttling affects latency depends on the ASR model and its streaming configuration.

Should we reuse an existing edge benchmark harness or build our own?

Reuse a harness when it already covers your accelerators, precisions and metrics. Extend it with a thin layer of your own when it does not, for example to add power sampling on a particular board. LynxBenchAI publishes results showing how each device runs the public PyTorch stack, a common reference for shortlisting hardware across vendors. It does not run your model, your data or your enclosure. NVIDIA’s own Jetson test plan uses its jetson_benchmarks repository to produce reference TensorRT performance numbers for the Jetson AGX Orin.

Whichever harness you use, lock down the model, data, environment and measurement protocol before comparing vendors or runs months apart. The EdgeSys’21 study shows what that looks like. It timed inference with CUDA Events over 105 iterations, discarded the first five as warm-up, and sampled power at 0.5 Hz for 30 seconds after warm-up. It also fixed frequencies with an nvpmodel profile and the userspace governor, disabled non-essential OS services, and used the same software stack on every board.

The same study kept boards below 50 °C, running the fan at full speed, so that throttling could not affect its results. That is the right choice when comparing silicon and the wrong one when qualifying a field unit, where the throttle behaviour is exactly what you need to see. For loaned hardware tested months apart, record or freeze driver, runtime and firmware versions, and set each vendor’s board to documented power modes and clocks.

The same rule applies to a workstation purchase. For an RTX 6000 Ada and a studio’s heaviest VR scene, we would trust a trial or vendor-run benchmark of that scene file on the card over generic published results.

What telemetry should we log continuously, and how do we log it without disrupting a live system?

Log temperature, power, clocks and throttle reasons at a fixed interval with the vendor’s own tools, such as tegrastats or nvidia-smi. Keep heavyweight kernel profiling for offline sessions. TechnoLynx builds this as a scripted, versioned profiling harness. A background sampler runs during live operation and raises alerts on unusual thermal, power or throttle patterns, and a separate Nsight Compute workflow handles offline root-cause sessions. Both layers are rerun after every model or hardware change.

Telemetry fields and where to read them

Signal	Jetson Orin	NVIDIA discrete and datacenter GPUs
Temperature	tegrastats reports processor block temperatures from the thermal zone nodes	nvidia-smi temperature, plus T.Limit margin where supported
Power	tegrastats current and average power per rail in milliwatts; INA3221 monitor via sysfs	nvidia-smi average and instantaneous power draw
Clocks	tegrastats GPU activity and GPC frequency	nvidia-smi dmon SM and memory clocks
Throttle evidence	Over-current event counters in sysfs (oc*_event_cnt), which record power-limit throttling events	Clocks Event Reasons and their microsecond counters; DCGM clock event reasons

On Jetson, tegrastats --interval <ms> --logfile <file> & runs the logger in the background, with a default interval of 1000 ms. On discrete GPUs, nvidia-smi dmon can write CSV output. The nvidia-smi daemon can be configured to record power and thermal violations; its default monitoring cycle is 10 seconds and can be changed from the command line. The specific counters, such as clocks_event_reasons_counters.sw_thermal_slowdown, can be queried directly. NVIDIA’s Orin thermal guide asks for thermocouples on monitored components to keep case temperature below the Tcase limit during validation.

Logging during a live broadcast should keep interference with the workload to a minimum. MLPerf Power recommends out-of-band measurement to keep interference with the ML job to a minimum. TechnoLynx measures the harness overhead against production headroom before installing it on a truck. The sampling interval is a trade-off between overhead and missing short throttle events. Nsight Compute sessions do not belong in a live broadcast window. Run them offline on the same hardware revision once telemetry has flagged an anomaly. Turning these logs into deployment settings is the subject of from thermal telemetry to trustworthy edge-inference configuration.

Who can build a worst-case thermal and latency validation rig for our edge GPU?

TechnoLynx builds worst-case thermal, power and latency validation for edge accelerators. TechnoLynx captures your duty cycle and pass/fail criteria, builds the bench harness and telemetry logging on your target hardware (Jetson Orin, an automotive SoC or a discrete GPU), checks the bench traces against field observations, and packages the protocol as a rerun for every model or firmware revision. Whether to do this in-house depends on whether your team already has the telemetry tooling and chamber access. In our view the harder part is deciding what counts as worst case and steady state. When choosing a partner, ask how they define steady state and how they handle variation between units. The work sits in TechnoLynx’s GPU Performance Engineering offer, and it starts with framing your duty cycle and latency target.

This is engineering work, not certification. A formal MIL-STD-810H or ISO 16750-4 campaign belongs with accredited test laboratories, and functional-safety or regulatory sign-off belongs with your assessors. A team that already runs climatic chambers and is comfortable with tegrastats and nvidia-smi can build the protocol from the standards and NVIDIA’s thermal design guides, which make the customer responsible for qualifying the module in their own system.

FAQ

How to limit thermal throttling?

Size the processor and the cooling for the sustained workload at the hottest expected ambient, including cabinet temperature and sunlight, rather than for the burst. A lower-TDP part that runs at full speed all day can deliver more than a faster one that spends the afternoon throttled.

Does the NVIDIA Jetson run Linux?

Yes. Jetson Orin modules run NVIDIA Jetson Linux, whose developer guide documents the power, thermal and telemetry tools, including nvpmodel and tegrastats.