A thermal throttling test is only worth running if it reproduces what the product will face: the shipped workload, on the production module, inside the production enclosure, at the worst ambient it will see, for long enough to reach thermal steady state. For a small enclosed device, that means at least two hours or three thermal time constants, whichever is longer, and Peakingtech’s stopping rule is three consecutive 15-minute readings agreeing within 0.5 °C. Use the clock-event reasons the driver reports to separate thermal throttling from power capping, and gate on the throughput the device sustains after the soak, not the early burst figure.

TechnoLynx runs this kind of sustained, instrumented test on a team’s own module and enclosure as part of its GPU performance engineering work. The first step is defining the worst-case duty cycle, the ambient and power envelope, and the acceptance thresholds for sustained throughput and latency.

The reason to be this deliberate is that throttling shows up as a performance problem. Most SoCs throttle rather than fail, which turns a thermal problem into a performance problem, and performance problems tend to get reported as software bugs. A thermal throttling test built around the real load catches that misattribution before a field trial rather than after one.

Designing a thermal throttling test around the real workload

A harness that runs the product’s own pipeline on each candidate module answers the procurement question directly. A proxy benchmark answers a different question. MLPerf Inference, the best-known proxy, requires every benchmark to run for at least 60 seconds so that power management and DVFS reach equilibrium, and it allows compilation and cache warm-up as untimed operations. Sixty seconds is a sensible rule for comparing results across submitters, but small enclosed devices have thermal time constants measured in tens of minutes.

LynxBenchAI publishes results showing how each candidate device runs the public PyTorch stack a team’s own models run on, so the first step is to check them when drawing up the shortlist of modules to put through the harness.

TechnoLynx starts this work by defining the representative workloads and the per-stage breakdown each module has to report. It then designs the harness, the timing hooks and the power telemetry around them, and builds and runs the harness on every candidate board.

Synthetic stress loads are no better a substitute. In a 2026 study published in Electronics, a Jetson AGX Orin in an outdoor enclosure at a +40 °C setpoint throttled under a stress test at GPU and CPU temperatures of 95.6 °C and 99.0 °C. Under the same conditions, the YOLOv8s inference workload sustained 108.8 FPS at 19.1 W, roughly half the 36.5 W of the stress test, with chip temperatures well below the throttling threshold. The authors conclude that synthetic stress tests substantially overestimate the thermal and power demands of that inference workload. A stress test can therefore condemn an enclosure the real model runs comfortably in.

Can we run our own detection and SLAM pipeline on each candidate board?

Yes, and published pipeline figures differ materially from standalone model figures. Fictionlab’s measurements on a Jetson Orin Nano 8GB in 15 W mode put YOLOv8n with TensorRT FP16 at 55 to 65 FPS standalone and 40 to 50 FPS alongside Nav2 and a camera driver. Running YOLOv8n and RTAB-Map SLAM together can pull detection down to 30 to 38 FPS, with SLAM holding 10 to 12 Hz. Those figures assume TensorRT engines built on the target device with matching JetPack and CUDA versions, which is one more reason the harness has to run on the board itself.

System contention widens the gap. In a 2026 roadside-perception deployment on a Jetson Orin Nano, average detector latency rose from 34.96 ms in controlled evaluation to 65.68 ms on the road. The authors attribute this to concurrent processes, display rendering and OS scheduling, and conclude that component-level latency benchmarks are upper bounds on deployment performance, not targets.

Per-stage decomposition is what makes these numbers usable. Phase-level profiling splits a pipeline into its functional stages. That pinpoints the stage that dominates latency and avoids the misleading conclusions an aggregate end-to-end figure can produce. For a detection and SLAM stack, we would time capture, preprocessing, detector inference, tracking or SLAM update and post-processing as separate spans, so that a slowdown under heat can be pinned to the stage that absorbs it.

A repeatable per-shipment harness for local LLM and diffusion workloads

A sustained test of local LLM inference needs a fixed workload that keeps the decode phase busy. A 2026 edge LLM study used an identical 258-token prompt that produces long-form output. The prompt was chosen to impose sustained decode load and stress thermal management and memory bandwidth rather than burst performance, and the model was loaded once and kept warm across iterations. Determinism matters for repeatability. The lm-Meter authors fixed prompts, capped generation length and set sampling temperature to zero so that token generation was deterministic during profiling. The same discipline applies to a Stable Diffusion fine-tuning run (fixed seeds, step counts and image sizes), so that two shipments are compared on identical work.

Running the harness on every incoming shipment, rather than once at qualification, follows from how easily the baseline shifts. Throttling thresholds on Jetson modules depend on the JetPack release, the sensor and the thermal policy, and Fictionlab advises verifying them on the deployed system rather than relying on one fixed temperature. A harness that can be rerun unchanged on the next delivery, firmware update or board revision is the only way to know whether any of those has moved.

What instrumentation captures clock speed, temperature, power and per-stage latency at the same time?

Only a handful of signals need to go on one timeline, and on NVIDIA hardware each has a documented source:

• GPU and CPU clocks: on Jetson Orin, the current GPU frequency can be read from the devfreq cur_freq sysfs node. On discrete GPUs, nvidia-smi can poll chosen fields in a loop at a set interval and write the results as CSV.
• Die temperature and violation state: nvidia-smi dmon reports power violations as a percentage and thermal violations as a flag.
• Power: Jetson Orin modules carry a three-channel INA3221 monitor, and its rail voltage and current can be read through sysfs.
• Enclosure surface temperature: on-die sensors tell you about throttling, while thermocouples on the outer surface tell you about certification and touch temperatures. The two are not related by a constant offset.
• Per-stage latency: lm-Meter inserts std::chrono::steady_clock timers immediately before and after each phase of the inference engine, and reports end-to-end latency accuracy of at least 99.99% against reference measurements.

On Jetson, tegrastats and jtop give a live view of power and thermals. Two caveats belong in every harness design. First, instrumentation with significant overhead distorts the latency and energy signals it is meant to capture, so timers should be lightweight and telemetry polling should stay off the inference threads. Second, on-board power sensors are approximations. A study of nvidia-smi across more than 70 GPUs found its power error is roughly ±5% of the reading rather than the ±5 W NVIDIA states, and that on A100 and H100 only 25% of runtime is sampled. The same study notes near-perfectly linear readings on each GPU, so same-GPU comparisons stay reliable while comparisons across different GPUs would be erroneous. For watts-per-inference figures that will leave the lab, we would put an external meter on the supply.

How long should a thermal throttling test run before throttling reliably shows up?

Longer than most bench runs. Peakingtech puts it plainly: a 10-minute run on a system with a 40-minute thermal time constant reaches roughly 22% of its final temperature rise, while a two-hour run reaches about 95%. The arithmetic follows a first-order heating curve, where the fraction of the rise reached after time t is 1 − e^(−t/τ). With τ = 40 minutes, 1 − e^(−10/40) ≈ 0.22 and 1 − e^(−120/40) ≈ 0.95. A short run reads cool and looks fine because, for ten minutes, it is.

On a passively cooled Orin Nano in 15 W mode, Fictionlab saw the SoC climb from about 45 °C at idle to 78 to 85 °C within 8 to 12 minutes of sustained YOLOv8n load. Yet in a 2026 thermostatic-chamber study of a Jetson AGX Orin 64GB, the first sustained GPU frequency reduction did not arrive until 22.25 minutes, at 98.56 °C. The stopping rule should therefore come from the data, so log continuously and use Peakingtech’s rule of stopping once three consecutive 15-minute readings agree within 0.5 °C, after the two-hour minimum. Between runs, the device should return to its idle temperature band. One Raspberry Pi 5 study used a cooldown of at least 15 minutes and started each run only once the SoC was back in that band.

The pass/fail threshold is not the spec-sheet number

The number printed on a spec sheet and the number a shipment has to pass serve different purposes. They should be set separately, and the acceptance threshold should be the stricter of the two. The roadside-perception authors argue that any embedded AI deployment specification must reserve headroom explicitly for system overhead, and that best-case throughput figures should not appear in deployment specifications. We would accept on sustained throughput after soak at worst-case ambient, with margin for the overhead the field adds, and publish only figures the acceptance test already guarantees.

Gating field trials on a fixed 15 W power envelope

We would set a 15 W field-trial gate as a single joint criterion: sustained frame rate at or above target while power stays under 15 W for the full duration of every scenario in an agreed matrix. The cap itself needs defining first. On a Jetson Orin Nano, the 15 W mode limits the module power budget, but total input-side draw in a typical developer-kit setup commonly reaches roughly 13 to 16 W under sustained load, before cameras and USB peripherals are added. A gate written against “15 W” that doesn’t say whether it means the module budget or the battery-side draw will get argued about once the results arrive.

LynxBenchAI publishes results showing how each candidate device runs the public PyTorch stack, so a team starts by checking them to shortlist modules before running the 15 W gate on each one.

TechnoLynx sets this gate up with the team in stages. First come the cap, the target frame rate and the scenario taxonomy. Next comes the harness, with per-stage latency and power measured together on the target module. Last come repeated runs to confirm reproducibility, and a locked report format the team can reuse for later hardware or firmware revisions.

Should a thermal throttling test use worst-case continuous load or a representative mixed duty cycle?

It depends on what a failure in the field costs, and the two tests are not exclusive.

Situation	Primary gate	Secondary check
Field failure is costly or safety-relevant; risk tolerance is low	Worst-case continuous load at worst-case ambient	Representative duty cycle, to confirm typical behaviour
Field use is well characterised; the worst case is rare or transient	Representative mixed duty cycle across the scenario matrix	Worst-case continuous run as a stress overlay
Duty cycle is not yet known	Worst-case continuous load until field data exists	Measure the duty cycle, then revisit the gate

Duty cycle changes the power picture sharply. MLPerf Power notes that tiny systems often run at duty cycles below 5%, so devices with milliwatt-level peak power often draw only microwatts on average. Representative scenes can also overturn assumptions about which conditions are hardest. In the roadside study’s pipeline, where detection and classification are decoupled, frame rate rose as conditions got worse: 27.01 FPS in dense urban traffic against 42.32 FPS on dark rural roads, because a suppressed detector triggers fewer classifier calls. For a robot or drone harness, we would build a scenario matrix spanning low and high light, flat and complex terrain, and sparse and dense artefact density. The dense, well-lit cases should be treated as the compute worst case rather than assumed easy.

How do you set a pass/fail threshold under a 15 W cap, and what do you tune when the module fails it?

The threshold should be set before the first run and written as a pair: sustained FPS at or above target and measured power at or below 15 W, both held for the full scenario duration across repeated runs. Agreeing it before anyone sees results stops the threshold drifting towards whatever the favourite module achieved. Repeated runs matter because an average can hide run-to-run spread. In a Raspberry Pi 5 study, removing a dwell guard from the DVFS scheduler left mean throughput essentially unchanged but raised the FPS standard deviation sixfold, from 0.070 to 0.426.

When a module fails the gate, we would tune in order from cheapest to most invasive:

1. Check the engine. Fictionlab’s Orin Nano figures assume TensorRT FP16 engines built on the target, and note that raw PyTorch models usually deliver lower throughput and can cost more energy per inference.
2. Cap the clock rather than dropping the power mode. On the fanless Pi 5, a 25% CPU frequency cap cost only about 15% of FPS while board power fell about 26%. The capped states did not throttle across 30-minute runs, whereas full speed did.
3. Reduce the work per frame. In the roadside system, full per-frame detection was thermally unsustainable on the Orin Nano, so detection ran on a subset of frames and tracking carried the labels in between.
4. Change the cooling. Fictionlab recommends active cooling for sustained 15 W load, ambient that can exceed 30 °C, small or airflow-restricted enclosures, or more than 30 minutes of continuous inference. The trade-offs for sealed designs are covered in thermal and power budgeting for fanless embedded inference.

A 15 W gate run on bench hardware still leaves one gap: it may not reproduce the throttling the final enclosure causes.

Open bench versus production enclosure: why the same module throttles differently

A dev board on an open bench and the same module in its production housing are two different thermal systems. A thermal throttling test on the first says little about the second. Natural convection depends on orientation, so a board tested flat on a bench behaves differently from one mounted vertically in a bracket. The authors of a 2019 smartphone throttling study used a commercial phone instead of experimental boards because form factor and packaging affect thermal behaviour. Toradex’s guidance is to test the complete system under worst-case conditions before finalising the enclosure, measuring SoC junction temperature, ambient temperature and power consumption.

LynxBenchAI publishes device results on the public PyTorch stack that help a team shortlist modules, but they do not measure the reader’s own enclosure, so the first step after shortlisting is still a soak in the production housing.

TechnoLynx runs the soak in a production-representative enclosure at the agreed worst-case ambient and correlates the result with any thermal simulation the team has. It then sets throttling thresholds and fallback modes from what the module actually does in that enclosure. A model that performs in a development environment is a different artefact from one that performs on its deployment target, and closing that gap is engineering work, not a hand-off.

Ambient temperature is the other half of the problem. Safety standards are evaluated at 25 °C, while a utility closet sits nearer 35 °C and a vehicle cabin goes higher still, and every degree of ambient becomes a degree of case temperature. Real deployments can also run cooler than the lab. The roadside-perception trial recorded an average GPU temperature of 48.15 °C on the road, against 54.51 °C in controlled evaluation, and credited air-conditioned airflow around the device. The authors warn that enclosed vehicles, warm climates or slow traffic may remove that benefit. On battery or solar platforms, the power supply adds a second axis. MLPerf Mobile recommends a full charge before benchmarking to avoid power-saving modes, and Jetson’s nvpmodel GUI warns separately when low input voltage, rather than heat, is throttling the system. A thermal simulation of the enclosure helps narrow the design options, but we would trust it only once it has been checked against a real soak.

At what junction temperature does a GPU or SoC start throttling, and how much throughput is lost at each step?

Junction temperature is the temperature of the active silicon, which is always higher than the package, the heatsink or the surrounding air. On Jetson Orin NX and Orin Nano, NVIDIA’s thermal design guide puts software throttling at about 99 °C on the die and shutdown at 105 °C, with the die sensors quoted to about ±3 °C. On Jetson AGX Orin, software thermal management engages first and limits CPU and GPU clocks by a computed throttle degree. If that cannot hold the temperature, hardware throttling engages as a last resort. Its settings cannot be altered, and a thermal trip resets the system. Orin hardware throttling caps clocks at three documented levels: light, medium and heavy, listed as 50%, 75% and 87.5% clock throttled. On discrete NVIDIA GPUs, nvidia-smi reports the slowdown temperature at which the hardware begins cutting clocks to cool.

The throughput lost at each step has to be measured, not read off a table. In the AGX Orin chamber study, the first sustained reduction cut GPU frequency by 6.7%, and a 3.5% CPU reduction followed 1.53 minutes later. Further staged reductions and persistent CPU throttling followed, and frequency recovered only after a restart. On a Nexus 6P, enabling the stock thermal governor cut a game from 35 FPS to 23 FPS, a 34% loss. Time spent at the two highest GPU frequencies fell to zero, while time at 390 MHz rose from 15% to 67%. That histogram is the method we would reuse. Log clock state with every throughput sample, then report sustained throughput per clock state, so that each throttling step has a measured cost. How accuracy changes alongside throughput once a model is throttled is covered in characterising accuracy and throughput under thermal throttling.

Is a frame drop caused by thermal throttling, power capping or a queue backlog?

The clock-event reasons answer most of this question. nvidia-smi reports the factors currently reducing clock frequency, and if every reason reads “Not Active” the clocks are running as high as possible. It also keeps microsecond counters for how long each factor has been reducing clocks, with separate fields for software power capping, software thermal slowdown, hardware thermal slowdown and hardware power brake.

Cause	Clock behaviour	What to check
Software power cap	Clocks held below the requested level because the GPU draws too much power	SW Power Cap reason and its counter; power sitting at the enforced limit
Software thermal slowdown	Clocks lowered to keep GPU or memory temperature under the maximum operating temperature	SW Thermal Slowdown reason
Hardware thermal slowdown	Core clocks cut by a factor of two or more because temperature is too high	HW Thermal Slowdown counter
External power brake	Core clocks cut by a factor of two or more after an external power-brake assertion, for example from the power supply	HW Power Brake Slowdown reason
Jetson power limit	Hardware clock throttling of GPU and CPU when module power exceeds the configured limit	oc*_event_cnt sysfs counters
Queue backlog	Clocks as high as possible, every event reason “Not Active”	Per-stage timers show waiting time growing ahead of one stage

Two further details matter. On Jetson, the host OS is not notified of hardware throttling events, so the drop in clock rates has to be detected by sampling CPU cycle counters. The shape of the slowdown is also diagnostic. SysOM-AI treats uniform slowdown across all GPU kernels as a sign of a hardware cause such as thermal throttling, and slowdown confined to specific kernels as a sign of a software change. A power cap is not always harmful, either. A 2025 GROMACS study found that a cap typically leaves performance unaffected until it falls below a workload-specific threshold: on the A100, around 150 W for small benchmarks and 250 W for larger ones.

Catching throttling and clock drift during a production run

The same telemetry used in the soak, logged next to every inference result in production, is what makes throttling detectable and auditable in the field. Online profilers make metrics available to the application while the model runs, which allows adaptive scheduling instead of after-the-fact analysis. Alerts need debouncing to avoid flapping. The Pi 5 scheduler enforced, as safety invariants, a hysteresis band of at least three standard deviations of sensor noise, a dwell of at least twice the thermal time constant, and three confirming samples before any transition. We would raise an alert when the clock stays below the qualified baseline for a confirmed interval, record the driver and SDK versions with each result, and treat any update to them as a reason to rerun the soak.

Comparing candidate modules on derated throughput, not datasheet figures

The fair comparison between candidate modules is worst-case sustained throughput after soak at the rated ambient, checked against the application’s minimum throughput, latency and accuracy. Peak figures mislead because a power mode is a ceiling, not a guarantee: a Jetson set to 40 W in an undersized case can deliver less than one set to 25 W that never throttles. The roadside-perception authors report a consistent 20–30% relative degradation across three baselines when moving from static-image evaluation to real-world streaming deployment. In the 2026 edge LLM study, an iPhone 16 Pro, running back-to-back iterations one second apart, ran 44.1% below peak from the eighth onwards.

LynxBenchAI publishes results showing how each candidate device runs the public PyTorch stack, which a team uses to draw up the shortlist before any soak, though they do not measure the team’s own pipeline or enclosure.

TechnoLynx runs the soak on each shortlisted module at the rated ambient and ports and tunes the pipeline for it, including any reduced-fidelity fallback mode. It then checks throughput, latency and accuracy against the requirement before anyone commits to a module.

Putting a passively cooled edge board and a server GPU on equal terms

Comparing a fanless edge board with a server GPU needs one steady-state definition applied to both. NVIDIA’s Jetson AGX Orin thermal design guide defines total module power as the average board power while the target workload runs under worst-case conditions in steady state. Applied to every candidate, that definition makes a reasonable common basis. For watts-per-inference reporting, MLPerf Power is the published methodology we would cite. It measures full system power, including peripherals that cannot be powered down on edge devices. It requires a SPEC-certified power meter, such as the Yokogawa WT310, for inference, and asks submitters to disclose cooling, power management and environmental conditions. Its 60-second minimum measurement window works for averaging power, but it is not a thermal soak. For passively cooled boards, we would report both the MLPerf-style figure and the post-soak figure. Ambient should be fixed and stated; MLPerf Mobile, for example, requires a room temperature between 20 and 25 °C.

Does the cheaper module still meet the requirement once it derates?

The test is direct. Run the soak, take the derated throughput and latency, and compare them with the minimum the application needs, whether that is a clinic’s daily sample volume, a vehicle’s reaction window or a line’s inspection rate. If the derated figure clears the requirement with margin, the cheaper module is viable. If it does not, the options are extra cooling, queueing, a reduced-resolution or lower-frame-rate mode, a model switch or a different module, and each has its own cost. Derating also changes the economics: a cheaper part that needs a fallback mode or extra cooling can end up costing more per request than the part it replaced. Safety-critical functions such as trailer-sway detection leave little room for a degraded mode, so for those we would size for the derated case with no fallback assumed. A module with no sustained throttling data at all is a risk in its own right, and one to close before rollout rather than after.

Who can run a thermal throttling test on our production hardware?

TechnoLynx runs sustained, instrumented thermal throttling tests on a team’s own module, enclosure and workload, and turns the results into pass/fail gates, tuning changes and runtime alerts. Building the capability in-house makes sense when a team already owns a thermal chamber, has telemetry engineers and expects to qualify hardware repeatedly. Bringing in a specialist makes sense when the decision is close and nobody has yet run a soak to steady state on this platform. When choosing a partner, ask whether they test inside the production enclosure, log clock-event reasons alongside per-stage latency, and fix thresholds before the first result. The engineering work sits under GPU performance engineering, and LynxBenchAI is the measuring tool for comparing devices before a shortlist is drawn up.

The engagement does not suit consumer PC or laptop throttling problems, which are better taken to the hardware vendor’s support. It produces engineering evidence, not environmental certification or regulatory sign-off. We would leave those to accredited test houses and the product’s safety owners, with a well-documented soak for them to work from. Teams that only need to compare devices before buying may find LynxBenchAI’s published results enough on their own, and teams with an established hardware validation lab may need only the method.

FAQ

How do I check the power mode on a Jetson?

Run sudo nvpmodel -q to see the current mode and sudo nvpmodel -m with a mode ID to set a new one; the IDs vary by module. The chosen mode persists across reboots until it is changed.