To see how inference throughput and accuracy change as a chip throttles from peak to sustained clocks, log four signals on one timeline: clock frequency, power draw, junction temperature and the throughput your application actually delivers. Then check accuracy separately, at pinned peak clocks and at the throttled plateau clock, since any loss is expected to come from dropped frames and missed deadlines rather than the arithmetic. Report the sustained plateau with its throttle curve instead of a peak figure. Thermal throttling exists to hold the die inside safe limits by lowering clock caps as it heats.
TechnoLynx can build this instrumentation for production inference, starting by framing the throttling risk for the specific deployment (a field fleet, an in-vehicle unit or a soak rig) and setting the margins and alert thresholds that matter. TechnoLynx then picks the telemetry stack, implements low-overhead logging and alerting, and validates both on representative hardware before rollout, within its Production AI Reliability practice.
The rest of this article covers four situations that share the same root cause but need different tooling: watching a Jetson fleet in the field, raising a live alert in a vehicle, publishing a benchmark number, and reading soak-test logs.
Why sustained throughput falls below peak on an edge module
Throttling is fast, and it can be large. In an 80-case study of CNN inference on a Raspberry Pi 4B, the heavy load of CNNs could push the processor into thermal throttling and degrade performance within a few seconds. Hysteresis-based active cooling prevented throttling in every case and raised throughput by up to roughly 90% compared with no cooling. The same study ran the uncooled board across ambient temperatures from 22 °C to 36 °C and measured throughput variations of up to 27.7% against the best result in that range, so ambient temperature is a first-order variable whenever active cooling is not an option. If your enclosure is sealed or fanless, the budgeting side of this problem is covered in our piece on thermal and power budgeting for fanless embedded inference.
On Jetson Orin, NVIDIA documents software throttling, hardware throttling and shutdown as separate thermal actions.
- Software throttling. NVIDIA’s Orin NX and Orin Nano thermal design guide gives 99 °C as the maximum SoC operating temperature at which the specified clocks hold. Above it, software reduces clock speeds, and short power fluctuations that push junction temperature over the threshold cause temporary clock reductions. The kernel’s devfreq cooling device throttles the GPU at the
gpu-sw-slowdownpassive trip point. Each cooling state corresponds to a maximum allowed frequency for the CPU, GPU and EMC clocks. In the log this appears as a stepped fall in GPU frequency that tracks temperature. - Hardware throttling. When software management cannot hold temperature, hardware clock throttling engages as a last resort. It significantly reduces performance, and its settings cannot be changed. The firmware caps CPU and GPU clocks at three levels, labelled light, medium and heavy with throttled percentages of 50, 75 and 87.5. The host OS is not notified of these events. You see them only as a drop in clock rates through a tool that samples cycle counters.
- Power-limit throttling. On Orin NX and Orin Nano, average and instantaneous power limits on the VDD_IN channel trigger hardware clock throttling of the GPU and CPU for a configurable period. This is why NVIDIA says MAXN is not a maximum-performance mode: hardware throttling engages once total module power exceeds the TDP budget.
- Shutdown. The SoC shuts the module down at 105 °C to protect its reliability.
The practical consequence is that “is it throttling?” has at least three distinct answers on one module, and only some of them leave a trace in the OS.
What telemetry should be logged alongside throughput and accuracy?
Log the minimum set that lets you attribute a throughput change to a cause.
| Signal | Where to read it on Jetson Orin | What it tells you |
|---|---|---|
| GPU clock | GR3D_FREQ in tegrastats, or the devfreq cur_freq sysfs node |
Current GPU frequency, per GPC in tegrastats |
| CPU and EMC clocks | cpufreq cpuinfo_cur_freq, EMC_FREQ in tegrastats |
Current CPU and memory-controller frequencies |
| Junction temperature per zone | tegrastats, jtop | Approach to trip points, reported per sensor group as the maximum sensor in the group |
| Thermal margin | TMARGIN: 105 °C minus the equal-weighted average of the CPU, GPU and SoC zones | A single headroom number to alert on |
| Rail power | INA3221 sysfs nodes (rail voltage and current) | Rail voltage in millivolts and current in milliamperes |
| Over-current events | oc*_throt_en and oc*_event_cnt under hwmon |
Which OC events are enabled, and how many have occurred |
| Delivered throughput and latency | Your own application, timestamped | The only signal that measures the work that matters |
Three caveats belong in the same design document. Internal Jetson power sensors underestimate actual power by up to 50% in most cases. The signal’s envelope is preserved, though, and the gap stays nearly constant for a given device. Relative trends are therefore usable as read, but absolute watts need calibration against an external meter, which the same study brings to within ±3%. Second, GPU and CV rails can be powered off at idle, and tegrastats then reports −256 °C for those sensors. A logger that averages that sentinel into a trend line will produce nonsense, so filter it at ingest. Third, all of these tools report module state. None of them tells you whether the model’s output is still correct.
Logging thermal throttling on a Jetson module in the field
For a fleet of field units, use the platform’s own tools instead of building custom instrumentation. Tegrastats ships with JetPack and reports memory and processor usage, power and thermal data at a configurable interval. It can also write to a file for scripts to consume. jtop (the jetson-stats package) covers Orin, Xavier, Nano and TX modules and can run as a standalone monitor or be imported into a Python script. The package ships an example logger that opens a jtop session and writes every stats sample as a row to a CSV file until the session ends. That example is close to the whole fleet logger you need. The work left is to add your application’s throughput counter to each row and to keep the writer off the inference thread so that logging never stalls a frame.
Running this on every unit, not just test units, means budgeting storage and upload across the fleet. Keep a low-rate rolling record on every unit, with a higher-rate buffer that is flushed only when margin crosses a threshold.
How early can you see the threshold coming? Earlier than the built-in governor reacts. A 2026 study of sustained LLM serving on edge SoCs found that vendor governors act only near hardware throttling thresholds and have no knowledge of upcoming work. Built-in throttling reacts only after headroom has largely been consumed. Alerting on TMARGIN and its rate of change, not on the trip point itself, gives you warning time instead of a post-mortem.
Can the car tell us the inference chip is about to throttle?
Yes, but raw telemetry alone does not do it. A real-time deployment where the model’s output feeds a decision (a strategy call, a safety function) needs a guardrail layer on top of the counters. The layer we would build reads thermal and power margin continuously, predicts when the next trip point will be reached, and raises an alert or changes the operating state before clocks drop.
A published Orin trace shows clocks holding until temperature is near the limit. The first sustained GPU frequency cut came at 22.25 minutes and 98.56 °C, a 6.7% reduction, followed 1.53 minutes later by a 3.5% cut in CPU frequency. Further staged reductions followed, and frequency recovered only after a restart. The zones are also coupled: under GPU-only heating, each 1 °C rise on the GPU raised the CPU and SoC zones by 0.86 °C and 0.85 °C. An alert keyed to the GPU sensor alone will therefore underestimate how fast the CPU-bound parts of the pipeline lose headroom.
Proactive triggers have worked in published tests. A 2026 study on a passively cooled Raspberry Pi 5 running YOLOv8n escalated DVFS state when temperature was above a floor and its derivative exceeded a threshold for a confirmed number of samples, with a dwell time to prevent oscillation. Over 30-minute sustained runs at ambient below 27 °C, that scheduler removed every observed throttling event and delivered a 6.8% higher frame rate than a temperature-only reactive baseline, using 1.9% less energy per frame. The uncontrolled full-speed configuration throttled 1,823 times in the same 30 minutes. The study derived every guard parameter from on-device sensor-noise and step-response measurements, not from heuristics, and that is the part worth copying.
For the pit-wall question, wire through TMARGIN, its slope, the current GPU clock as a fraction of its cap, and the over-current event count, alongside the model’s own latency. Two cautions apply. A threshold tuned on the bench may not transfer to the airflow and ambient conditions in the car, so it needs on-track validation. For proving throttling to the FIA after the fact, check the series’ technical regulations. What you can do is keep timestamped logs in a fixed schema, as MLPerf Power does when it records timestamped current, voltage and power to CSV during a run, and retain them with the session record.
Peak TOPS versus sustained throughput: the metric set
Report both: peak burst throughput, labelled as such, and sustained throughput measured on a thermally steady plateau with a stated run length and ambient. Publish the curve between them. A single peak figure hides the plateau a long-running workload actually depends on. An edge LLM study measured the iPhone 16 Pro at 40.49 tok/s peak, then saw it settle at 23.67 tok/s in its Hot state, a 41.5% reduction. The Galaxy S24 Ultra fell a milder 15% to a governor-regulated plateau. The authors conclude that for an always-on workload the operative ceiling is the plateau, not the peak.
A useful sustained metric set contains peak throughput, plateau throughput, the plateau’s coefficient of variation, time to first throttle, power at the plateau and ambient temperature. The same study reports plateau throughput, its CV and power per platform, and the ThermE work measures time to first throttle.
How long must a benchmark run before the number counts as sustained?
Long enough to reach and confirm the plateau on that device in that enclosure. No fixed duration works for every machine. MLPerf conventions give minimum run lengths. MLPerf inference rules set a 600-second duration for each scenario. MLPerf Power requires at least 60 seconds of power data for most scenarios. MLPerf Mobile pairs a 60-second minimum with a room temperature of 20 to 25 °C and 5-minute cooldowns between tests.
Those floors are not proof of a steady state. In the Orin trace above, first throttling came at 22.25 minutes, so a 600-second run (10 minutes) would have ended before the drop and reported unthrottled throughput as sustained. The authors of the 20-iteration LLM study also note that their session length captures plateau onset but not drift over longer horizons. We would keep running until throughput and temperature have both stopped trending for a disclosed window, and state that window next to the number.
Should a benchmark report a throttle curve instead of one number?
Report the curve and the number together. The curve shows how fast the device reaches its plateau and whether it ever really settles. Two devices with the same plateau can differ in deployability: one may reach it in seconds, while the other may hold peak for twenty minutes and then fall off.
Telling DVFS throttling apart from a defect in soak-test logs
Expected throttling follows a known model: a clock drop that coincides with a temperature trip point or a power-limit event, scales with ambient, and recovers when load or temperature falls. A defect breaks one of those links. Continuous logging of clock, power, temperature and throughput during the soak is what makes that classification possible. A pass/fail summary at the end cannot do it. That is also why we profile a soak before tuning anything: the right fix depends on which of these signatures is present, and a generic remedy applied first only hides the evidence.
| Log signature | Likely reading |
|---|---|
| GPU clock steps down as temperature crosses the software trip point, then recovers on cooling | Expected software DVFS throttling |
| Clocks halve or worse and the throttle-reason field reports HW thermal slowdown | Expected hardware thermal throttling, close to the limit |
oc*_event_cnt increments while temperature is well below trip points |
Power-limit throttling; check the power mode and supply |
| HW slowdown flagged with neither high temperature nor high power draw | Suspect the supply: NVML lists an external power-brake assertion as one cause |
| Throughput falls while clocks, power and temperature are all flat | Not accelerator throttling; look at host preprocessing, I/O or memory |
| Same unit, same ambient, plateau lower than its siblings | Candidate defect: thermal interface, contact or fan |
| Clocks stay reduced after the load stops and recover only on restart | Investigate; the published Orin trace showed this persistence |
On discrete NVIDIA GPUs the same pattern generalises through NVML, which reports the reason for a clock change as a bitmask that separates a software power cap, software thermal slowdown and hardware slowdown. DCGM exposes the clock-event-reason bitmask as field 112 and cumulative thermal violation time as field 241. Ambient has to be in the log as well. In a climate-chamber study of a Jetson AGX Orin at +40 °C, a synthetic stress test triggered throttling at 95.6 °C on the GPU and 99.0 °C on the CPU, while a TensorRT FP16 YOLOv8s workload in the same conditions sustained 108.8 FPS at 19.1 W, well below the throttling threshold. A stress test is not your workload, and ambient is not a constant.
If you log only GPU-side signals, the soak will blame the accelerator for bottlenecks that live on the host. For the full rig design (fixtures, duration, external power measurement), see our guide to instrumented sustained-operation profiling for thermal and power behaviour.
Does inference accuracy change when the chip throttles?
The arithmetic should not change, but the delivered result can. Lowering a clock within its validated operating range changes how fast a digital circuit computes, not what it computes. So a fixed compiled engine on throttled clocks should produce the same outputs for the same inputs. We expect accuracy loss under throttling to enter through the system around the model: dropped or skipped frames, stale inputs, deadline timeouts that fall back to a default, or a scheduler that switches to a cheaper model variant when it runs short of time. No source behind this article measures task accuracy against clock frequency directly, so treat this as an engineering expectation to test, not a published finding.
The test separates the two effects. MLPerf Mobile already runs accuracy mode on the full dataset separately from performance mode. We would add a second axis: run the accuracy set once at pinned peak clocks and once with clocks capped at the plateau frequency observed in the soak, then compare outputs element by element as well as on the task metric. Then replay a recorded real-time stream at both clock settings, with your production deadlines enforced, and score the outputs the pipeline actually emitted. If only the second comparison differs, the accuracy problem is a latency problem, and the fix belongs in the pipeline, not the model.
