The spec sheet describes a moment, not a steady state
A GPU data sheet lists a boost clock frequency — per NVIDIA’s published specifications, roughly 2.1 GHz for an H100 SXM. That frequency is real. The chip does reach it. What the spec sheet omits is how long it stays there under a production AI workload, and the answer, for sustained dense compute, is usually “not very long.”
Boost clocks are transient by design. They represent the maximum frequency the chip will sustain when thermal headroom exists and power budget allows. Under the sustained full-load conditions characteristic of neural network training or large-batch inference, both headroom and budget are consumed within minutes. The clock settles to a lower, sustainable frequency — and that settled frequency is what determines actual throughput over hours and days.
This isn’t a defect. It’s thermal physics, and it governs performance more directly than any software optimisation. It also means a performance figure belongs to the whole executor and the physical situation it sits in — device, backend, driver, framework, runtime, chassis — rather than to a part number on an invoice.
Power limits as performance governors
Modern data center GPUs operate within a power envelope managed by onboard firmware. Per published specifications, the NVIDIA A100 SXM has a default TDP of 400W; the H100 SXM is rated at 700W. These are not average power draws — they are limits. When the chip’s instantaneous power consumption approaches the limit, the firmware reduces clock frequency to keep power within bounds.
For AI workloads that fully exercise tensor cores, the power limit is typically the first constraint that activates. Dense matrix multiplications — the dominant operation in both training and inference — drive nearly every functional unit on the die simultaneously. This is the highest-power operating regime the GPU encounters, and it means the power governor engages earlier and more aggressively than in workloads that leave portions of the die idle.
The implication is direct: training throughput is often a function of the power budget, not the theoretical peak FLOPS. Two identical GPUs at different power limits (configurable via nvidia-smi -pl) will produce measurably different throughput. One running at a 300W limit will sustain a lower clock than one at 400W, and the throughput difference is roughly proportional to the clock difference for compute-bound workloads.
There is a second-order effect worth naming, because it surprises people who expect a linear payoff. Raising a GPU’s power limit rarely scales sustained performance proportionally with the added watts. Once the silicon is voltage- and frequency-limited at the top of its curve, each extra watt buys a smaller clock increment than the last, and beyond a point the extra power is dissipated as heat that the cooling path must then carry away — which can pull the steady-state clock back down. The headline TDP is a ceiling on what the firmware will allow, not a dial that converts watts into throughput one-for-one. The same asymmetry runs in the other direction: capping power often costs less throughput than the watt reduction suggests, which is why datasheet power and clock figures are structurally poor predictors of what a system delivers on a real workload.
Thermal throttling: gradual, not catastrophic
The word “throttling” implies an emergency — something overheating and desperately pulling back. In data center GPU operation, thermal management is more mundane and more continuous than that.
As the GPU die heats under sustained load, the firmware progressively reduces clock frequency to maintain junction temperature below the rated maximum (typically around 83°C for recent NVIDIA data center GPUs, per published thermal specifications). This is a smooth, continuous process, not a cliff edge. The clock doesn’t drop from its boost frequency to the base frequency in one step; it decreases gradually over minutes, stabilising wherever the power dissipation matches the cooling capacity.
Cooling capacity itself is a system-level property. It depends on the server chassis design, fan speed profiles, ambient temperature, and most critically, the thermal load from neighbouring components. In an 8-GPU DGX node, the interior GPUs see higher ambient temperatures than the edge cards. The same chip, running the same workload, settles at different sustained clocks depending on its position in the chassis. We have observed steady-state frequency differences of roughly 60–90 MHz between the hottest and coolest GPU positions in the same node (an observed pattern across the dense systems we work on, not a benchmarked rate) — enough to produce visible throughput variation across cards.
This interacts with why AI performance changes over time in a direct way: the thermal trajectory of the first 15 minutes of a workload is characteristically different from the next eight hours. Early measurements capture a GPU at above-steady-state frequencies and under-steady-state temperatures. The performance they report is real but temporary.
Why are boost clocks misleading for sustained AI workloads?
GPU spec sheets prominently feature boost clock frequencies. Marketing materials build performance claims around them. Benchmark results that happen to be measured during the boost-clock phase inherit this flattering number.
The problem isn’t that boost clocks are fictitious — the chip does reach them. The problem is that they describe a capability that exists under specific thermal and power conditions, not a guarantee that holds under production load. For workloads that run for hours or days, the boost clock is a brief initial state that the system moves through on its way to steady-state operation.
The steady-state frequency — sometimes called the sustained or operating frequency — is what actually determines sustained throughput. In practice, it is typically on the order of 100–300 MHz below the advertised boost frequency for data center GPUs under heavy load, which translates to roughly a 5–15% throughput gap between the boost-phase number and the steady-state number.
That gap is well-known to hardware engineers and largely invisible to the software engineers, data scientists, and procurement teams who consume benchmark results and spec sheets. Surfacing it is one of the basic requirements for honest performance reporting: the operationally relevant figure is the one the machine holds under load, not the best instant it reached.
Boost phase vs. sustained operation: what the numbers actually mean
| Parameter | Spec sheet / boost phase | Sustained operation |
|---|---|---|
| Clock frequency | Boost clock (advertised maximum) | Typically 100–300 MHz lower after thermal settling |
| Throughput | Peak, measured in first minutes | Roughly 5–15% below peak after thermal equilibrium |
| Thermal state | Below junction limit, rising | Settled at or near junction limit |
| Time to observe | Typically first 1–5 minutes | Typically after 15–30 minutes of full load |
| Multi-GPU scaling | 8× single-GPU (implied) | Less than 8× due to positional thermal variation |
Evidence class for this table: clock and thermal ranges are drawn from published NVIDIA data center specifications; the throughput gap and settling times are observed patterns across the dense GPU systems we work on, not a published benchmark.
Dense GPU environments amplify the problem
Single-GPU testing in an open bench or lightly loaded chassis produces the most optimistic thermal behaviour. The GPU has ample cooling airflow, minimal thermal interference, and stays close to boost frequencies longer.
Production deployments are dense. Eight GPUs per node. Multiple nodes per rack. The thermal load per unit volume is substantial, and the cooling infrastructure has to handle the aggregate heat output under sustained operation. The air entering each successive GPU in the airflow path is warmer than what the previous GPU expelled, so interior positions run hotter and settle at lower sustained clocks than the cards at the edges. The aggregate throughput of an 8-GPU node is therefore not 8× the single-GPU throughput measured on an open bench.
Rack-level effects add another layer. Hot aisle temperature rises as more nodes in the rack reach full load. If the data center’s cooling capacity is marginal or unevenly distributed, pods of nodes can experience sustained above-target ambient temperatures, pushing GPU steady-state clocks lower across the board.
These are operational realities that no single-GPU benchmark captures. They are also realities that the mythology around sustained GPU utilization often obscures — a GPU can report 100% utilisation while operating at a thermally reduced clock that delivers substantially less throughput than the spec sheet implies.
What a run does and does not record
These governors act on the machine whether or not any instrument watches them, and it is worth being precise about which instruments do. A LynxBenchAI run discards a warm-up phase, then counts completed iterations inside a continuous timed window at a saturated batch size. It does not record power draw, clock behaviour, or temperature, and it does not wait for a particular thermal state before measuring. Each test is one declared window, not a median across repeated trials.
That boundary matters in both directions. A low score does not localise the cause to power or thermals — the run reports completed work, not a diagnosis of why the work was slow. And the integrity of the window comes from being declared, not from being long enough to expose every governor described above.
What the physical envelope does explain is why a self-run number is worth more here than a published one. A run happens on the reader’s own machine, in the chassis and cooling it actually has, rather than on a reference system with a different envelope. Because two runs of the same device can differ for reasons no score separates, a device’s population of submissions on the public leaderboard carries more information than any single result.
Living with the physics
None of this is fixable by software optimisation or clever engineering. Power limits and thermal physics are hard constraints. The practical response is not to fight them but to account for them.
Measure under sustained, thermally settled conditions rather than reporting the first five minutes of a cold start; in our experience equilibrium takes roughly 15 to 30 minutes under full load. Where you can instrument it, report power draw alongside throughput — performance-per-watt is a more stable and more informative metric for workloads that are power-limited, and comparing GPUs at equal power budgets often reveals a different ranking than comparing at default settings. And design capacity around steady state, not peak: planning that assumes boost-clock throughput will typically overcount by roughly 5–15%. As discussed in how peak vs. steady-state performance diverge, the gap between what the hardware can do briefly and what it does continuously is the gap between optimistic planning and realistic planning.
The physics wins either way. The open question for any given fleet is whether the measurement regime acknowledges it before deployment or discovers it afterwards — and what a team would have to record, beyond completed work in a declared window, to tell those two situations apart.
Frequently Asked Questions
How do GPU power limits shape sustained AI performance independently of the silicon’s headline capability?
Dense tensor-core workloads drive nearly every functional unit on the die simultaneously, which is the highest-power regime a GPU sees. Firmware enforces the configured TDP by lowering clock frequency as instantaneous draw approaches the cap, so sustained throughput tracks the power budget, not the theoretical peak FLOPS. Two identical GPUs at different nvidia-smi -pl settings will produce measurably different throughput on the same workload.
Why are boost clocks transient by design, and why does that matter for long-running AI workloads?
Boost clocks describe a capability that holds only while thermal headroom and power budget remain available. Under sustained training or large-batch inference, both are consumed within minutes, and the chip settles to a steady-state frequency typically 100–300 MHz below the advertised boost — roughly a 5–15% throughput gap. For workloads measured in hours or days, the settled frequency is what determines real output; the boost number is a brief initial state.
When is thermal throttling a normal operating behaviour rather than evidence of a fault?
In data center GPUs, thermal management is a continuous, smooth reduction in clock frequency to hold junction temperature below its rated maximum (around 83°C for recent NVIDIA parts). It is not a cliff edge or an emergency — it is the firmware doing what it was designed to do. Throttling becomes diagnostic only when the steady-state clock sits far below what the cooling capacity and power budget should allow, which points at the system rather than the chip.
When a GPU’s power limit is raised or capped, why does sustained AI performance rarely move in proportion to the power budget in either direction?
At the top of the voltage-frequency curve each extra watt buys a smaller clock increment than the last, so returns diminish well before the headline TDP. Much of the added power also becomes heat the cooling path must remove, and if dissipation outruns the chassis the steady-state clock settles back down. Capping power is asymmetric for the same reason — it often costs less throughput than the watt reduction implies, because the firmware was already trading clock for heat.
How does rack and chassis density change the thermal envelope enough to make two physically identical GPUs diverge in sustained throughput?
Density determines how much heat neighbouring components add to each card’s local ambient. Interior positions in an 8-GPU node inhale pre-warmed air and settle at lower sustained clocks than edge cards — we have observed roughly 60–90 MHz of spread within the same chassis — and hot-aisle effects extend that logic across a rack. Identical silicon therefore diverges in sustained throughput purely on physical position, before any difference in the chip itself.
If a benchmark run reports only completed work inside a declared window, how should a reader reason about power and thermal governors it never records?
Treat the number as an outcome, not an explanation. A run that discards a warm-up and counts iterations inside a declared window tells you what the machine delivered in its own chassis and cooling; it does not tell you which governor set that ceiling, because power, clock, and temperature are not recorded. If you need attribution, you need separate telemetry alongside the run — and because a single window is one trial rather than a median, a population of submissions for the same device is more informative than any one result.
When a GPU’s clock speed visibly rises and falls during a long AI workload, what does that fluctuation tell you — and what does it not tell you — about the system’s sustained capability?
Fluctuation tells you the firmware is actively balancing power draw and junction temperature against the requested work, which is normal behaviour on any modern GPU under varying load. It does not tell you the card is faulty, nor does it tell you where the sustained ceiling sits — that only shows up once the system has thermally settled and you measure across a window long enough to average the oscillation. Reading a sustained figure off a moment of the fluctuation, high or low, is the error to avoid.
