GPU load testing through generative AI workloads should report what the hardware holds once it is warm and saturated, not the best instant it managed. LynxBenchAI’s methodology is built on that position: discard the warm-up, count completed iterations inside a continuous timed window, and do it at a workload size already raised until throughput stops improving. A boost-clock burst really happens and a peak specification really is a ceiling, but neither describes what a deployed model gets from the hardware hour after hour.

To score your own device this way, install the Personal Edition with pip install lynxbench-ai; it scores the hardware on the release’s fixed models, not on your application.

Why does sustained performance matter more than peak for AI workloads?

For AI and other throughput-oriented GPU work, the only performance that counts operationally is what the device can hold under realistic, heavy load. Transient turbo behaviour is not deployable performance. A 2026 arXiv study of LLM inference on edge devices puts this in operational terms: for an always-on workload, the operative ceiling is the plateau throughput, not the peak.

Temperature affects throughput at cluster scale too. Meta’s Llama 3 report (2024) recorded a 1–2% swing in training throughput over the course of each day, caused by higher mid-day temperatures affecting GPU dynamic voltage and frequency scaling. Such issues are minor in isolation. The authors of Guard (2026), a straggler-detection system for large training jobs, point out that they add up to significant inefficiency over multi-week runs.

A burst figure tells you the device can reach a level. The sustained figure tells you whether it stays there long enough to finish the job. The general contrast is covered in peak versus steady-state performance in AI; what follows is what a load test has to do to capture the second number.

A passed stress test is not a sustained-throughput figure

The assumption sounds reasonable: if a GPU survives a burn-in, it should deliver its rated performance under AI load. It doesn’t follow. The Guard authors found that burn-in diagnostics are designed to check functional correctness under idealised conditions, and that they miss degradation that depends on the workload, comes and goes, or only appears in long-running mixed workloads. Burn-in tests run for short durations and focus on arithmetic correctness rather than sustained throughput or sustained data movement.

In Guard’s production observations, nodes drawing 10–15% less power than the cluster average showed lower FLOPS utilisation even though their GPU utilisation counters and frequencies looked normal. A stress test answers “does it break?”, and a load test answers “what does it hold?”.

What a peak GPU number tells you, and what it leaves out

A peak figure gives you an upper bound under ideal conditions. Theoretical peak compute assumes the compute units are fully saturated and every Tensor Core is in use. What it leaves out is how close real work gets to that bound. On a bf16 GEMM benchmark, the AxoNN authors measured a best sustained 280 Tflop/s on an A100, which is 90% of peak, and 125 Tflop/s on one MI250X GCD, which is 65% of peak. They concluded that vendor-advertised peak is almost always out of reach in practice.

Power ratings show the same pattern. In one measurement study, an 8-GPU NVIDIA H100 HGX node peaked at about 8.4 kW against a manufacturer rating of 10.2 kW. That is 18% lower, with the GPUs near full utilisation.

Peak numbers still have a use. They bound what is possible and show how much of the hardware a given workload leaves idle. The mistake is reading them as delivered throughput, which is the core of why spec-sheet benchmarking fails for AI.

How power, heat and frequency scaling pull a burst down to a plateau

By default, an NVIDIA GPU’s clock floats: it idles when there is no work and boosts when a workload starts. Power throttling begins when average power reaches the power limit, and the driver then lowers the clock to stay under it. NVIDIA’s TensorRT documentation says this happens by design, especially on GPUs with lower power limits such as the T4 and A2. Heat sets a second limit. Thermal throttling starts at a slowdown temperature that differs by device. Higher temperatures also increase leakage current, so a poorly cooled, power-limited card settles at a lower clock even before it throttles thermally. Guard reports a temperature rise from 50°C to 77°C cutting core frequency from 1.93 GHz to 1.38 GHz.

The controller’s timing matters too. An empirical study of GPU frequency behaviour under ML workloads found that on lower-performance parts (RTX 3060M, T4, L4), frequency changes at fixed 20 ms intervals and depends on the previous 80 ms of work. The A100 and RTX PRO 6000 Blackwell changed frequency immediately. The same study found that as a kernel’s workload got heavier, throughput rose while frequency fell.

Gaps between inferences in a benchmarking harness skew results too. If a harness synchronises after every inference, the gaps lower average power, the clock throttles less, and the measured throughput ends up higher than what the GPU delivers when fully loaded with no gaps. NVIDIA built trtexec to leave almost no gaps between kernel executions for exactly this reason. According to the same documentation, two runs of one engine on one GPU can differ by tens of percent depending on whether the clock floats or is locked, or whether throttling starts partway through. On embedded hardware, instrumenting this behaviour is the subject of thermal and power profiling for embedded AI workloads under real load.

When does sustained performance differ most from peak?

In one edge-inference study, the gap was widest on the phones: an iPhone 16 Pro peaked at 40.49 tok/s and settled at 23.67 tok/s once it reached its Hot state, a 41.5% drop. A Samsung Galaxy S24 Ultra lost about 15% on its way to a governor-regulated plateau, under a configuration chosen to keep it within that budget. A laptop with an RTX 4050 averaged 131.7 tok/s with a 2.2% coefficient of variation. Its GPU warmed from 55°C to 70°C and no throttling was observed.

In that laptop result, the first run was slower than the warm average, not faster. The RTX 4050’s cold-start run, at 93.8 tok/s, reflected model loading and was excluded. Discarding the warm-up protects a figure in both directions: it removes flattering boost behaviour and penalising start-up costs alike.

On a well-cooled card that never reaches its power limit or slowdown temperature under a given workload, we would expect peak and sustained to sit close together, because the throttling mechanisms never kick in. Phones, laptops, small-form-factor parts and low-power inference cards are where we would expect a burst number to mislead most.

Reading a headline number that never says how long it ran

If a figure doesn’t state its warm-up, timed window or workload size, read it as possibly a burst until shown otherwise. LynxBenchAI holds that a figure’s integrity comes from the window being declared, not from the window being long. trtexec, for example, warms up for at least 200 ms and then runs at least 10 iterations or 3 seconds, whichever is longer, and every one of those settings can be changed. No particular setting is correct in itself. The result becomes readable when the settings are stated.

Even a declared figure describes one GPU, not every unit of its SKU. Across HPC clusters, one GPU power-management study (SC22) measured an average 8% performance variation (up to 22%) between GPUs of identical architecture and SKU, with outliers up to 1.5x slower than the median GPU.

Checklist: questions to put to any GPU AI number

Check	Why it changes the reading
Was a warm-up discarded?	Early iterations include compilation and allocation, and a cold start can be slower than warm runs.
Is the timed window stated?	Clock changes make measurements over very short spans, such as 20 ms, unstable.
Was the GPU kept loaded with no idle gaps?	Gaps reduce throttling and inflate throughput.
At what batch size or concurrency?	Below saturation, throughput is still climbing, so the figure measures an underfed device.
Were clocks floating or locked, and did throttling start mid-run?	This alone can move a result by tens of percent.
Is it one window or a median of repeats?	A single declared window is not a median across trials.
Which model and precision?	Lower precision changes throughput substantially.

While a run is going, nvidia-smi dmon shows temperature and clock frequency, so throttling is visible as the clock drops, and nvidia-smi -q reports instantaneous and average power draw. For a step-by-step version on your own machine, follow a practical protocol for benchmarking your PC for AI.

Does a long agentic or coding task change how I should read a GPU number?

Yes. The longer the task, the more of its runtime sits on the sustained plateau and the less on any initial boost. METR’s time-horizon work measures AI agents by the length of tasks they complete at a given success rate, using task suites that together span tasks from one second to 30 hours. That is a capability measure, not a hardware one. What it does is make task duration an explicit axis of evaluation rather than an afterthought.

On the hardware side, long generated responses are decode-heavy. The edge-inference study notes that sustained decode throughput is limited by memory bandwidth, which matters when evaluating always-on agents that produce long outputs. The same study found that, on the two phones it tested, agents querying more often than every few minutes would run at a sustained plateau or Hot-state ceiling, not the peak.

Comparing NVIDIA and AMD when each saturates at a different workload size

A fair comparison across vendors holds model architecture and numerical precision fixed and scales the workload until each device is saturated. A single fixed workload favours whichever device class it happens to suit. The DEEP-GAP study (2026) shows how far apart saturation points can be even within one vendor: the NVIDIA T4 needed batch sizes of 256 or more to reach peak throughput, while the L4 peaked at 16–32.

Between vendors, Artificial Analysis’s System Load Test runs 3-minute phases at increasing concurrency until a serving system stops returning more tokens. On DeepSeek R1, an AMD MI300X system peaked at about 4,100 output tokens/s at about 750 concurrent queries, while an NVIDIA H200 system peaked at about 3,250 tokens/s at about 475. In the same tests, the NVIDIA systems gave faster per-query output at lower concurrency. So which system looked better depended on the concurrency level tested.

Each method fits a different question. A serving load test like Artificial Analysis’s answers how a particular serving stack behaves on particular large models under concurrent queries. LynxBenchAI compares GPUs on one fixed catalogue, adapting only the batch size. For choosing hardware across vendors and device classes on equal terms, LynxBenchAI holds its saturation-scaled, fixed-catalogue method to be the better tool. More on how such lists are built is in our guide to measuring GPU benchmarks for AI.

How LynxBenchAI reports a sustained figure: one declared window at each device’s saturation point

LynxBenchAI reports sustained practical peak. Each test discards a warm-up and then counts completed iterations inside a continuous timed window, at a batch size already raised until throughput plateaus within a defined noise band. Model architecture and precision stay the same for every device within a release, and only the batch size adapts. The run declares a timed window after a discarded warm-up. It makes no claim about the machine’s thermal state inside that window.

The 26Q3 results show why per-category reading matters. The NVIDIA H200 scores 1914.6 for inference in the LynxBenchAI 26Q3 results, against 1304.0 for the AMD Instinct MI300X. The H200 also scores higher for training, at 812.0 against 727.2. The MI300X scores higher for compute, at 1023.3 against 480.8. Individual tests vary further: for fp16 DistilGPT2 inference the H200 scores 4070.9 and the MI300X 3318.5, while for fp16 MobileNetV3 inference the scores are 3178.2 and 733.8. The ordering between these two devices depends on category and model. The serving result above, where the MI300X reached higher peak throughput at high concurrency, comes from a different workload and test method.

The bounds are stated, not implied. The NVIDIA A100-SXM4-80GB records 0.0 for fp8 inference on every model in the 26Q3 results. A precision a device cannot execute is scored as zero rather than quietly skipped. GT is an ordinal aggregate over the three category scores, with no 0–100 scale. The H200’s 1045.0 against the MI300X’s 986.7 tells you which is ahead overall, not by what percentage. Scores compare within the 26Q3 release, not across releases.

Press, Pro and Enterprise are contact-gated. Only the LynxBenchAI Personal Edition has shipped.

Is the plateau compute or memory bandwidth?

To tell which limit you have hit, compare the workload’s arithmetic intensity (FLOPs performed per byte loaded from memory) with the device’s compute and bandwidth ceilings. That is the roofline model. A layer with heavy computation and little memory traffic is compute-bound, and its memory system sits idle. A layer with heavy memory traffic and little computation is memory-bound, and its compute units are underused. In LLM inference, prefill is mostly compute-bound and decode is memory-bound.

The SLIM study (2026) traced an H100 serving plateau to DRAM-bandwidth saturation in decode-phase attention kernels, whose arithmetic intensity stays nearly constant as active context grows. At the plateau, average Tensor Core activity was below 30% and SM issue below 10%, while DRAM read bandwidth averaged close to 80% and often peaked near 100%. The long-term trend points the same way. Over 20 years, peak server FLOPS grew 3.0x every two years, against 1.6x for DRAM bandwidth and 1.4x for interconnect bandwidth.

Signal at the plateau	Compute-bound	Memory-bandwidth-bound
Arithmetic intensity	High, as in batched GEMM during prefill.	Low and roughly constant as context grows, as in decode attention.
Tensor Core and SM activity	Near the hardware limit.	Well below it, for example under 30% Tensor Core activity in SLIM’s H100 measurements.
DRAM bandwidth	Below its ceiling.	At or near saturation.
Effect of quantisation	Little or none once already compute-bound at large batch.	Improves performance while the workload stays memory-bound.
What the sustained figure is evidence for	The device’s usable compute.	The device’s memory system, more than its TFLOPS.

A decode-bound number is evidence about bandwidth, so reading it as proof of the compute rating on the box is a category error. Finding where a live service actually stalls is profiling work on that service, covered in TechnoLynx’s guide to performance engineering for production AI under load.

Planning capacity against the number the GPU holds

A capacity plan built on a burst figure plans against the peak, while for an always-on workload the operative ceiling is the plateau. TechnoLynx covers the economic side in the hidden cost of GPU underutilisation, and the operational side, where the number that matters is the one a service holds day after day, in what the SRE book teaches about running production AI. Before any of that, settle which regime your workload actually runs in: compute-bound or bandwidth-bound, bursty or always-on, and saturated or underfed at the batch size you can afford. A declared, saturated window then tells you how the hardware behaves in that regime, and a peak number cannot.