The first 200 iterations looked great
A team kicks off a training run, watches the throughput counter climb during the first few hundred steps, and records the number for the weekly status report. Two hours later, the counter has dropped 15%. By the overnight checkpoint, it’s fluctuating between values that differ by 20%. Nobody changed anything.
This is normal behaviour, and the fact that it surprises people reveals a widespread assumption: that hardware performance is a fixed property you measure once and then rely on. In practice, AI workload performance is a time-varying signal shaped by thermal dynamics, power management, memory allocation patterns, scheduling behaviour, and framework-level optimisation decisions that play out over minutes to hours.
One clarification before the mechanisms, because it changes how the rest of this reads. Temporal behaviour belongs to the AI Executor — the device together with the backend, driver, framework, and runtime that drive it — not to the silicon on its own. The same card under a different CUDA version, a different allocator setting, or a different data loader traces a different curve.
Warmup effects are real and measurable
When a GPU begins executing a workload, several subsystems are still reaching their operating state.
CUDA contexts need to be initialized. Kernel launches incur higher overhead on first invocation because the runtime is compiling and caching PTX code. Memory pools haven’t been pre-allocated yet, so early allocations trigger expensive system calls. The GPU’s clock frequency is ramping from idle to boost state. The host-side data pipeline — data loaders, augmentation routines, prefetch buffers — is still filling.
The combined effect is that the first several minutes of a workload are systematically unrepresentative of what follows. Throughput measured during this warmup phase is typically either lower than steady-state (because subsystems are still initializing) or briefly higher than steady-state (because the GPU is at peak boost frequency before thermal limits kick in).
It is worth separating warmup from the cold-start problem, since the two are often conflated. Cold start is the first-request penalty paid when nothing is resident yet — the model isn’t loaded into GPU memory, the serving process hasn’t spun up, the container is still scheduling. Warmup is the subsequent ramp once execution has begun: contexts initializing, clocks climbing, pools filling. Both inflate or depress the earliest numbers, and both are expected. Neither makes a short benchmark invalid on its own — they just mean the benchmark measured the cold-and-warming regime, and it has to say so.
We’ve discussed the gap between peak and steady-state measurement in how peak and sustained performance diverge — warmup effects are one of the primary mechanisms through which that divergence manifests.
Thermal and power dynamics reshape the performance curve
After warmup completes and the GPU reaches sustained load, thermal dynamics become the dominant source of performance variation.
Modern data center GPUs like the NVIDIA A100 and H100 are designed to operate near their thermal limits under AI workloads. The GPU starts at boost clock frequencies, delivers peak throughput for minutes to tens of minutes, then gradually reduces clock speed as junction temperature rises toward the thermal limit. The clock reduction is not a failure — it’s by design. The power management firmware maintains temperature within safe operating range by trading clock frequency for thermal headroom.
The practical effect is a throughput curve that starts high and settles lower. In our experience the settlement lands somewhere around 5–15% below the initial peak, depending on workload intensity, cooling configuration, and ambient conditions (observed range across the systems we have profiled; not a benchmarked rate). In dense multi-GPU nodes — eight GPUs sharing an enclosure — thermal interaction between neighbouring cards makes this more pronounced, and the cards at interior positions settle at lower clocks than the ones at the edges.
As detailed in how power and thermal constraints govern sustained performance, these physical constraints are first-class determinants of what the hardware actually delivers over time. Any measurement that ignores them captures a transient state, not the operating reality.
Memory pressure builds over time
Long-running workloads often experience performance changes driven by memory dynamics that only become visible after extended execution.
Framework-level memory allocators (PyTorch’s caching CUDA allocator, for instance) manage GPU memory through pooling and caching strategies. Early in a run, the allocator is building its pool, and allocations are fast because free memory is abundant. As the run progresses and more memory is allocated, fragmentation can increase, leading to occasional expensive defragmentation or fallback to CPU-side memory operations.
In inference serving, KV cache growth over long contexts can push memory utilization toward capacity limits, triggering eviction policies or degrading batch scheduling efficiency. The first thousand requests might perform well, but performance at the hundred-thousandth request — after hours of continuous serving — can look materially different.
Garbage collection behaviour in Python-heavy stacks adds another time-dependent factor. Periodic GC pauses create throughput dips that are invisible in short benchmarks but visible in production monitoring.
Training runs add a subtler wrinkle on top of these system effects. Loss can appear to increase or plateau over a long run — a learning-rate warmup or decay schedule, a curriculum that shifts to harder examples, or a periodic evaluation phase can all move the curve — without anything being wrong. In our experience the temporal shape of a training metric reflects the schedule and the data as much as the hardware, so a rising loss segment is not, by itself, evidence that the system is misbehaving.
Scheduling and system-level drift
Beyond the GPU itself, the broader system introduces its own time-dependent behaviour.
OS-level scheduling decisions affect CPU-side preprocessing performance. Under sustained load, the kernel’s scheduler may migrate threads, compete with background processes, or encounter NUMA-related latency if memory affinity isn’t carefully managed.
Network-attached storage or distributed file systems exhibit throughput variation under sustained read patterns, especially when multiple nodes compete for I/O bandwidth. A data pipeline that kept up for the first hour of training may fall behind as other jobs on the same storage fabric increase their load.
Multi-tenant environments add another layer. Performance on a shared cluster at 2 AM (light load) versus 2 PM (peak utilization) can differ substantially, not because of anything the workload did, but because the system context changed. One trap here: GPU utilization can read 90% or 96% while sustained throughput is still drifting downward. High utilization only says the GPU was busy — not that it was doing useful work at a stable rate. When the headline utilization stays pinned but throughput falls, the temporal bottleneck has usually moved off the compute units and onto something else: clocks throttled by thermals, the data pipeline stalling, or memory traffic dominating. Utilization is a presence signal, not a productivity signal.
Temporal effects that shift AI performance
Evidence class for the whole table: engineering-observed patterns from profiling work, not a published benchmark. Timescales are indicative.
| Effect | Timescale | Mechanism | Impact on measured performance |
|---|---|---|---|
| Warmup / initialization | Typically first 1–5 minutes | CUDA context init, PTX compilation, memory pool setup | Initially lower or briefly higher throughput |
| Thermal settling | Typically 5–30 minutes | Junction temperature rises, clocks reduce to maintain thermal limits | Sustained throughput settles below initial peak |
| Memory pressure | Hours | Allocator fragmentation, KV cache growth, GC pauses | Intermittent throughput dips, increased tail latency |
| System-level drift | Hours to days | OS scheduling changes, storage contention, multi-tenant interference | Variable throughput depending on external load |
Does a GPU actually degrade over time?
The question people usually mean is not the one they ask. “Do graphics cards wear out” is a hardware-lifetime question — fans, thermal paste, and VRM components do age, and a card whose cooling has degraded will throttle earlier than it did when new. But that is a multi-year effect, and it is almost never what someone is looking at when a number drops between Monday and Wednesday.
What moves a performance figure on that timescale is the within-run temporal behaviour described above, plus changes in the executor around the silicon: a driver update, a framework version bump, a different allocator configuration, a neighbouring tenant. Diagnosing “degradation” therefore starts with holding the executor fixed and re-running the same workload with the same declared window. If the number reproduces, nothing degraded; the earlier figure just came from a different point on the curve.
Timing methodology is where this goes wrong most often. A naive PyTorch loop that wraps a forward pass in time.time() reports a number the system never sustains, for three reasons: CUDA kernels are launched asynchronously, so without torch.cuda.synchronize() you are timing queue submission rather than execution; the first iterations include compilation and allocation costs; and a single iteration says nothing about the rate the machine holds. The correction is mechanical — synchronize, discard warmup iterations, then time a continuous run of many iterations and divide.
Why does time-varying performance matter for measurement?
If AI performance is time-varying, then the question “what’s the throughput?” has no single correct answer. The answer depends on when you measured: during warmup, at thermal peak, after thermal settling, during a memory fragmentation event, or at steady state.
This is why measurement methodology must specify a temporal protocol. How long was the workload run before measurement began? Over what time window was the measurement taken? Were warmup iterations excluded? At what workload size?
Without these details, a benchmark number is ambiguous. Two measurements of the same hardware running the same workload can disagree substantially if they were taken at different points on the performance-over-time curve. The disagreement isn’t noise — it’s the natural consequence of measuring a time-dependent phenomenon at different times.
A worked example of that discipline: a LynxBenchAI run discards a warm-up phase, then counts completed iterations inside one continuous declared timed window, at a workload size already raised until throughput stops improving inside a defined noise band. Each test is one such window — not a median across repeated trials — and the window is declared so a reader can see exactly what the figure covers. What the protocol deliberately does not assert is anything about the machine’s temperature or clock condition inside that window. One window is one window; its integrity comes from being declared, not from being long, and it says nothing about variance across a day, a chassis, or a season. Figures produced under different release names are not comparable to each other either, because the thing being measured changed between them.
There is one more distinction worth keeping clean, because search traffic keeps collapsing it. Inference-time scaling — deliberately spending more compute per request to get a better answer, through longer reasoning chains or wider sampling — is a design choice that changes the workload. Performance drift during a sustained run is the machine’s behaviour under a fixed workload. Both make latency rise. Only one of them is something you chose.
Performance drift has a model-quality sibling worth keeping separate from it: data drift versus model drift sets out how each one changes the production reliability response, and model drift vs hardware drift separates the two decay curves.
An argument about drift on one specific machine can stop being an argument. LynxBenchAI is a benchmarking methodology for AI hardware — sustained performance across the complete hardware-and-software stack, reported per precision, with bounded optimisation — and the Personal Edition is a pip install lynxbench-ai away (Python 3.11+, Linux or Windows via WSL2, roughly 15–30 minutes for a run). What comes back is not a verdict on the silicon. It is a number with a stated scope.
So the question to hold onto is not “did performance drop?” but: which declared window produced the figure you are comparing against, and does the new one cover the same thing?
Frequently Asked Questions
Why does observed AI performance often change between the first second of a workload and the tenth minute?
The first seconds are dominated by initialization: CUDA context setup, PTX compilation, memory pool construction, and the host-side data pipeline filling. The GPU is also ramping from idle to boost clock. By the tenth minute those transients have resolved and thermal settling has begun pulling clocks down from their initial peak. The curve you observe across that window is the superposition of warmup recovery and thermal descent.
How do warmup and cold-start effects differ, and why does neither invalidate a short benchmark by itself?
Cold start is the first-request penalty paid when nothing is resident yet — the model isn’t loaded, the serving process hasn’t spun up, the container is still scheduling. Warmup is the subsequent ramp once execution has begun: contexts initializing, clocks climbing, memory pools filling. Both depress or inflate the earliest numbers, and both are expected behaviour. A short benchmark that captures either regime is still informative as long as it declares which regime it measured and what it excluded.
Why isn’t every change in sustained performance a sign of a fault?
Clock reduction under thermal load is by design — power management firmware trades frequency for thermal headroom to keep junction temperature in range. Allocator fragmentation, KV cache growth, and scheduler migration are also expected behaviours of healthy systems under sustained execution. Treating every dip as a fault leads to chasing phantoms; the discipline is to characterize the expected curve first, then flag deviations from it.
What should a benchmark report disclose so that temporal variance can be reasoned about?
At minimum: what warmup was excluded, the timed measurement window the figure covers, the workload size used, and the full executor — device plus backend, driver, framework, and runtime. A LynxBenchAI run discards a warm-up phase and then counts completed iterations inside one continuous declared window at a saturated workload size; it asserts nothing about the machine’s thermal or clock condition inside that window, which is itself part of the disclosure.
When people ask whether a GPU “degrades over time”, are they describing silicon that has worn out, or the within-run temporal effects that actually move a performance number?
Almost always the latter. Cooling hardware does age over years, and a card with degraded thermal paste or a failing fan will throttle earlier than it once did. But a figure that changes between two runs this week is far more likely explained by warmup, thermal settling, memory pressure, a driver or framework change, or a noisier neighbour on the same node. Hold the executor fixed, re-run with the same declared window, and see whether the number reproduces.
How do you measure inference time correctly in a framework like PyTorch, and why do naive timing loops report a number the system never actually sustains?
CUDA kernels launch asynchronously, so timing a forward pass without torch.cuda.synchronize() measures queue submission rather than execution. Early iterations also carry compilation and allocation costs, and a single iteration says nothing about a sustained rate. The correction is to synchronize, discard warmup iterations, then time a continuous run of many iterations and divide — and to report the window, not just the mean.
Why is inference-time scaling (spending more compute per request) a different question from performance drifting during a sustained run?
Inference-time scaling is a deliberate design choice: longer reasoning chains or wider sampling change the workload in exchange for answer quality. Temporal drift is what a machine does to a fixed workload as thermals, memory, and system context evolve. Both push latency up, which is why they get conflated, but only one of them is a decision you made — and only one is fixed by changing the measurement protocol rather than the system.
