GPU usage, the percentage nvidia-smi prints as GPU-Util, is the share of a short sampling window during which at least one kernel was executing somewhere on the device. It does not tell you how many streaming multiprocessors (SMs) that kernel occupied, how much arithmetic it did or how much work was completed. The number answers “was the GPU busy?”, not “how fast is it?”.
The mistake is easy to make. ML model monitoring tools and dashboards show utilisation before any other metric (in nvidia-smi, in Grafana, in cloud provider consoles), so high readings reassure people and low ones worry them, often with little connection to what the system is actually doing. Teams that treat the percentage as “high is good, low is wasted compute” are wrong often enough to make real engineering and procurement mistakes on the strength of it.
What does GPU usage actually measure, and what does it not measure?
GPU utilisation is a time measurement. A paper on evolving HPC services for ML workloads on HPE Cray EX systems puts the distinction cleanly: utilisation is the average time a resource was accessed, while saturation is the degree to which that resource was loaded, one temporal and one spatial. The USE method draws the same line between how long a resource was busy and how much extra work it could not service.
NVML, on which nvidia-smi is based, defines two such time shares. GPU utilisation is the percentage of time one or more kernels were executing, and memory utilisation is the percentage of time device memory was being read or written. Neither considers how many SMs were in use during that time.
Utilisation is also not allocation. A job can hold a GPU at full allocation while the device sits idle waiting for data, a network synchronisation step or an abandoned notebook session.
How does the driver calculate GPU utilisation, and why can a nearly idle kernel read 100%?
NVIDIA’s own definition is the percentage of time over the past sample period, between 1/6 of a second and 1 second depending on the GPU, during which one or more kernels was executing. How much of the chip the kernel used never enters the calculation.
A kernel launched with a single thread block runs on one SM, so it reads 100% GPU utilisation while active, but its SM utilisation is at most one over the number of SMs, which is under 1% on an H100. Trainy’s worked example makes the arithmetic concrete: an H100 has 132 SMs, so a kernel that runs for 10 seconds on one of them registers 100% utilisation and 1/132, or 0.7%, SM efficiency. In the extreme case, a kernel that only reads and writes memory while doing zero FLOPS also reads 100%.
How that sampling behaves in practice, and what it means for AI benchmark testing, is covered in more depth in what GPU-Util means and what it misses in AI GPU utilisation testing.
Four ways a full reading hides idle compute
Real workloads produce the same gap:
- Decode. In a GPU-level study of large-batch LLM inference, compute utilisation stayed below 20% through most of each decode step, and no model exceeded 35% average warp usage.
- Low model FLOPS utilisation. Trainy describes a training job sitting at 100% GPU utilisation while using only 20% of the GPU’s theoretical compute.
- Synchronisation and stalled kernels. A study of MFU as a power proxy notes that high utilisation may correspond to memory-bound execution, synchronisation overhead or stalled kernels without implying productive throughput.
- Deadlock. Communication between GPUs often runs as kernels, so a system deadlocked inside a communication kernel looks fully utilised while completing no useful work, as Modal’s engineers point out.
What is the difference between coarse GPU utilization (as reported by nvidia-smi), SM utilization, and memory utilization, and why can they diverge sharply on the same workload?
They disagree because they measure different things at different granularities. SM utilisation reports the fraction of time the SMs spend executing instructions, a finer view than kernel utilisation, which only asks whether a kernel was running anywhere on the GPU. SM occupancy goes further and measures active warps against the hardware maximum. A memory-bound workload can show high DRAM read activity and low compute at the same time, which is the pattern the H100 decode study found: most SMs active, low average warp usage, high DRAM reads.
The counters worth reading instead of, or alongside, GPU-Util are these:
| Counter | What it reports | What it cannot tell you |
|---|---|---|
| nvidia-smi GPU-Util (NVML) | Share of the sample window with at least one kernel running | How many SMs ran or how much math they did |
| NVML memory utilisation | Share of time device memory was read or written | Anything beyond that time share |
DCGM SM activity (DCGM_FI_PROF_SM_ACTIVE) |
Fraction of SMs with work over the interval; warps waiting on memory still count as active | Efficiency: NVIDIA calls 0.8 or more necessary but not sufficient for effective use |
DCGM SM occupancy (DCGM_FI_PROF_SM_OCCUPANCY) |
Resident warps relative to the maximum per SM | Whether the warps are productive: higher occupancy does not necessarily mean better usage |
| Model FLOPS utilisation (MFU) | Observed throughput relative to the system’s theoretical peak FLOPs | Memory-bound performance: MFU stays low even when power and utilisation are high |
Occupancy deserves its caveat stated plainly. NVIDIA’s Nsight Compute guide says higher occupancy does not always produce higher performance, although low occupancy always reduces the ability to hide latency. Even high SM utilisation can hide functional units inside each SM that sit unused, which is why Modal’s glossary points to pipe utilisation as the next level down.
Collecting all of these at once has a practical limit: some DCGM metrics need multiple passes and cannot be read together on certain GPUs, so DCGM multiplexes them by statistical sampling. For the kernel-level view, Nsight Compute reports each unit’s throughput as a percentage of its peak sustained rate. Our guides to ML model metrics that explain GPU bottlenecks versus those that mislead and to reading the output of profiling tools for AI inference take those instruments further.
Does 0% or a jumpy reading mean the GPU is doing no useful work?
A zero means no kernel ran during those samples. Periodic drops to zero come from checkpoint I/O, input preparation, CPU-only evaluation and synchronisation, and the profiler timeline has to be correlated with application phases before a cause is named. Host overhead is the other big source: if Python and the framework cannot select and launch the next kernel before the GPU finishes the current one, the GPU idles. Horace He describes the healthy case, where PyTorch queues kernels asynchronously and runs ahead of the GPU, and the unhealthy one, where operators are so small that the GPU spends most of its time as an expensive paperweight.
The H100 decode study saw the same effect between steps: an idle gap attributed to CPU processing, growing with batch size and reaching up to 30% of decode time for OPT-1.3B. Slow host-to-device transfer over a narrow PCIe link also leaves the GPU waiting for data, which lowers the reading.
One cheap test separates these from a real shortage of work. Overhead does not scale with problem size, so if doubling the batch raises runtime by only around 10%, the workload is likely overhead-bound. In serving, a low reading can simply mean low request volume or small batches. The places these stalls hide are traced step by step in where the real bottlenecks behind low GPU utilisation sit.
Why can a GPU show low utilisation while still delivering high throughput?
Utilisation counts time, not work, so a device whose kernels finish faster spends more of the window idle when host time stays fixed. A study on GPU underutilisation describes the mechanism directly: as GPU compute grows, kernel execution time shrinks, and if CPU time does not shrink with it, the GPU becomes idle.
Is there a “good” GPU utilization percentage to aim for: 70%, 90%, 96% , or does the question itself misread what the metric reports?
For the coarse nvidia-smi figure, no: a single small kernel running back to back can read 100%, so a target set on that number targets the wrong quantity. For predictable GPU workloads, Modal notes that GPU kernel utilisation of 90% or more is both desirable and achievable, unlike CPU utilisation at that level. It does not imply high MFU. The separate question of whether a full reading is a problem on AI workloads has its own page.
Thresholds do exist for other counters. NVIDIA’s DCGM guidance says SM activity below 0.5 likely indicates ineffective use and 0.8 or more is necessary but not sufficient.
Why does the same percentage mean different things on two GPU and software stacks?
GPU utilisation is a vendor-defined counter, and its semantics differ across vendors and architecture generations, so equal values can correspond to very different throughput and power behaviour. The MFU study shows how far apart they sit. On NVIDIA cards, utilisation was already at 40-55% at batch 1 and saturated near 90% well before MFU did, while on the AMD MI210, ROCm reports utilisation from the GRBM_COUNT counter, which acts as a binary activity flag and stays pinned at 100% at every batch size. AMD’s SMI library documents its busy percentage as direct output from the gpu_busy_percent sysfs file.
This is the counter-level version of a position LynxBenchAI holds about performance generally: the unit of AI performance is the AI Executor, meaning the device plus the backend it runs through, its driver, framework and runtime, and no figure belongs to silicon alone.
Why is “make utilisation higher” not always the right optimisation goal?
The right optimisation depends on the regime. If the time goes to memory transfers, more FLOPS will not help, and if it goes to large matmuls, cutting Python overhead will not help. A deadlocked communication kernel reads full utilisation while doing nothing, and in the H100 inference study larger batches consumed substantial GPU memory without proportional throughput gains while significantly degrading latency. Batching generally raises FLOPs per byte moved but adds per-task latency.
TechnoLynx’s GPU Performance Engineering work takes on that diagnosis and the tuning that follows: kernels, memory, parallelism and profiling. How misreading the metric turns into a procurement bill is covered in the hidden cost of GPU underutilisation in AI infrastructure.
If a utilisation reading cannot tell you the GPU is saturated, what can?
Saturation shows up in the throughput curve, not in a counter. Raise the workload, watch completed work per second, and the point where it stops improving is where some resource is fully loaded. The H100 decode study found its bottleneck exactly this way: throughput plateaued past a batch-size knee, and the cause was DRAM bandwidth saturation in attention, with most compute still idle. Lyceum’s guide to low GPU utilisation reaches the same conclusion from the operations side: nvidia-smi utilisation can mislead, and samples per second is the metric that matters.
LynxBenchAI is built on that reading. It counts completed iterations inside a declared window and says nothing about occupancy within it. The method runs in this order:
- Hold the model catalogue and numerical precision identical on every device in the release.
- Discard a warm-up phase so start-up effects do not enter the count.
- Count completed iterations inside one continuous, declared timed window.
- Raise the batch size until throughput stops improving inside a defined noise band, so each device is measured at its own saturation point.
- Bind the result to the full executor: device, backend (
cudafor NVIDIA,cudavia ROCm for AMD,xpufor Intel, or CPU), driver, framework and runtime.
The timed window is a declared interval after a discarded warm-up; the run asserts nothing about the machine’s clock or thermal condition inside it. What it gives is an outcome measure, and executing a real workload and timing it is stronger evidence than a dashboard proxy or a datasheet figure. The same argument applies to headline specifications, as covered in when TOPS is the wrong metric alongside GPU utilisation.
Binding results to the executor is visible in the published data. In the 26Q3 results, the NVIDIA GeForce RTX 5090 on native Linux scores 985.8 on the LynxBenchAI 26Q3 Index for inference, and the same card under WSL scores 891.8. The card is the same and the stack is not, and each executor carries its own score. Those scores compare within 26Q3 only.
To get completed-work numbers for your own machine, run the LynxBenchAI Personal Edition with pip install lynxbench-ai. It runs the published catalogue rather than your production model, so the output tells you where your executor sits among other submissions of the same device; it does not tell you how your own service will behave.
A saturation-scaled score settles how much work an executor holds on a fixed catalogue, while your own workload’s ceiling still has to come from timing that workload, with DCGM and Nsight explaining why the curve flattens where it does. Utilisation earns a place on the dashboard as an idle alarm and a hint about host stalls, and the questions about speed belong to the throughput curve.
