The most-watched number in the monitoring panel is also the most misread
Of all the metrics available in a GPU monitoring stack, utilization is the one people fixate on first. It shows up in nvidia-smi, in Grafana dashboards, in cloud provider consoles, and in every performance review meeting. High utilization makes people feel good. Low utilization triggers concern. Both reactions are frequently disconnected from what is actually happening in the system.
The reason is straightforward but easy to overlook: GPU utilization is not a performance metric. It is a device-activity proxy — a signal about how much of a particular sampling window had active kernels — and the gap between that signal and actual system performance is wide enough to produce genuinely wrong conclusions in both directions.
This piece is about the metric itself: what it measures, what it hides, and what you should measure instead when the question is whether a device is actually saturated.
What the utilization counter actually reports
The utilization number you see in nvidia-smi or through NVML is the percentage of time over the last sampling interval (often one second) during which one or more GPU kernels were executing. That is all. Not “percentage of compute capacity used.” Not “fraction of theoretical throughput achieved.” Not “efficiency.”
The formula matters, because it explains the pathology. The driver samples device activity and asks a binary question — was work resident? — then reports the fraction of samples that answered yes. A kernel occupying a single streaming multiprocessor out of dozens answers yes exactly as loudly as a kernel that fills the entire device. That is why a nearly idle GPU can register as 100% busy: the counter has no denominator for how much hardware the resident work engaged.
This definition has consequences in both directions. A kernel can execute continuously — pinning the counter at 100% — while doing inefficient work: poor memory access patterns, redundant computation, low arithmetic intensity. The device is busy in the counter’s sense, while useful work per unit time sits far below what the hardware can deliver.
Conversely, a workload can deliver high effective throughput while the counter shows 60%, because the work arrives in dense bursts followed by brief host-side orchestration or memory-bound operations that the sampler does not classify as active. The system is productive. The metric cannot see it in the way you would expect.
What GPU utilization does and does not tell you
| Utilization reading | What people assume | What it actually means |
|---|---|---|
| High (>90%) | GPU is working hard and efficiently | At least one kernel was resident for most of the sampling window — says nothing about kernel efficiency, occupancy, or throughput |
| Low (<50%) | GPU is underused; money is being wasted | Kernels were inactive for much of the window — could be a real bottleneck, or a memory-bound or burst-structured workload |
| Averaged over time | A proxy for system health | An activity signal that hides workload phase structure, burst patterns, and the peak-versus-steady-state distinction |
| SM utilization | Same thing, finer grain | A genuine occupancy signal — fraction of streaming multiprocessors doing work — and often orders of magnitude below the coarse figure |
Why utilization and throughput do not track each other
The disconnect comes from the fact that utilization measures activity, while performance is an outcome.
In throughput terms, what matters is how much useful work the system completes per unit time — tokens generated, images processed, training steps finished. That depends on the full execution pipeline: whether kernels are efficient, whether data movement is well organised, whether the software stack is exploiting the hardware’s strengths, whether the system is operating in a favourable regime.
Utilization captures none of that. It captures whether kernels were scheduled. A high-utilization system running wasteful memory access patterns will show a healthy dashboard while delivering mediocre throughput. A well-optimised system that finishes work faster — with efficient attention kernels like FlashAttention, operator fusion via torch.compile, and tight memory management — may show lower utilization, because it completes the same work in shorter bursts.
We see this inversion regularly in the engagements we work on: an optimisation that improves real throughput noticeably also decreases the utilization number, because each batch finishes faster and the device spends more of each window idle between dispatches. (Observed pattern across TechnoLynx and LynxBenchAI runs; not a published benchmark rate.) If utilization is your success metric, you have just been told your optimisation made things worse. It did not.
Averaging destroys the structure you needed
Utilization is averaged over a sampling window, and that averaging throws away the workload’s temporal structure.
A serving workload handling variable-length requests may have busy and quiet periods inside every one-second interval. The average lands at 55%, but the real execution pattern is bimodal: near 100% during computation, near 0% during queueing, preprocessing, or waiting for the next batch. The 55% is a statistically real number that describes no actual moment in the system’s operation.
Workloads with distinct phases — graph capture under torch.compile, warm-up, steady-state inference, intermittent garbage-collection pauses — produce traces that blend those phases into one figure. The result tells you nothing about which phase dominated, which was the bottleneck, or what steady state actually looks like. This is one reason benchmarks fail to match real AI workloads.
Understanding that peak and steady-state performance reflect fundamentally different temporal regimes sharpens the problem. Utilization averaged across those regimes is averaging across the most important dimension the system has, then presenting the result as a single meaningful state.
The high-utilization trap
There is a mirror-image failure that gets discussed less: treating high utilization as confirmation that the system is performing well.
High utilization means the device had resident kernels most of the time. It does not mean those kernels were doing the right work efficiently, that useful output was flowing at a high rate, that end-to-end latency was acceptable, or that the user experience met its target.
You can manufacture high utilization by over-batching a latency-sensitive serving system — the GPU stays busy because there is always a large queue, while individual request latency spikes because each request waits longer before service. The dashboard says healthy. The users disagree. You can also manufacture it with poorly optimised kernels that occupy compute units without converting that occupancy into output.
Coarse GPU utilization versus SM utilization versus memory utilization
The headline number hides a second layer entirely. The figure nvidia-smi reports is coarse: it answers whether any kernel was resident during the window. SM (streaming multiprocessor) utilization — visible through Nsight Systems, DCGM, or the NVML utilization-rate companions — measures what fraction of the device’s SMs were doing work, which is much closer to a real occupancy signal. Memory utilization is a third, independent axis: it reports how much of the sampling window the memory controller spent moving data, which is why a bandwidth-bound kernel can show high memory utilization and unremarkable SM occupancy at the same time.
The three diverge sharply on the same workload. A single small kernel occupying one SM out of dozens can pin the coarse counter at 100% while SM utilization sits in single digits — busy by the scheduler’s definition, nearly idle by the hardware’s. We see this gap most often with tiny launch-bound kernels, under-batched inference, and serial dispatch patterns where the host cannot feed the device fast enough to fill it. When the coarse number looks healthy and throughput does not, SM utilization is usually where the real story is. Pinned coarse utilization is the start of a question, not the answer to one.
There is also a portability problem. Any utilization percentage is bound to the AI Executor that produced it — device, backend (cuda for NVIDIA, cuda via ROCm for AMD, xpu for Intel, or CPU), driver, framework, and runtime. Two stacks sampling the same nominal workload can report different occupancy because they schedule, fuse, and dispatch differently. The same percentage does not mean the same thing across executors.
If the dashboard cannot settle saturation, what can?
Utilization is not useless. It is insufficient on its own, and actively misleading when elevated to the status of a performance metric.
The productive way to use it is as one signal among several, read alongside measures that describe outcomes directly: throughput, request latency distributions (p50, p95, p99), memory bandwidth counters, kernel-level traces from Nsight Systems, and end-to-end timing that shows where wall-clock time goes. If utilization is low and throughput is also low, that is worth investigating — but the investigation should chase where time goes, not how to make the device busier. If utilization is low and throughput meets its target, the number is describing the workload’s structure, not a defect.
For the specific question “is this device saturated?”, the answer is a measurement, not a reading. A LynxBenchAI run counts completed iterations inside a continuous timed window, after a discarded warm-up — an outcome measure rather than an occupancy measure. Saturation is then established from the throughput curve rather than from any dashboard dial: batch size is raised until throughput stops improving inside a defined noise band. That procedure answers the question a utilization percentage structurally cannot, and it says nothing about occupancy inside the window because occupancy is not what it measures.
Executing a real workload and timing it is stronger evidence than any dashboard proxy or datasheet figure. A reader suspicious of their utilization graph can install LynxBenchAI with pip install lynxbench-ai, run the same catalogue on their own machine, and compare completed work against other submissions of the same device on the public leaderboard — which replaces “my utilization looks wrong” with a number that has a population to sit in. Only the free, non-commercial Personal Edition has shipped.
In production monitoring, the same caution is what application performance management tools for AI inference are built to enforce — and where they still leave gaps.
So the question worth carrying back to the monitoring panel is not “why is utilization low?” but: what timed window, on what declared executor, would settle whether this device is doing the work you are paying it for?
Frequently Asked Questions
What does GPU utilization actually measure, and what does it not measure?
The counter in nvidia-smi and NVML reports the percentage of time during the last sampling window — usually one second — that at least one kernel was executing on the device. It does not measure compute capacity used, fraction of theoretical throughput achieved, memory bandwidth consumed, or kernel efficiency. It is a scheduling-activity proxy, not a productivity measure.
Why can a GPU show low utilization while still delivering high throughput?
A well-optimised workload often completes work in dense bursts, then waits briefly for the next batch, host-side orchestration, or memory-bound operations the sampler does not count as active. The system is productive across the wall-clock window while the activity sampler sees gaps. Improvements like FlashAttention, operator fusion via torch.compile, or tighter memory management can raise real throughput while pushing the utilization number down.
Why is “make utilization higher” not always the right optimisation goal?
Because utilization measures activity, not outcome. You can drive it upward by over-batching a latency-sensitive serving system — keeping the device busy at the cost of p95 and p99 request latency — or by running poorly optimised kernels that occupy the hardware without converting activity into useful output. Either way the dashboard improves while the user experience or the cost per token gets worse.
Is there a “good” GPU utilization percentage to aim for — 70%, 90%, 96% — or does the question itself misread what the metric reports?
The question misreads the metric. A percentage of a binary activity sample has no target value, because the same figure can come from a saturated training job, a single launch-bound kernel occupying one SM, or an over-batched server trading latency for occupancy. There is no threshold that separates those cases. The answerable version of the question is whether throughput has stopped improving as batch size rises, which is a curve, not a percentage.
What is the difference between coarse GPU utilization (as reported by nvidia-smi), SM utilization, and memory utilization, and why can they diverge sharply on the same workload?
Coarse utilization answers a binary question — was any kernel resident during the window. SM utilization measures what fraction of the streaming multiprocessors were doing work, which is far closer to real occupancy. Memory utilization reports how much of the window the memory controller spent moving data. A small kernel on one SM out of dozens can pin the coarse counter at 100% while SM utilization sits in single digits, and a bandwidth-bound kernel can show high memory utilization with unremarkable SM occupancy.
How is GPU utilization actually calculated by the driver, and why does that formula explain how a nearly-idle kernel can register as 100% busy?
The driver samples the device repeatedly across the interval and asks whether any work was resident, then reports the fraction of samples that answered yes. There is no term in that formula for how much of the parallel hardware the resident work engaged. A kernel using one SM answers yes exactly as a kernel filling the whole device does, so a nearly idle GPU can report 100% busy without contradiction.
If a utilization dashboard cannot settle whether a device is saturated, what measurement does — and how does a timed run with a discarded warm-up and a batch-size sweep answer the question instead?
A LynxBenchAI run counts completed iterations inside a continuous timed window after a discarded warm-up, which is an outcome measure rather than an occupancy measure. Saturation is then read off the throughput curve: batch size is raised until throughput stops improving inside a defined noise band. That gives a decision — saturated or not — grounded in completed work on a declared AI Executor, rather than an inference drawn from an activity percentage.
