You open the monitoring panel, and the number looks wrong
GPU utilization: 38%.
If you’ve deployed AI workloads in production, you’ve probably had this moment. The number feels like an accusation — the hardware is “idle,” money is being burned, and something must be broken. The instinct is to treat utilization as a scoreboard: higher is better, lower means waste.
That instinct is understandable, but it leads to some of the most persistent misdiagnoses in AI infrastructure. Low utilization is common in real AI systems, and by itself it doesn’t tell you whether the hardware is being wasted. In many cases, the system is working as hard as the workload allows — the bottleneck just isn’t where the utilization counter is looking.
What the utilization number actually captures
GPU utilization, as reported by tools like nvidia-smi, is not “percentage of performance being used.” It is a measurement tool’s estimate of how much time the GPU had at least one active kernel during a sampling window. That definition is narrower than most people realize.
It focuses primarily on compute unit activity and averages it across a time window that can hide the internal structure of the workload entirely. A workload that runs intense compute bursts separated by periods of memory-bound operations, synchronization, or host-side orchestration will report lower utilization than a workload that keeps the compute units continuously busy — even if both workloads are delivering comparable throughput for their respective tasks.
This isn’t a flaw in the metric. It’s a limitation of what a single aggregated number can express. Utilization is a proxy for one aspect of device activity, not a summary of system performance. The confusion happens when people treat it as the latter.
There is a second, less-discussed failure mode running in the other direction. A kernel that occupies the device continuously while doing very little arithmetic — a launch-bound sequence of tiny operators, a spin on a synchronization primitive, a poorly shaped gather — will register as high utilization. The percentage moves; throughput does not. Utilization is an occupancy signal, and occupancy and productivity are not the same variable.
AI workloads are often not compute-bound
A large fraction of real inference work — especially autoregressive decoding with transformer models — is dominated by memory behaviour rather than arithmetic throughput. When a workload is memory-bandwidth-bound, the compute units physically cannot stay fully occupied because they spend time waiting for data. The utilization counter shows this as “low utilization,” but what it actually reflects is that compute is not the limiter.
The same pattern appears with irregular operators, small batch sizes, frequent synchronization between operations, and workloads with unfavourable memory access patterns. In all of these cases the GPU is not “idle” in any meaningful sense — the system is doing real work, it’s just that the work doesn’t look like continuous arithmetic from the compute units’ perspective.
We see this regularly with serving workloads that handle variable-length sequences: the system is handling real requests and delivering real throughput, but the utilization dashboard makes it look like the GPU is coasting (observed across our engagements; not a benchmarked rate). The dashboard is measuring the wrong subsystem for that regime.
The pipeline reality: work arrives in stages, not as a steady stream
AI execution is not one long kernel that runs forever. It’s a pipeline. Work has to be prepared on the host side, moved to the device, scheduled, launched, executed, synchronized, and the results moved back. Some workloads involve multiple back-and-forth stages per step, especially when the framework does host-side orchestration between device kernels.
Utilization counters reward continuous device activity, but many AI workloads are inherently bursty or staged at the device level. Inference services, in particular, often alternate between short, intense bursts of GPU work and gaps where the system is doing queueing, preprocessing, or waiting on upstream components. If your monitoring window averages those gaps together with the bursts, you get a utilization number that makes a bursty-but-productive system look idle by construction.
The bottleneck rule is simple and powerful: the system can only run as fast as its slowest stage. When the slowest stage is outside the GPU compute units — a common situation — the compute units cannot remain saturated, and low utilization becomes the expected outcome, not a defect.
Common bottleneck categories when utilization appears low
| Bottleneck type | Symptoms | What utilization shows | Actual limiter |
|---|---|---|---|
| Compute-bound | GPU near thermal limits, high arithmetic activity | High (as expected) | Arithmetic throughput of the compute units |
| Memory-bandwidth-bound | High HBM traffic, moderate utilization | Lower than expected | Data movement speed, not compute capacity |
| Host-bound | Gaps between kernel launches, CPU at high load | Low, with idle periods | CPU preprocessing, data loading, or orchestration overhead |
| Pipeline-bound | Intermittent utilization spikes and dips | Variable, averaging low | Synchronization between stages, PCIe transfers, or upstream dependencies |
| Launch-bound | Thousands of tiny kernels, low arithmetic per launch | Deceptively high | Per-launch overhead, not the kernels themselves |
Evidence class for this table: observed-pattern — these are regimes we encounter repeatedly across profiling work, not rows from a published benchmark suite.
Why the CPU-to-GPU ratio of the host changes the picture
The host matters more than utilization dashboards imply, and it matters more the more orchestration your workload does. Agentic and multi-step pipelines are the clearest case: tool calls, retrieval steps, parsing, routing decisions, and control flow between model invocations all execute on the CPU. The GPU waits during that work. On a box with a generous CPU-to-GPU ratio — many cores per accelerator — that orchestration overlaps with device work and hides itself. On a box provisioned with few cores per accelerator, the same pipeline starves the device and the utilization counter drops, even though nothing about the model or the GPU changed.
This is why “low GPU usage, no CPU bottleneck” is such a common and misleading report. Aggregate CPU utilization can look modest while a single orchestration thread is saturated, because per-core saturation on the critical path does not show up in a host-wide average. The relevant question is not how busy the CPU is overall but whether the thread feeding the device ever stops feeding it.
When is low GPU utilization the correct operating point?
There’s a particularly important scenario where low utilization isn’t a problem to fix — it’s a design choice.
Inference services that optimize for latency deliberately avoid the conditions that maximize utilization. Low batching keeps response times predictable. Avoiding aggressive queueing prevents tail-latency spikes. Multi-tenant isolation means giving up global packing efficiency in exchange for fairness and stability.
These trade-offs are often correct for the service’s actual objective. Pushing for higher utilization in a latency-sensitive system typically means increasing batching, which means individual requests wait longer, which means the service gets “busier” by the dashboard’s definition but worse by the user’s definition.
So treating utilization as a maximization target is only valid if throughput is your only objective. If latency, predictability, or isolation matter — and in serving workloads they almost always do — then the “right” utilization number might be substantially below 100%, and that’s engineering, not waste.
The question that actually helps
If utilization isn’t the diagnosis, what is?
The shift we find most useful is to stop asking “why is utilization low?” and start asking “where does time go?” The answer to the first question is often just “the bottleneck is somewhere else” — true but not actionable. The answer to the second points at the actual limiter: memory traffic patterns, host-side scheduling overhead, PCIe transfer latency, synchronization contention, or something external to the GPU entirely. Tooling exists for this — CUDA graph capture to collapse launch overhead, NVIDIA Nsight Systems or PyTorch’s profiler to see the host-and-device timeline side by side — and all of it answers the second question rather than the first.
There’s a useful sanity check before you start fixing anything: separate an expected idle pattern from a real stall. A memory-bandwidth-bound decode step or a deliberately low-batch latency service will report low utilization by construction — that is the workload’s natural shape, not a bug. A genuine stall looks different: the GPU sits idle while a host-side data loader, a synchronization barrier, or an upstream dependency keeps it starved of work it could otherwise be doing. The tell is whether the limiter is intrinsic to the computation or an avoidable gap in feeding it. The first is the workload; the second is worth fixing.
This is also where the so-called “30% rule” comes apart. People sometimes cite a threshold — utilization should sit above 30%, or 50%, or some other figure — as if a single number could certify health. It can’t. Utilization is context-dependent: the same model on the same hardware will land at very different numbers depending on batch size, framework orchestration, and input pipeline shape. There is no portable heuristic that maps a utilization percentage onto “healthy” or “wasted,” because the metric measures one subsystem’s occupancy, not whether the system is doing the right work efficiently. A threshold rule is a scoreboard dressed up as diagnosis.
Understanding that performance is an execution property of the full system is what makes this reframe stick. Utilization is one observation of one stage in a pipeline. It’s useful as a clue. It’s dangerous as a scoreboard.
Settling the argument with a measurement instead of a percentage
There is a substitution available here, and it is the practical payoff of the whole reframe. A percentage tells you how occupied one subsystem looked during a sampling window. A sustained-throughput measurement tells you what the machine actually holds: sustained behaviour under heavy load, not a transient burst, is the operationally relevant figure, because a burst is not deployable performance.
The methodology behind that figure is deliberately narrow about what it observes. Workload size is raised until throughput stops improving inside a defined noise band — saturation is found, not assumed, which is why a laptop GPU and a data-centre accelerator can each be measured where they genuinely saturate rather than at some borrowed batch size. A warm-up phase is discarded, then completed iterations are counted inside a continuous timed window. The integrity is in declaring that window, not in its length; the run does not claim to capture a thermal state or a settled clock regime, because it does not observe one.
A reader arguing about an underutilized card can produce a number for it: pip install lynxbench-ai, fifteen to thirty minutes, on the machine in question. That converts “the GPU looks idle” into “here is what this machine sustains under load, at this precision” — which is a claim you can act on. Results are named per release, and results from different release names are not comparable with each other; each release measures different things. The Personal Edition is the released one, free for non-commercial use.
In a running system, the move from “why is utilization low?” to “where does time go?” is the opening step of profiling AI inference — the applied-engineering counterpart to locating the real limiter.
So the closing question is not what your utilization number is. It is narrower and more answerable: which stage of your pipeline stops improving first when you raise the load, and did you measure that, or infer it?
Frequently Asked Questions
Why is low GPU utilization common on AI workloads even when nothing is broken?
Because many AI workloads — particularly autoregressive decoding, small-batch inference, and pipelines with host-side orchestration — are limited by memory bandwidth, data movement, or scheduling rather than arithmetic throughput. The compute units cannot stay saturated when they are waiting on the slowest stage of the pipeline, so low utilization becomes the expected outcome, not a defect.
What does the GPU utilization percentage reported by tools like nvidia-smi actually count, and what does it miss?
nvidia-smi reports an estimate of how much time the GPU had at least one active kernel during a sampling window. It captures compute unit activity averaged across that window, so it misses the internal structure of the workload — memory-bound phases, synchronization gaps, host-side orchestration, and PCIe transfers all register as “not utilization” even when the system is doing productive work.
Where do AI workload bottlenecks usually live — compute, memory bandwidth, data movement, or scheduling?
In real systems they are spread across all four, and most often outside the compute units. The table above maps five common regimes: compute-bound, memory-bandwidth-bound, host-bound, pipeline-bound, and launch-bound. Memory bandwidth dominates transformer decoding; host preprocessing and orchestration dominate many serving stacks; synchronization and PCIe transfers show up as intermittent utilization patterns.
How should a team diagnose an “underutilized” GPU before assuming the hardware is wasted?
Stop asking “why is utilization low?” and start asking “where does time go?” That reframes the diagnosis toward the actual limiter — memory traffic, host scheduling, PCIe latency, synchronization, or upstream dependencies — rather than the symptom. Once the real limiter is identified, you can decide whether it is fixable, fundamental, or simply the natural shape of the workload.
Why can a GPU report high utilization while doing very little useful work — and why does the percentage move without throughput moving?
Utilization measures occupancy, not productivity: any kernel resident on the device counts, however little arithmetic it performs. A long sequence of tiny launch-bound operators, a spin on a synchronization primitive, or a badly shaped memory gather can keep the device nominally busy while delivering almost no completed work. That is why the percentage can climb while throughput stays flat — the two variables are not linked.
How does the CPU-to-GPU ratio of a host shape apparent GPU utilization, particularly for agentic or multi-step AI pipelines where orchestration work sits on the CPU?
Agentic pipelines run tool calls, retrieval, parsing, and routing on the CPU between model invocations, and the GPU waits during that work. A host with many cores per accelerator overlaps that orchestration and hides it; a host with few cores per accelerator starves the device and the utilization number drops with no change to the model. Aggregate CPU load can still look modest, because a single saturated orchestration thread on the critical path does not show up in a host-wide average.
What can a reader conclude about their own machine from a single sustained-throughput measurement that a utilization percentage cannot tell them?
A sustained measurement states what the machine holds under load at a given precision, with the workload raised until throughput plateaus inside a defined noise band and a warm-up phase discarded before the timed window is counted. A percentage only reports how occupied one subsystem looked during a sampling interval. The measurement settles the “is this card wasted?” argument empirically — pip install lynxbench-ai, fifteen to thirty minutes — whereas the percentage can only start it, and results from different release names are not comparable with one another.
When does a low GPU utilization number actually indicate a real data-pipeline or scheduling stall worth fixing, versus an expected idle pattern for the workload?
An expected idle pattern is intrinsic to the computation: a memory-bandwidth-bound decode step or a deliberately low-batch latency service will report low utilization by construction. A real stall is an avoidable gap in feeding the GPU — a host-side data loader, a synchronization barrier, or an upstream dependency keeping the device starved of work it could otherwise be doing. The diagnostic test is whether the limiter is intrinsic to the work or an avoidable gap in supplying it; the first is the workload’s natural shape, the second is worth fixing.
What is the so-called “30% rule” people cite for AI workloads, and does GPU utilization map to any such heuristic in a meaningful way?
The “30% rule” is the habit of citing a fixed utilization threshold — above 30%, or 50%, or some other figure — as if a single number could certify a workload as healthy or wasteful. It does not map to anything meaningful, because utilization is context-dependent: the same model on the same hardware lands at very different numbers depending on batch size, framework orchestration, and input pipeline shape. The metric measures one subsystem’s occupancy, not whether the system is doing the right work efficiently, so no portable threshold can diagnose performance.
