Low GPU usage in an AI job often means the GPU is waiting for something. It might be the next batch from the CPU, a stream of tiny kernel launches, or memory that cannot feed the arithmetic units fast enough. NVIDIA’s NVML defines GPU utilisation as the “percent of time over the past sample period during which one or more kernels was executing on the GPU”. It does not show how many streaming multiprocessors (SMs) were busy or how much maths they did. A reading of 100% does not mean the chip is saturated.

TechnoLynx, the engineering team behind LynxBenchAI, takes on the work that comes next: finding the constraint that actually limits throughput in a running training or inference job and fixing it, as part of its GPU performance engineering work.

The GPU-Util column counts time, not work

It is natural to read GPU-Util as a fuel gauge. The trouble is what the GPU-Util counter measures and misses. Trainy points out that NVIDIA’s own documentation describes utilisation only vaguely, and that Datadog’s NVML docs give a better, kernel-executing definition. Under that definition you can reach 100% GPU utilisation by only reading and writing memory while doing no floating-point operations at all.

The SM count makes the gap concrete. An NVIDIA H100 has 132 SMs, so a kernel that runs for ten seconds on one SM registers 100% utilisation and an SM efficiency of 1/132, about 0.7%. Trainy describes an LLM training job that sat at 100% GPU utilisation while using only about 20% of the GPU’s theoretical maximum. Arthur Chiao’s write-up makes the same point from the other side: a single small task occupying a fraction of the device can make nvidia-smi report the GPU as fully occupied. Utilisation in the USE sense is busy time. Saturation is the degree to which the resource has extra work it cannot service, and NVML measures only time in use.

The broader argument that low numbers can be the right outcome is in why GPU utilisation is not performance.

Counters that show how much of the GPU the work uses

NVIDIA’s Data Center GPU Manager (DCGM) exposes profiling counters that get closer to saturation than NVML’s utilisation field does. Each one answers a narrower question:

Counter	What it records	What it cannot tell you
nvidia-smi / NVML GPU utilisation	Share of the sample period with at least one kernel running	How many SMs were busy or how much maths ran
SM activity (DCGM_FI_PROF_SM_ACTIVE)	Fraction of time at least one warp was active on an SM, averaged over all SMs	Whether active warps were computing: warps waiting on memory also count as active
SM occupancy (DCGM_FI_PROF_SM_OCCUPANCY)	Resident warps relative to the maximum an SM supports	Efficiency on compute-limited work, where higher occupancy does not necessarily mean better use
Tensor activity (DCGM_FI_PROF_PIPE_TENSOR_ACTIVE)	Activity of the Tensor Core pipeline	Which kernel or source line produced it
Memory bandwidth utilisation	Fraction of cycles with traffic to or from device memory; about 80% is the practical maximum	On its own, whether the job is memory-bound

DCGM gives rough thresholds for SM activity: 0.8 or higher is necessary but not sufficient for effective use, and below 0.5 probably means the GPU is being used ineffectively. The DCGM profiling metrics are averages over an interval, not a kernel trace. NVIDIA’s guidance is to use DCGM to find an underused device and then move to Nsight Systems or Nsight Compute when you need to pin the problem on specific code. NVIDIA also warns that one high value does not alone prove a job is compute-bound or memory-bound. Several related counters have to be compared over a representative interval and checked against application throughput.

Why is my GPU usage so low when nothing else looks maxed out?

One common answer to why is my GPU usage so low is that the GPU is starved of input. Mohan et al. (VLDB 2021) split these data stalls into fetch stalls, where training waits on storage I/O, and prep stalls, where it waits on CPU pre-processing. Either kind leaves the GPU idle. Stalls appear whenever the GPU processes data faster than it can be fetched or pre-processed, that is, when G > min(F, P). In their study, data stalls accounted for up to 65% of training time. Faster GPUs did not help: the stalls threw away the extra speed even on DGX-2 servers.

That study also shows why raising the batch size is not always the fix. With larger batches, training became CPU-bound on data preparation, and epoch time stayed the same even though the GPU compute time per epoch fell. Pre-processing rate can be measured on its own: cache the data, give the loader every CPU core, and disable GPU computation so that only the loader runs. In a profiler, read the CPU and GPU timelines side by side. Gaps between kernels point to data, launch or synchronisation stalls. In PyTorch, setting num_workers above zero prepares batches in parallel while the GPU works, and pin_memory=True allows asynchronous host-to-device copies.

Other causes leave the GPU busy but under-fed:

• Batch too small. There is not enough work to fill the CUDA cores, so the device is under-used while technically active.
• Launch overhead. When kernels are tiny, launching them can take longer than running them, which makes the job CPU-bound on the launch side.
• Host-to-device bandwidth. A PCIe Gen3 link or a reduced lane count can leave the GPU waiting on transfers.
• Memory-bound inference. Decode can be limited by memory bandwidth while kernels keep running, which the decode section below covers.

Adding GPU power without fixing the upstream bottleneck does not make training faster. The cost side of the same observation, idle accelerator hours as a budget line, is covered in the hidden cost of GPU underutilisation.

How do I calculate MFU for my GPU?

Model FLOPs utilisation (MFU) is the ratio of observed throughput to the theoretical maximum throughput the hardware would reach at peak FLOPS with no memory or communication overhead. Chowdhery et al. proposed it in the PaLM paper (2022) as an implementation-independent replacement for hardware FLOPs utilisation (HFU). HFU depends on how the system and compiler happen to execute the model, including rematerialisation.

In practice:

1. Measure sustained throughput in tokens per second.
2. Multiply by the model FLOPs per token. For dense transformer training this is about 6 × parameters (forward plus backward); for an inference forward pass it is about 2 × parameters.
3. Divide by the peak FLOP/s of the device at the precision you actually run.

The FLOPs count is where it goes wrong. A study of 608 production jobs using a hardware-counter metric (Overall FLOP Utilisation, OFU, built from Tensor Pipe Activity and SM clock) found a 288-GPU mixture-of-experts job reporting 54.27% MFU. Hardware counters put it at 25.58%, because the framework’s FLOPs counter missed the model’s latent-space projections. In another run, activation checkpointing raised the actual work per step from about 3F to 4F FLOPs, and correcting the formula moved reported MFU from 26% to 33%. Under PaLM’s definition, MFU is implementation-independent while HFU depends on choices such as rematerialisation, so decide which you report before comparing numbers.

Hardware counters make a useful cross-check. Across the 608 jobs, MFU and OFU correlated at r = 0.53. In a separate 174-job comparison, averaging per job raised the correlation to r = 0.977. The same authors call OFU a coarse signal: it shows that a problem exists, but finding the cause still takes profiling.

What counts as a good MFU for LLM training and inference?

For LLM training, Trainy puts most runs at around 35–45% MFU, and the fleet-scale OFU study treats 35–50% as healthy. The two ranges overlap closely. We would treat the high 30s as respectable and anything well below it as a reason to profile. For historical reference, the PaLM paper lists GPT-3 at 21.3%, Gopher at 32.5% and Megatron-Turing NLG 530B at 30.2% with self-attention counted.

Inference has no single good figure, because prefill and decode have different ceilings. Pope et al. (2022) report MFU from 14% to 76% across their configurations, with 76% reached by weight-gathered layouts at large batch, where communication overhead is minimal. ZeroEntropy puts decode at batch 32 on an H100 at 8–12% MFU and attributes the cap to memory bandwidth rather than poor tuning. The useful judgement is the gap between your number and the ceiling for your workload’s shape, not the gap to 100%.

High MFU is not always the goal. Smaller batches give lower latency and worse MFU, at a higher cost per token. A latency-bound service can be correctly configured with an MFU that would look poor on a training job.

Why is MFU usually well below 100%, and where does the gap go?

Ivanov et al. (MLSys 2021) found that data movement is the main bottleneck in transformer training. Compute has improved so much that, by Amdahl’s Law, training has become memory-bound. By optimising data layouts they cut data movement by up to 22.91% and reached 1.30x on a BERT encoder layer. At kernel level, Trainy saw a softmax kernel with high utilisation and low SM efficiency. Naive softmax is memory-bound, which is the problem fused kernels such as FlashAttention were designed to solve.

On multi-GPU runs, communication also costs MFU. Databricks reports poor use of network bandwidth in AllGather collectives, because ring latency grows linearly with GPU count. ZeroEntropy attributes a multi-node training run at 18% to communication overhead from poorly tuned tensor or pipeline parallelism.

Then there is the matrix shape. GEMM kernels pad matrices to tile sizes. For well-aligned matrices of 4096 or more, the mean overhead is 2–3%, but below 512 it can exceed 50%. Launch overhead and the data stalls described above also leave the GPU short of work. The MFU number only tells you that a diagnosis is needed. For which metrics actually point at a cause, see ML model metrics that explain GPU bottlenecks versus those that mislead.

Why is MFU so low during LLM inference decode?

Decode is memory-bound by construction. Weights and the KV cache have to move from high-bandwidth memory to the compute cores once per forward pass. At small batch sizes the weight load dominates, and at large batch and long context the KV cache load dominates. Pope et al. give a 500B+ model at batch 512 and context 2048 whose KV cache totals 3 TB, which must be reloaded for every token while the compute core sits essentially idle. Databricks makes the same point for smaller models: at low batch, generation speed depends on how fast parameters move from GPU memory, so achieved memory bandwidth predicts token speed better than peak compute does.

That is why MFU is not the most relevant measure for low-batch decode. Databricks proposes model bandwidth utilisation (MBU): achieved memory bandwidth, meaning (parameter size + KV cache size) / time per output token, divided by peak bandwidth. MBU is the relevant measure when decode is memory-bound at small batch, and MFU becomes relevant once large batches make the system compute-bound. ZeroEntropy argues that decode MFU around 10% is a ceiling, not a failure to optimise. You raise it by changing the workload’s bytes-per-FLOP ratio, not by squeezing harder. Quantising weights from FP16 to INT8 or INT4 is one way to do that, because less data has to move per token.

Why did MFU drop when I switched to FP8 or a newer GPU?

The denominator moved. Databricks found MFU lower with FP8 than with BF16 on H100, because FP8’s theoretical peak is double BF16’s even though FP8 performs more operations per second. The arithmetic follows from the definition: if peak doubles and throughput rises by less than 2x, MFU falls while the run gets faster.

A newer GPU can do the same thing. Dense FP16 Tensor Core throughput went from 125 TFLOP/s on V100 to 989 TFLOP/s on H100, a denominator about 7.9 times larger (989 ÷ 125). If throughput on the new card rises by less than that factor, MFU comes out lower even when every step finishes sooner. A metric built from Tensor Core counters, such as OFU, is precision-agnostic for exactly this reason. When comparing across precisions or hardware generations, we would put wall-clock throughput first and MFU second. Throughput is the outcome the user sees, and MFU is a ratio whose denominator just changed.

How teams raise MFU in practice: batch, kernels, parallelism, precision

The first step is measurement: DCGM to find the underused device, then Nsight Compute to see whether the hot kernels are memory-bound or compute-bound. How to read that output is the subject of profiling AI inference and what the numbers mean, and which profiling tools measure what covers the tooling.

Larger matrix multiplications run more efficiently on accelerators, which is why fusing operations and growing tokens per GPU both raise utilisation. Batching is essential for inference throughput, and continuous batching is indispensable for shared online services. Batching has a limit, though: once the system is compute-bound, each doubling of batch size adds latency without adding throughput. Fused kernels such as FlashAttention target memory-bound steps such as softmax, and ZeroEntropy reads a 70B FP16 prefill at 25% MFU as a likely sign that FlashAttention or kernel fusion is missing. For parallelism, Databricks shards within smaller blocks of the cluster so that each collective involves only a subset of GPUs. Precision cuts both ways: lower formats raise throughput while possibly lowering reported MFU, as the FP8 case shows.

Measuring what the machine holds, in LynxBenchAI

Neither the utilisation counter nor the datasheet measures what the device sustains under load. LynxBenchAI holds that specification metrics such as AI TOPS, peak FLOPS or memory bandwidth cannot predict real AI performance, because memory locality, data reuse, execution strategy and software behaviour depend on the workload and cannot be read off a datasheet.

LynxBenchAI runs the same model catalogue at the same precisions on every device in a release and changes only the batch size. It raises the batch until throughput stops improving within a defined noise band, so a laptop GPU and a data-centre accelerator are each measured at their own saturation point. It then discards a warm-up phase and counts completed iterations inside a continuous, declared timed window. The result is what the machine sustains under load, not its best moment. The run does not claim to capture a thermal state or a settled clock regime it does not observe.

The published results include one card measured on two platforms. In the 26Q3 results, the NVIDIA GeForce RTX 5090 on native Linux scores 985.8 for inference, while the same card under WSL scores 891.8. The two entries share a device name and differ in platform. Scores are comparable within the 26Q3 release only, and the full set is in the AI hardware directory of LynxBenchAI results.

To settle the question on the machine in front of you, run pip install lynxbench-ai on Python 3.11 or later, on Linux or Windows via WSL2. A run needs about 15 GB of free disk, downloads about 2 GB of model data once, and takes 15–30 minutes. The Personal Edition is released and free for non-commercial use. The Press, Pro and Enterprise editions are not released. It tells you what the device can sustain. More on the method is on the LynxBenchAI page.

FAQ

Why does my GPU usage keep dropping to 0?

Periodic zeros can come from checkpoint I/O, input preparation, CPU-only evaluation work or synchronisation. Validation often runs on the GPU as well, so line up the profiler timeline with application phases before choosing a fix such as asynchronous checkpointing.

What does it mean when a CPU is bottlenecking the GPU?

It means the CPU cannot prepare data as fast as the GPU consumes it, so the GPU sits idle between batches. A faster GPU does not help until the upstream stage speeds up, because training never runs faster than its slowest component.

Is an allocated GPU the same as a busy one?

No. Allocation means the GPU is reserved for a user or job, while utilisation means its compute units are doing work. A job can hold a GPU at full allocation while it waits on data, a network synchronisation step or an unused notebook session.