The GPU-Util figure in nvidia-smi is the percentage of time over the past sample period during which one or more kernels was executing on the GPU. Depending on the product, that sample period is between 1/6 of a second and 1 second. That makes nvidia-smi GPU utilization a record of whether the GPU was busy during a sampling window. It does not tell you how much of the chip’s compute was in use. The figure gives no indication of whether the running kernel used all the cores available.

LynxBenchAI reports a score bounded by declared conditions and one release’s fixed model catalogue, not by your own application.

How nvidia-smi GPU utilization is sampled

The definition is narrow, so the reading is narrow too. Within each sample period, the percentage is the share of time during which one or more kernels was executing. A CUDA kernel that only reads and writes memory, doing zero FLOPS, still counts as executing. So does a kernel that parallelises the workload poorly.

Two practical consequences follow from the sampling. A single long kernel reads as fully busy however few SMs it uses. And each reading covers only the past sample period, not the whole run.

Why can nvidia-smi show 100% while the GPU does far less work than it could?

nvidia-smi can show 100% while the GPU does little useful work because the metric only measures whether a kernel is executing. Trainy’s write-up on the metric points out that you can reach 100% GPU utilization by reading and writing memory while doing zero floating-point operations. Trainy also gives an example of a kernel that runs for 10 seconds on a single SM of an H100, which has 132 SMs. That kernel registers 100% utilization while its SM efficiency is 1/132, or 0.7%.

By their account, most LLM training runs reach about 35% to 45% MFU. MFU is the ratio of observed throughput to the theoretical maximum throughput of the system at peak FLOPs. GPU-Util alone gives no indication of that gap between busy and productive.

Busy does not mean efficient at SM level either. NVIDIA’s DCGM documentation defines SM activity as the fraction of time at least one warp was active on a multiprocessor, averaged over all multiprocessors. It counts warps waiting on memory requests as active. A high percentage of SM utilisation is a prerequisite for good performance, not evidence of it, because the kernels on those SMs may suffer significant stalls.

Energy measurements tell the same story from another direction. The Bench360 study (2025) found that energy per token and GPU utilisation were not strongly correlated across inference engines. vLLM was the most energy-efficient engine across the GPUs tested, even though its average utilisation was not always the highest.

Metrics to log beside GPU-Util when you test GPU utilization

GPU-Util alone gives an incomplete picture, so the useful set pairs it with other readings. The table below is what we would put next to GPU-Util on any AI workload.

Metric	What it reports	Where it comes from	What to watch for
GPU-Util (utilization.gpu)	Share of the sample period with at least one kernel executing	nvidia-smi / NVML	Says nothing about SM count, arithmetic or efficiency
Memory utilisation (utilization.memory)	Utilisation of the memory interface, reported alongside compute	nvidia-smi / NVML	Trainy calls NVIDIA’s definition of it vague
SM activity	Fraction of time at least one warp was active per SM, averaged over all SMs	DCGM profiling metrics	0.8 or above is necessary but not sufficient for effective use
Tensor activity	Fraction of cycles the tensor (HMMA/IMMA) pipe was active	DCGM profiling metrics	20% can mean few SMs fully busy or all SMs lightly busy
MFU	Observed throughput relative to theoretical peak FLOPs	Computed from tokens per second	Hard to monitor continuously or layer by layer
Energy per token	Power multiplied by generation time, per token	NVML power telemetry	Not strongly correlated with utilisation across the engines Bench360 compared
Throughput and latency (TPS, TTFT, TPOT, queue time)	Work completed and how long requests waited	Serving-side timing	Measured with timing hooks and the server thread pool

For the coarse counters, NVIDIA’s support guidance for host-side logging gives a query including utilization.gpu, utilization.memory and memory.used with --format=csv -l 5. Adding -f <filename> redirects the output to a file. The same guidance asks you to size the query granularity to the use you have in mind.

The finer metrics come with their own sampling caveats. Some DCGM metrics cannot be collected together on some GPUs, so DCGM multiplexes them by statistical sampling, and collecting at higher frequencies can return zeros while it regroups them. DCGM profiling also conflicts with Nsight Systems and Nsight Compute. NVIDIA recommends pausing DCGM collection while those tools are in use. NVIDIA’s post on measuring GPU occupancy says Nsight Systems samples SM Active at 10 kHz by default when that option is enabled, far finer than nvidia-smi’s 1/6 to 1 second period. Our wider treatment of ML model metrics that explain GPU bottlenecks versus metrics that mislead goes further into which of these to alert on.

What GPU utilisation should an AI job reach, and when does a low reading mean a bottleneck?

NVIDIA’s DCGM documentation says an SM activity of 0.8 or greater is necessary, but not sufficient, for effective use of the GPU. For LLM training, Trainy gives the 35% to 45% MFU of most runs for reference.

Diagnosing and closing utilisation gaps on a production system is engineering work, which TechnoLynx covers under GPU Performance Engineering.

A low GPU-Util reading means that for part of the sample period no kernel was executing. That follows directly from the definition.

A low GPU usage reading does not by itself mean a less efficient engine. As our article on GPU utilization not being performance argues, high utilisation tends to reassure people and low utilisation tends to worry them, and both reactions are often disconnected from what the system is doing. Bench360’s authors suggest that vLLM’s batching and scheduling keep the GPU active only as long as necessary.

vLLM’s gpu_memory_utilization is a memory budget, not a compute reading

vLLM’s --gpu-memory-utilization setting is the fraction of GPU memory the model executor may use, from 0 to 1, with a default of 0.92. It is a per-instance limit, so two vLLM instances sharing one GPU could each be set to 0.5. Unless you set the KV cache size explicitly, vLLM infers it from this fraction. The KV cache is what PagedAttention allocates in fixed-size blocks rather than as one contiguous buffer.

The setting is a memory fraction, not a compute measure. One consequence catches people out. We would not read nvidia-smi’s memory-used figure as a load indicator on a vLLM host.

How does request batching change observed GPU utilization compared to a single-stream benchmark?

Batching raises compute utilisation and throughput, which GPU-Util does not measure. Anyscale’s analysis of continuous batching describes LLM inference as memory-bound: throughput is largely determined by how large a batch fits in GPU memory, because loading data to the compute cores takes longer than computing on it. Batching loads the parameters once and reuses them across many sequences, which raises compute utilisation and throughput. A single-stream run, one request at a time, forgoes that reuse.

The form of batching matters. With static batching the GPU is underutilised whenever sequences finish at different lengths, because finished slots wait for the longest sequence. Iteration-level scheduling, introduced by Orca, decides batch size per iteration and inserts a new sequence as soon as one completes.

As concurrency rises the effective batch grows, and total tokens per second rises until it reaches a saturation point for the available GPU resources, beyond which it may fall. NVIDIA’s benchmarking guidance notes that throughput generally saturates around the engine’s maximum batch size, while latency keeps rising as excess requests queue. Anyscale observed that as a serving system saturates, there are fewer chances to inject new requests immediately, so request latency goes up.

Any sample period in which no kernel executes lowers the reading, so gaps between requests show up as dips. Tokens per second, read next to SM and tensor activity, track the batching effect far better than GPU-Util does.

How do concurrency, queuing and request variability change observed performance compared to a single-stream benchmark?

Latency constraints and random arrivals lower the throughput a system delivers, by an amount that depends on the system. The MLPerf Inference paper (Reddi et al.) defines four scenarios (single-stream, multistream, server and offline) because different applications have different figures of merit. In the server scenario, queries arrive at random and the system has to protect tail latency, so it may process inferences at a batch size below the one that fully uses the hardware. The same paper reports that the server scenario’s latency constraint cost some systems as little as 3% of their offline throughput and others as much as 50%.

That spread is the point. Two devices that look the same in a saturated, offline-style run can separate sharply once arrivals are random and latency is bounded. NVIDIA draws the same line between load testing, which simulates many concurrent requests to see how a deployment handles real traffic, and performance benchmarking, which measures the model’s own throughput and latency. Anyscale’s throughput comparison submitted every request in a burst so that compute was saturated, and it measured latency separately on a realistic workload.

The BurstGPT study found that concurrency patterns from non-LLM workloads misrepresent LLM serving performance. It also found that a scheduling optimisation effective for conversation services can lose efficiency for API services.

When is a synthetic benchmark still a useful proxy, and when does it stop?

A synthetic benchmark stays useful while your question is about the executor (the device plus its software stack) under identical, controlled conditions. It stops being a useful proxy once your question is about your service’s behaviour under your own arrival pattern, sequence lengths and latency target. Executing real software workloads is the reference standard. Estimating performance from specifications is the fallback for when execution is not possible. This is why even a fixed-catalogue run tells you more than a spec sheet, as our explainer on why spec-sheet benchmarking fails for AI argues.

We read the simplification of workload shape in synthetic benchmarks as structural, not careless. Reproducibility requires fixing exactly what varies in production: when requests arrive, how long they run and what else shares the GPU. A benchmark that let those vary would not give the same answer twice. The cost is real. BurstGPT’s authors report that LLM serving systems are often evaluated under unrealistic workload assumptions and may degrade when deployed, and that synthetic workloads do not reflect the patterns of auto-regressive serving.

We treat realism as a set of dimensions rather than a yes-or-no property. One run can match production on model architecture and numerical precision but not on arrival pattern. Another can replay real arrival traces on a model you will never deploy. “More realistic” only means something once you name the dimension that matters for your decision, which is why the useful question is “realistic on what?”. Task choice alone can reorder results. Dehghani et al. (2021) showed that the relative performance of ML methods can change significantly just by choosing different benchmark tasks.

What does benchmark saturation mean, and why does a suite everything scores well on stop being useful?

We use benchmark saturation in two senses, and they pull in opposite directions. By suite-level saturation we mean every serious contender scoring near the ceiling. We would then not trust the remaining differences to tell procurement which system to buy. Noise here is not trivial. Sinha et al. (SC22) measured an average performance variation of 8% across GPUs of identical architecture and SKU, with outliers up to 1.5 times slower than the median GPU.

At device level, saturation is a method: you drive the workload until throughput stops improving, so each executor is measured at its own ceiling rather than at an arbitrary load. LynxBenchAI uses the second meaning. GT is an unbounded aggregate with no reference-device normalisation, so no 0–100 ceiling exists for scores to bunch against. GT is an ordinal aggregate over the training, inference and compute scores. A GT difference is an ordering, not a percentage.

How LynxBenchAI handles saturation, warm-up and failed runs

For each test, LynxBenchAI discards a warm-up phase and then counts the iterations completed inside one continuous timed window. The workload size has already been raised until throughput stops improving, so what is reported is what the machine holds under load, not its best instant. Every device in a release runs the same model catalogue at the same precisions. Only the batch size adapts, and it rises until throughput plateaus within a defined noise band. The models are prepared once, before any target device is known, and the stricter of two vendors’ constraints is applied to all vendors. A workload that fails or times out is recorded as a zero rather than dropped.

In the 26Q3 results, the NVIDIA GeForce RTX 3080 on native Linux scores 240.5 for inference. Its fp8 inference entries for Flan-T5 Small, DistilGPT2, DistilBERT and GraphGPS (Peptides) are each 0.0, meaning that precision produced no result on that executor. A precision that produces no result is scored as zero rather than quietly skipped.

The timed window is declared, not certified as settled. LynxBenchAI asserts nothing about the machine’s thermal or clock condition inside the timed window. There is good reason for that restraint. Barrett et al. (OOPSLA 2017) found that only 30.0% to 43.5% of virtual machine and benchmark pairs consistently reached a steady state of peak performance, even in a tightly controlled environment. That study concerns language virtual machines rather than GPUs, but it shows that discarding warm-up does not by itself guarantee a steady state. MLPerf takes a different route, with a 60-second minimum run intended to measure the equilibrium behaviour of power management and DVFS. Our methodology states its window and its limits rather than claiming the equilibrium was reached.

How should I reason about a published score whose catalogue, precisions and window are fixed while my stack differs?

Read the published score as a measurement of executors on the release’s own models, at its own precisions and inside its declared window, and not as a forecast of your application. Each test is one timed window rather than a median of repeated trials. Given the variation between identical cards reported by Sinha et al., we would not build a decision on a small gap between two devices. The order we would work through it:

1. Find the category and model in the catalogue closest to your workload. A transformer encoder at fp16 inference tells you more about a BERT-style service than the aggregate does.
2. Check whether your production precision produced a result at all on the executors you are considering, since a 0.0 entry means it did not.
3. Compare executors only within the same release, and treat the result as an ordering under shared conditions.
4. Measure the part the score does not cover, meaning your own model, serving stack, arrival pattern and latency budget, on your own system.

Running the Personal Edition, which installs in 15–30 minutes, scores the release catalogue on your own machine. It still does not run your workload. The published device pages and the LynxBenchAI results give the comparison set. Our guides on profiling AI inference and what the numbers mean and on running a task-specific LLM evaluation that survives a procurement review cover building the measurement your own workload needs.

Can I compare GPU utilization figures between two different benchmark releases?

No, not as like for like. LynxBenchAI scores are comparable only within one release. 26Q3 is the only major release so far. Comparing a figure produced under one release name with one produced under another is a mismatch fatal to the conclusion. We would apply the same logic to raw utilisation figures collected under different configurations. Concurrency, maximum batch size, request rate and sampling settings all directly change measured throughput and latency, and nvidia-smi’s sample period itself differs by product. Our overview of measuring GPU benchmarks for AI sets out which conditions have to match before two numbers can sit in one table.

If you want to know whether a GPU suits a particular serving workload, read the published scores for the shortlist, check that your precision produces a result, and then measure your own request mix on your own stack.

FAQ

Why is my GPU utilization 0 percent?

A 0% GPU-Util reading means no kernel was executing on the GPU during the sample period. The process may be running on the CPU, waiting for input, or not using that GPU at all.

What does high GPU usage mean?

High GPU usage means a kernel was executing for a high share of the sample period. It does not show whether the kernel used most of the SMs or did much arithmetic, and a memory-only kernel can reach 100%.