Watch the GPU utilisation gauge climb to 95%, call the model “efficient,” and move on. That single reflex is responsible for more wasted optimisation cycles than any other habit we see in inference teams. A utilisation percentage tells you the device is busy. It does not tell you whether it is busy doing useful arithmetic, waiting on memory, or stalling on data that hasn’t arrived from the host yet. The core problem is that “ML model metrics” gets treated as one flat dashboard — accuracy, latency, throughput, GPU utilisation — with every number read as equally trustworthy for the same purpose. It isn’t. Some of those metrics describe whether the model is correct. Others describe whether the serving path is fast. And within the performance metrics, a single number like utilisation can point in three different directions depending on what the kernel is actually blocked on. Confusing these categories is how teams spend weeks chasing a figure that was never going to move. Two families of metrics that answer different questions Before any bottleneck discussion, separate the metrics by the question they answer. This is the split that most dashboards blur. Model-quality metrics — accuracy, precision, recall, F1, mAP, perplexity — measure whether the model produces correct outputs. They are properties of the trained weights and the data distribution. They do not change when you swap an A100 for an H100, quantise to FP8, or add a second serving replica. If accuracy drops after a deployment change, you have a correctness regression (often a preprocessing or dtype mismatch), not a performance problem. Serving-performance metrics — latency (p50/p95/p99), throughput (requests or tokens per second), GPU utilisation, memory-bandwidth utilisation, and host-to-device transfer time — measure how efficiently the deployment turns hardware into answers. These do move when you change hardware, batch size, precision, or the serving stack. Keeping these families separate matters because they fail independently and get fixed differently. A model with excellent accuracy can have terrible p99 latency. A model that’s blazingly fast can be quietly wrong. We treat this quality-versus-serving split as the first thing to establish on any audit, and it’s the same discipline we apply when choosing what to measure across multi-platform edge deployments, where the hardware varies but the metric taxonomy shouldn’t. Why can high GPU utilisation appear alongside low end-to-end throughput? This is the contradiction that trips up most teams, so it’s worth being precise about the mechanism. “GPU utilisation” as reported by nvidia-smi — the field NVIDIA documents as the percentage of time one or more kernels were executing on the device — measures occupancy over time, not arithmetic efficiency. A kernel that spends most of its cycles waiting on data from HBM (high-bandwidth memory) still counts as “the GPU is busy.” The device is occupied; it just isn’t computing. So a memory-bound kernel can pin nvidia-smi utilisation near 100% while the tensor cores sit mostly idle, and end-to-end throughput stays flat. The utilisation number is doing exactly what it’s designed to do — it’s just not the number that answers “why is this slow?” This is a benchmark-class distinction that shows up in any profiler trace: NVIDIA’s Nsight Compute reports separate “SM throughput” and “memory throughput” figures precisely because they diverge. The practical consequence: a utilisation percentage is a symptom that still needs profiling to interpret, not a bottleneck diagnosis on its own. If you optimise against utilisation directly — throwing bigger batches at a kernel that’s already saturating memory bandwidth — you’ll watch the number stay stubbornly high while latency gets worse. We see this pattern regularly, and the resolution is almost never “add more work”; it’s a profiler trace that names the real constraint. The same reasoning underlies applying the three pillars of observability to GPU utilisation, where the metric is treated as one signal among several rather than the verdict. Which metrics actually point toward a compute-, memory-, or host-bound bottleneck? Bottlenecks fall into three broad classes, and each leaves a different fingerprint across the metrics. The table below maps the observable signal to the likely constraint and the profiler evidence that confirms it. Treat the “likely constraint” column as a hypothesis to verify, not a conclusion. Metric-to-bottleneck diagnostic table Observable signal Likely constraint Confirm with High nvidia-smi util, low SM/tensor-core throughput, memory throughput near peak Memory-bound — kernel stalls on HBM reads/writes Nsight Compute memory-vs-compute roofline; look for a memory-bound point High SM throughput, tensor cores near peak, util high Compute-bound — genuinely arithmetic-limited Nsight Compute achieved FLOP rate vs device peak Low GPU util, gaps between kernel launches in the timeline Host-bound — CPU preprocessing, Python overhead, or data loading stalls the pipeline Nsight Systems timeline showing idle GPU between kernels Low util, large memcpy bars on the timeline Transfer-bound — PCIe host-to-device copies dominate Nsight Systems H2D/D2H transfer durations; check pinned memory Good throughput at batch=1, collapses under concurrency Serving/scheduling-bound — queueing or batching policy, not the kernel Serving-framework request-queue metrics + latency percentiles The value of this mapping is that it turns a single ambiguous number into a small set of testable hypotheses. Low utilisation, for instance, is not automatically a problem — a host-bound pipeline shows low GPU utilisation precisely because the GPU is starved, and no amount of kernel optimisation fixes that. The fix lives on the CPU side: better data loading, pinned memory, or overlapping transfers with compute using CUDA streams. Reading low utilisation as “the model is underperforming and needs a faster kernel” sends teams down exactly the wrong path. How do serving-framework metrics relate to real kernel behaviour? Modern LLM serving stacks — vLLM, TensorRT-LLM, SGLang — expose their own utilisation and throughput metrics, and these operate at a different layer than nvidia-smi. When vLLM reports “GPU KV-cache utilisation,” it’s telling you how full the paged-attention KV-cache is, which governs how many concurrent sequences it can batch. That’s a scheduling-and-memory-capacity metric, not a measure of arithmetic efficiency. A vLLM instance can report high cache utilisation and high request throughput while individual decode kernels remain memory-bandwidth-bound — the two facts are compatible and describe different levels of the stack. The reconciliation is hierarchical: serving-framework metrics tell you whether the scheduler is keeping the device fed and batched; kernel-level profiler traces tell you whether each kernel is spending its time well. You need both. A framework metric that looks healthy can sit on top of kernels that are leaving throughput on the table, and the only way to see that gap is to drop below the framework into a profiler. Teams working on reading DGX Spark benchmarks in terms of utilisation rather than peak FLOPs hit the same layering question — the headline number and the kernel reality are related but not interchangeable, and memory bandwidth is frequently the constraint the headline hides. A worked example: the flat-utilisation trap Assume a team is serving a mid-sized transformer for a batch-inference workload. They observe: nvidia-smi utilisation: ~92%, steady Throughput: ~180 requests/sec, not improving despite two rounds of “optimisation” Each optimisation attempt added larger batches; latency got worse, utilisation stayed at ~92% The naive read: “92% utilisation means we’re nearly maxed out; there’s little headroom.” Under that read, the flat throughput looks like a hardware ceiling and the next step is “buy a bigger GPU.” A profiler trace tells a different story. Nsight Compute shows memory throughput near device peak while achieved tensor-core FLOP rate sits well below peak — a classic memory-bound signature. The batches were making it worse because they increased memory traffic without adding arithmetic the tensor cores could hide behind. The real levers here are ones that reduce memory movement per output: operator fusion, quantisation to a lower-precision format so each weight fetch moves fewer bytes, or a KV-cache strategy that reuses computation. None of those are suggested by the utilisation number; all of them are suggested by the roofline point in the trace. The measurable outcome of getting this right is the same one that motivates any performance audit: real speedup when you finally target the true bottleneck instead of the assumed one. Chasing utilisation would have bought a bigger GPU and reproduced the same memory-bound plateau on more expensive hardware. This is why we anchor bottleneck work to profiler evidence rather than dashboard readings — a discipline the broader GPU performance and cost work is built around. When do metrics stop being enough? Metrics are a triage layer. They tell you that something is off and roughly where to look, but the moment two plausible explanations produce the same dashboard signal — and they routinely do — you’ve reached the limit of what a metric can decide. High utilisation with flat throughput is consistent with both memory-boundedness and a scheduling problem. Low utilisation is consistent with host-boundedness, transfer-boundedness, or a serving-config issue. A number cannot disambiguate these; a trace can. The threshold is simple to state: when a single metric is compatible with more than one bottleneck class, stop optimising and start profiling. Nsight Systems for the pipeline-level view (where is the GPU idle, how big are the transfers) and Nsight Compute for the kernel-level view (is this kernel compute- or memory-bound) are the tools that turn a symptom into a diagnosis. Everything upstream of that — the dashboards, the framework metrics, the utilisation gauges — is there to tell you which trace to take, not to replace it. FAQ How does ml model metrics work in practice? “ML model metrics” isn’t one measurement but two families that answer different questions: model-quality metrics (accuracy, precision, recall, mAP, perplexity) measure whether outputs are correct, and serving-performance metrics (latency, throughput, GPU/memory utilisation, transfer time) measure how efficiently the deployment runs. In practice, you read them separately because they fail independently and get fixed differently — a correctness regression and a latency regression have nothing to do with each other. Which ML model metrics measure model quality versus serving performance, and why keep them separate? Quality metrics are properties of the trained weights and data distribution; they don’t move when you change hardware, precision, or replica count. Serving metrics do move under those changes. Keeping them separate prevents category errors — for example, reading a latency problem as a model problem, or assuming that faster serving implies better accuracy. Each family points at a different set of fixes. Why can high GPU utilisation appear alongside low end-to-end throughput? Because nvidia-smi utilisation measures how much of the time a kernel was executing, not how efficiently it computed. A memory-bound kernel that spends its cycles waiting on HBM still counts as “busy,” pinning utilisation near 100% while the tensor cores sit mostly idle and throughput stays flat. Utilisation is a symptom that needs a profiler trace to interpret, not a bottleneck diagnosis on its own. Which metrics actually point toward a compute-, memory-, or host-bound bottleneck? The fingerprints differ: high utilisation with near-peak memory throughput but low SM throughput signals memory-bound; near-peak tensor-core throughput signals compute-bound; low utilisation with gaps between kernel launches signals host-bound; and low utilisation with large memcpy bars signals transfer-bound. Each is a hypothesis you confirm with Nsight Compute (kernel-level) or Nsight Systems (pipeline-level), not a conclusion from the dashboard. How do serving-framework metrics (e.g. vLLM utilisation) relate to real kernel-level behaviour? They operate at a higher layer. vLLM’s KV-cache utilisation describes scheduler and memory-capacity state — how many sequences it can batch — not arithmetic efficiency. A framework can report healthy throughput and cache use while individual decode kernels remain memory-bandwidth-bound. The two are compatible; you need framework metrics for scheduling health and kernel traces for per-kernel efficiency. When do metrics stop being enough, and profiling becomes necessary to find the real constraint? The moment a single metric is compatible with more than one bottleneck class. High utilisation with flat throughput fits both memory-boundedness and scheduling problems; low utilisation fits host-, transfer-, and config-bound cases. When a number can’t disambiguate the plausible causes, stop optimising against the number and take a profiler trace — Nsight Systems for the pipeline view, Nsight Compute for the kernel view. Metrics narrow the search; they rarely close it. The teams that move fastest are the ones that treat a utilisation gauge as the question “which trace should I take next?” rather than the answer — and that reflex is the difference between a GPU performance audit that ranks real bottlenecks and one that ratifies the assumed one.
