A dashboard says the GPU is 92% busy. Procurement reads that as “we’re out of headroom, buy more cards.” The trouble is that a busy-percentage gauge is one of the least informative numbers you can put in front of a decision. That single figure tells you a kernel was resident on the device — not that memory bandwidth was saturated, not that the compute cores were doing useful work, and certainly not that you’re getting the FLOP you paid for.

The three pillars of observability — metrics, logs, and traces — were formalised for distributed software systems, but they map cleanly onto GPU workloads and they answer a question the busy gauge cannot: where does the time and the capacity actually go? Applied properly, the three pillars turn “the GPU is busy” into a measured utilisation gap — the difference between effective FLOP and purchased FLOP, the fraction of memory bandwidth genuinely occupied, and the idle time bleeding out during data transfer. That gap is usually where the money is, and it is invisible to a monitoring dashboard.

How does the 3 pillars of observability model work for GPU workloads?

In its original framing, observability rests on three complementary telemetry types. Metrics are numeric time series — cheap to store, good for quantifying and alerting. Logs are discrete, timestamped events — good for reproducing a specific run. Traces follow a single request or unit of work across stages — good for attribution, showing where latency accumulates. Each answers a different question, and the model’s whole point is that no single pillar is sufficient.

On a GPU workload the mapping is direct. Device counters — SM occupancy, memory-bandwidth utilisation, PCIe throughput, tensor-core active cycles — are your metrics. Per-run records of batch size, model version, input shape, and the CUDA/driver stack in play are your logs. And a per-stage execution trace — host preprocessing, host-to-device copy, kernel launch, compute, device-to-host copy, postprocessing — is your trace pillar. Tooling here is mature: NVIDIA’s Nsight Systems and the CUPTI-backed profilers produce exactly these per-stage timelines, and DCGM exposes the device counters as a metrics stream you can scrape into Prometheus.

The reason all three matter together is a chain of dependency. Without the trace pillar you cannot attribute waste to a cause. Without metrics you cannot quantify how much waste there is. Without logs you cannot reproduce the run that produced it. A monitoring setup that has only the metrics pillar — which describes most teams we talk to — can tell you that something is off but never why.

Why is the GPU-busy metric misleading?

The busy-percentage number most dashboards surface is derived from whether any kernel was executing on the device during a sampling window. It says nothing about how well that kernel used the silicon underneath it.

Consider a common pattern in inference serving. A model runs at batch size 1 because requests arrive one at a time and nobody wired up dynamic batching. The kernel launches, occupies the device for a few milliseconds, the busy meter reads high — and yet SM occupancy sits at a fraction of capacity because the workload is too small to fill the streaming multiprocessors. This is the divergence point: a busy gauge hides the utilisation gap, and only the metrics pillar (SM occupancy, tensor-core active cycles) exposes it as a number.

The other classic failure is data-feed starvation. The compute cores stall waiting on a host-to-device copy over PCIe while the busy meter — sampling at coarse granularity — still reports high utilisation because the surrounding window contained some kernel activity. The idle compute during I/O is precisely what a trace exposes and a metric aggregate hides. When you look at an Nsight timeline and see a wide Memcpy HtoD band with the compute stream idle beside it, you are looking at purchased FLOP you never used. We wrote more about how batch size specifically reshapes this in what batching really changes for YOLO inference on GPU, and about which counters actually explain bottlenecks versus which mislead in ML model metrics that explain GPU bottlenecks.

What each pillar reveals about GPU utilisation

The value of the model is that the three pillars are not interchangeable. Each answers a question the others cannot, and a real audit uses all three in sequence.

Pillar	Signal on a GPU workload	The question it answers	What it cannot do alone
Metrics	SM occupancy, memory-bandwidth utilisation, tensor-core active cycles, PCIe throughput, power draw	How much capacity is used, and is that changing over time?	Cannot tell you which stage caused a stall — it aggregates across the pipeline
Logs	Per-run batch size, input shape, model version, driver/CUDA version, queue depth	Which run, under which conditions, produced this behaviour — reproducibility	Cannot quantify the gap or localise it within a run
Traces	Per-stage timeline: host prep → HtoD copy → kernel launch → compute → DtoH copy → postprocess	Where the time and idle capacity go — attribution to a specific stage	Cannot tell you if this is representative or a one-off without metrics + logs

Read the table left to right and you have the audit method. Metrics quantify the utilisation gap. Traces attribute it to a stage — small batch, HtoD stall, a serialised pipeline where preprocessing on the host blocks the next kernel. Logs let you reproduce the exact run so the fix can be verified against the same conditions. Miss any one and the investigation stalls at a different point: you know there’s waste but not where (no traces), you found a slow stage but can’t reproduce it (no logs), or you have a suggestive timeline but no idea whether it’s typical (no metrics).

How do I use traces to attribute wasted capacity to a cause?

Start from the per-stage trace, because attribution is the pillar most teams are missing. A Nsight Systems timeline (or a torch.profiler export viewed in the same tool) lays out each CUDA stream against wall-clock time. You are looking for three shapes.

First, wide copy bands with idle compute beside them. If the Memcpy HtoD region is comparable to or larger than the compute region, the workload is I/O-bound at the PCIe boundary, not compute-bound. The fix is usually pinned memory, overlapping copy with compute via separate CUDA streams, or moving preprocessing onto the device — not a bigger GPU. This is the same class of movement cost we unpacked in what “computationally expensive” actually means and where the cost lives.

Second, short compute kernels with low occupancy. If each kernel launch occupies the device briefly and the metrics pillar shows SM occupancy well under capacity, the batch is too small. Dynamic batching or request coalescing closes this gap without new hardware.

Third, serialised stages that should overlap. If host postprocessing sits on the critical path between two kernels and the device idles through it, the pipeline is serialised where it could be pipelined. Overlapping host and device work with asynchronous execution recovers that idle time.

Each of these is an observed pattern across the GPU workloads we profile in practice, not a benchmarked rate — the exact recovery depends on your model, your batch distribution, and your PCIe topology. But the diagnostic shape is consistent: the trace tells you which of the three it is, and that determines whether the answer is a code change or a procurement request.

Which counters quantify effective FLOP versus purchased FLOP?

Once a trace has localised the problem, the metrics pillar quantifies it. The counters worth collecting, exposed through DCGM or CUPTI, fall into three groups.

For compute utilisation: tensor-core active cycles and SM occupancy. The ratio of achieved throughput to the device’s published peak FLOP is your effective-vs-purchased figure — per NVIDIA’s published specifications, an H100 SXM lists on the order of 989 TFLOP/s of dense FP16 tensor throughput, and if your measured effective throughput is a fraction of that, the difference is the utilisation gap in the metric you paid for.

For memory: memory-bandwidth utilisation and HBM occupancy. A workload can be compute-idle yet bandwidth-bound; the busy gauge cannot distinguish these, but the bandwidth counter can.

For the boundary: PCIe throughput and host-to-device copy time. When this is high relative to compute, you have confirmed the I/O-bound diagnosis the trace suggested.

None of these is a single headline number, and that is the point — effective FLOP is a derived measurement, computed from the counters against the published peak, not something a dashboard hands you. Teams building this out systematically usually reach for a stack that correlates device counters with cost; we cover the tooling side in ML observability tools that turn GPU utilisation data into cloud decisions and the ongoing signal-selection question in what to track for GPU inference workloads.

FAQ

How does the 3 pillars of observability model work, and what does it mean in practice for GPU workloads?

The model rests on three complementary telemetry types: metrics (numeric time series), logs (timestamped events), and traces (per-request execution paths). On a GPU workload, metrics are device counters like SM occupancy and memory bandwidth, logs are per-run records of batch size and model version, and traces are per-stage timelines from host preprocessing through the copy-compute-copy cycle. The point of the model is that no single pillar is sufficient — you need all three to quantify, reproduce, and attribute waste.

What do metrics, logs, and traces each reveal about GPU utilisation that a single busy-percentage gauge cannot?

Metrics quantify how much capacity is actually used — SM occupancy and bandwidth occupancy, not just whether a kernel was resident. Traces reveal where time and idle capacity go across the pipeline stages, exposing I/O stalls the busy gauge averages over. Logs capture the conditions of a specific run so the behaviour can be reproduced and a fix verified. The busy gauge collapses all three into one number that only tells you a kernel existed.

Why is the GPU-busy metric misleading, and which pillar exposes idle compute during data transfer?

The busy metric is derived from whether any kernel was executing in a sampling window, so it reads high even when compute cores stall waiting on a host-to-device copy or when a small batch leaves most streaming multiprocessors idle. It hides the utilisation gap because it does not measure how well the silicon was used. The trace pillar exposes idle compute during data transfer: a per-stage timeline shows a wide copy band with the compute stream idle beside it.

How do I use traces to attribute wasted GPU capacity to a specific cause like small batch sizes or host-to-device stalls?

Read a per-stage timeline (from Nsight Systems or torch.profiler) and look for three shapes: wide copy bands with idle compute beside them signal a PCIe-bound stall; short low-occupancy kernels signal batches too small to fill the device; and serialised host stages on the critical path signal a pipeline that should overlap but does not. Each shape points to a distinct fix — pinned memory and stream overlap, dynamic batching, or asynchronous execution — rather than more hardware.

Which device counters and metrics should I collect to quantify effective FLOP versus purchased FLOP?

Collect compute counters (tensor-core active cycles, SM occupancy), memory counters (bandwidth utilisation, HBM occupancy), and boundary counters (PCIe throughput, host-to-device copy time), typically through DCGM or CUPTI. Effective FLOP is derived by comparing achieved throughput against the device’s published peak — for example, an H100 SXM’s roughly 989 TFLOP/s dense FP16 tensor figure per NVIDIA’s specifications. The shortfall between measured and peak is the utilisation gap.

How does workload-level observability feed into a GPU Performance Audit and the decision to optimise before procuring more capacity?

Workload-level observability is the instrumentation a GPU Performance Audit relies on: metrics quantify the utilisation gap, traces attribute it to a cause, and logs make the run reproducible so a fix can be verified. Together they distinguish “we’re out of headroom, buy more” from “we’re wasting the cards we have.” That distinction is what lets a team recover cost by optimising existing utilisation instead of expanding the fleet.

Where this leaves the procurement decision

The reason to instrument all three pillars before the next hardware cycle is that they change the question being asked. A busy dashboard leads to “we’re at capacity, buy more.” A workload-level view — effective FLOP against purchased FLOP, bandwidth occupancy, idle time during transfer — leads to “we’re using a fraction of what we bought, and here is the stage that’s wasting it.” Those are very different conversations to have with finance.

If you already have monitoring but no per-stage traces, you have the metrics pillar and none of the attribution. That is the gap a GPU Performance Audit is built to close: correlating device counters with per-stage execution to locate where capacity is wasted, so the decision to optimise or procure rests on a measured utilisation gap rather than a percentage that only ever meant a kernel was running.