Your GPUs are probably half idle An NVIDIA A100 provides 312 TFLOPS of FP16 compute. An H100 provides 989 TFLOPS. These are the numbers on the data sheet. The numbers in production are different. Typical AI training workloads achieve 30–50% of theoretical peak throughput. Inference workloads often achieve less. The gap between the hardware’s capability and the software’s utilisation of that capability represents wasted compute budget — compute that was purchased (or rented) but not used. This underutilisation is not a hardware deficiency. It is a software architecture problem — specifically, a mismatch between how the workload is structured and how the GPU hardware executes work. The GPU provides massive parallelism (thousands of cores), high-bandwidth memory (2–3 TB/s on modern architectures — hardware specification, not a benchmarked rate), and specialised compute units (tensor cores). Exploiting this hardware requires that the workload is structured to saturate these resources simultaneously. When it is not — when the workload serialises operations that could be parallel, when memory access patterns waste bandwidth, or when the kernel launch overhead dominates the compute time — the GPU sits partially idle while the wall-clock time extends beyond what the hardware could deliver. Where does the GPU utilisation go? GPU underutilisation has specific, diagnosable causes. Identifying which cause dominates a specific workload is the first step toward recovering the wasted compute. Symptom-to-bottleneck diagnostic map Symptom Likely Bottleneck Tool to Confirm Kernel operates below compute roofline but near memory bandwidth ceiling Memory bandwidth Nsight Compute roofline analysis Compute units idle during warp stalls; insufficient warps to hide memory latency Low occupancy Nsight Compute occupancy analysis (reports achieved occupancy and limiting factor: registers, shared memory, or block size) GPU spends more time idle between kernel launches than executing kernels Host-device serialisation Nsight Systems timeline profiling (shows when GPU was active, idle, and where serialisation points are) Kernel achieves a fraction of cuBLAS or tensor core throughput Inefficient kernel implementation Nsight Compute warp-level execution statistics and memory throughput analysis Memory bandwidth bottleneck. The GPU’s compute throughput exceeds its memory bandwidth by a ratio that has grown with each hardware generation. An A100 provides 2 TB/s of HBM2e bandwidth against 312 TFLOPS of FP16 compute — meaning a kernel that reads one FP16 value per operation is memory-bandwidth-bound, not compute-bound. Many common operations (element-wise operations, batch normalisation, activation functions, small matrix multiplications) fall into this category. The compute units are idle waiting for data from memory, regardless of how many compute units are available. The symptom is measurable: Nsight Compute’s roofline analysis shows the kernel operating well below the compute roofline but near the memory bandwidth ceiling. The fix depends on the operation: kernel fusion (combining multiple bandwidth-bound operations into a single kernel that reuses data in registers or shared memory), data layout optimisation (ensuring coalesced memory access patterns), and mixed-precision computation (using FP16 or INT8 to halve or quarter the memory bandwidth requirement per operation). Low occupancy. GPU occupancy measures the fraction of the GPU’s available warps (groups of 32 threads) that are active simultaneously. Low occupancy means the GPU cannot hide memory latency through warp switching — when one warp stalls on a memory access, there are not enough other warps ready to execute, and the compute units sit idle. Common causes: excessive register usage per thread (reducing the number of threads that fit in a streaming multiprocessor), excessive shared memory usage per thread block (reducing the number of blocks that can co-reside), or grid dimensions that do not saturate the GPU’s streaming multiprocessors. The diagnostic is direct: Nsight Compute reports achieved occupancy and the limiting factor (registers, shared memory, or block size). The fix requires adjusting the kernel’s resource usage — reducing register pressure through algorithmic restructuring, reducing shared memory usage through tiling strategies, or increasing the grid size to provide more concurrent blocks. Host-device serialisation. Every kernel launch, memory transfer, and synchronisation point between the CPU host and the GPU device creates a serialisation boundary where the GPU may be idle waiting for the host. In workloads with many small operations — such as model inference with complex branching logic, or training loops with frequent metric computation on the CPU — the cumulative host-device overhead can dominate the execution time. We have profiled inference pipelines where the GPU spent more time idle between kernel launches than it spent executing kernels. The fix is to reduce the number of host-device boundaries: batching operations into larger kernels, using CUDA graphs to replay a sequence of operations without per-launch overhead, moving decision logic from the CPU to the GPU where possible, and overlapping data transfer with computation using CUDA streams. Inefficient kernel implementations. Custom CUDA kernels that do not exploit the hardware’s memory hierarchy, warp-level primitives, or tensor cores leave performance on the table. In our experience across GPU optimisation engagements, a naive matrix multiplication that does not use shared memory tiling achieves 5–10% of the performance of a cuBLAS implementation (an observed range, not a benchmarked industry rate). A convolution kernel that does not use tensor cores on Volta+ architectures achieves a fraction of the hardware’s potential throughput. The three reasons GPUs don’t work out often trace back to these utilisation failures — the hardware was adequate, but the software architecture did not exploit it. Why profiling is not optional The causes of GPU underutilisation are not visible from wall-clock timing alone. In our experience across GPU profiling engagements, a training loop that takes 10 hours does not indicate whether the GPU was 90% utilised (near-optimal; further gains require algorithmic changes or hardware upgrades) or 30% utilised (significant room for improvement through software optimisation — an observed range, not a benchmarked rate). The only way to distinguish these cases is profiling. NVIDIA’s Nsight Systems provides timeline-level profiling: when the GPU was active, when it was idle, where the host-device serialisation points are, and how kernel launches overlap with data transfers. Nsight Compute provides kernel-level profiling: roofline analysis, occupancy analysis, memory throughput analysis, and warp-level execution statistics. Together, they provide a complete picture of where the utilisation is lost and what intervention would recover it. The investment in profiling is small relative to the compute cost it can recover. A workload running on 8× A100 GPUs at a cloud cost of £25/hour is spending £200/hour. A profiling session that identifies a 2× throughput improvement halves the compute cost for the lifetime of the workload — the profiling ROI is measured in days, not months. The organisational pattern we see repeatedly The pattern we encounter across our GPU Performance Audit engagements is consistent: the team has a workload running on GPU infrastructure, the workload is “too slow” (training takes too long, inference latency is too high, throughput does not meet the production requirement), and the proposed solution is more GPUs or better GPUs. Profiling reveals that the existing GPUs are 30–50% utilised (an observed range across our engagements, not a benchmarked industry rate), and the performance gap can be closed through software optimisation rather than hardware procurement. The optimisation path is systematic: profile → identify the dominant bottleneck (memory bandwidth, occupancy, serialisation, kernel efficiency) → apply the targeted fix → profile again → move to the next bottleneck. Each iteration recovers utilisation that translates directly to throughput improvement and compute cost reduction. If your GPU workloads are not achieving the throughput you expected from the hardware — and the diagnosis has not included systematic profiling — a GPU Performance Audit identifies the specific utilisation gaps and the interventions that close them, starting with the profiling data rather than the hardware upgrade proposal.
