Why wall-clock timing is not profiling A CUDA kernel takes 12 milliseconds to execute. Is that slow? The question is unanswerable without context. To use a worked example from our GPU profiling engagements: if the kernel is performing 500 billion floating-point operations and the GPU’s peak throughput is 312 TFLOPS, the theoretical minimum is 1.6 milliseconds — the kernel is achieving 13% of peak (an operational measurement, not an extrapolated benchmark), and there is significant room for improvement. If the kernel is memory-bandwidth-bound, performing element-wise operations on 4 GB of data on a GPU with 2 TB/s memory bandwidth, the theoretical minimum is 2 milliseconds — the kernel is achieving 17% of the memory bandwidth ceiling (again, an operational measurement from that worked profile), and the optimisation target is memory access patterns rather than compute efficiency. Wall-clock timing tells you how long the kernel took. Profiling tells you why it took that long — and what specific intervention would make it faster. Without profiling, GPU optimisation is guessing. With profiling, it is engineering. Minimum viable profiling workflow Before optimising anything, every team should complete these steps: Capture a system-level timeline with Nsight Systems. Run the full workload under Nsight Systems to record kernel launches, memory transfers, synchronisation events, and GPU idle gaps. Identify the top 3–5 kernels by cumulative execution time. Sort the timeline by kernel duration — in our experience across GPU profiling engagements, these dominant kernels typically account for 80%+ of GPU time and are the only ones worth optimising (an observed range, not a benchmarked industry rate). Profile each dominant kernel with Nsight Compute. Collect the roofline chart, achieved occupancy, memory throughput breakdown, and warp execution efficiency for each target kernel. Classify each kernel as compute-bound or memory-bound. Use the roofline position and arithmetic intensity to determine which hardware ceiling — compute throughput or memory bandwidth — is the binding constraint. Quantify the gap between current and achievable performance. Compare the kernel’s achieved throughput against the theoretical ceiling to calculate the optimisation headroom — as a planning heuristic from our engagements (not a benchmarked industry rate), 20% of peak means up to 5× potential improvement. Select the single highest-impact optimisation per kernel. Match the identified bottleneck to a specific fix: coalescing for uncoalesced memory access, mixed-precision for tensor core underutilisation, or kernel fusion for launch overhead. Re-profile after each change to verify the improvement. GPU optimisation is iterative — fixing one bottleneck shifts the constraint, so always re-measure before applying the next intervention. The roofline model: where does your kernel sit? The roofline model is the foundational framework for GPU kernel performance analysis. It maps every kernel to a two-dimensional space defined by two hardware limits: compute throughput (FLOPS) and memory bandwidth (bytes/second). The kernel’s arithmetic intensity — the ratio of compute operations to memory accesses — determines which limit the kernel is constrained by. Compute-bound kernels have high arithmetic intensity: they perform many operations per byte of data accessed. Large matrix multiplications (GEMM), convolutions with large filter sizes, and dense attention computations fall into this category. For these kernels, the optimisation target is compute efficiency: are the tensor cores being used? Is the warp execution efficient? Are there unnecessary branches or divergent execution paths? Memory-bound kernels have low arithmetic intensity: they perform few operations per byte accessed. Element-wise operations (ReLU, sigmoid, addition), batch normalisation, and small matrix operations fall into this category. For these kernels, the optimisation target is memory throughput: are memory accesses coalesced? Is the data layout cache-friendly? Can multiple operations be fused to reuse data in registers or shared memory? Nsight Compute generates the roofline chart automatically for any profiled kernel. The kernel appears as a point on the chart; its position relative to the compute ceiling and memory bandwidth ceiling indicates which limit is binding and how much headroom remains. In our experience reading these charts, a kernel sitting at 80% of the memory bandwidth ceiling is well-optimised for a memory-bound workload — further gains require reducing the memory access volume (through fusion or algorithmic change), not improving the access pattern. A kernel sitting at 20% of the compute ceiling is poorly optimised — there is a 5× potential improvement from better compute utilisation (an observed range across our engagements, not a guaranteed outcome). The profiling workflow We follow a systematic profiling methodology that moves from system-level to kernel-level analysis. This progression is important: in our experience across GPU profiling engagements, optimising an individual kernel that contributes 2% of total execution time is wasted effort if the system-level bottleneck is host-device serialisation. Step 1: System-level timeline (Nsight Systems). Run the workload under Nsight Systems to produce a timeline of GPU activity: kernel launches, memory transfers, synchronisation events, and idle periods. The timeline reveals the macro-level performance structure. Questions this step answers: What fraction of the total execution time is the GPU active? Where are the idle gaps? Which kernels dominate the execution time? Is there host-device serialisation that could be overlapped? Step 2: Identify the dominant kernels. Sort kernels by cumulative execution time. Typically across our engagements, 3–5 kernels account for 80% or more of the GPU execution time (an observed pattern, not a benchmarked industry rate). These are the optimisation targets. Optimising kernels that do not appear in the top contributors has negligible impact on total execution time. Step 3: Kernel-level profiling (Nsight Compute). Profile each dominant kernel individually with Nsight Compute. The detailed analysis includes roofline positioning, achieved occupancy, memory throughput breakdown (global, shared, L1, L2), warp execution efficiency, and instruction mix. For each kernel, the profiler identifies the specific bottleneck and often suggests the relevant optimisation. Step 4: Apply targeted optimisations. Based on the profiling data, apply the specific optimisation that addresses the identified bottleneck. For a memory-bound kernel: improve coalescing, fuse with adjacent operations, or change the data layout. For a compute-bound kernel: enable tensor core usage, reduce warp divergence, or increase ILP (instruction-level parallelism). For an occupancy-limited kernel: reduce register or shared memory usage per thread. Step 5: Re-profile and iterate. After applying the optimisation, re-profile to verify the improvement and identify the next bottleneck. GPU optimisation is iterative — fixing one bottleneck often reveals the next. The process continues until the kernel’s performance is within acceptable distance of the theoretical ceiling, or until the dominant bottleneck shifts to a different kernel or a system-level constraint. The GPU underutilisation patterns we encounter in production workloads are identified through this workflow. Without it, the team is optimising by intuition rather than by measurement. Common profiling findings and their fixes From our GPU Performance Audit engagements, the most common profiling findings are: Uncoalesced global memory access. Threads in a warp access non-contiguous memory locations, resulting in multiple memory transactions where one would suffice. The fix: restructure the data layout (Array of Structures to Structure of Arrays conversion) or modify the access pattern to ensure consecutive threads access consecutive memory addresses. Typical improvement across our engagements: 2–4× memory throughput for the affected kernel (an observed range, not a benchmarked industry rate). Tensor core underutilisation. The kernel performs matrix multiplications in FP32 without using tensor cores, despite running on hardware that supports them (Volta and later). The fix: convert to mixed-precision (FP16 input, FP32 accumulate) and use CUDA’s WMMA API or cuBLAS’s tensor core-enabled routines. Typical improvement across our engagements: 4–8× throughput for the matrix multiplication operations (an observed range, not a guaranteed outcome). Excessive kernel launch overhead. Hundreds or thousands of small kernels are launched sequentially, with the CPU overhead of each launch (5–15 microseconds) exceeding the GPU execution time of the kernel. The fix: fuse small kernels into larger ones, or use CUDA graphs to batch the launch sequence into a single replayable graph. Typical improvement across our engagements: 2–10× reduction in total execution time for launch-overhead-dominated workloads (an observed range, not a benchmarked industry rate). Register pressure causing spills. The kernel uses more registers per thread than the hardware provides, causing register spills to local memory (which is backed by global memory and is 100× slower in our experience). The fix: refactor the kernel to reduce register usage — simplify expressions, reduce loop unrolling, or split the kernel into stages that require fewer simultaneous live variables. Typical improvement across our engagements: 1.5–3× throughput recovery (an observed range, not a guaranteed outcome). What profiling tells you that benchmarks do not Benchmark numbers — training throughput, inference latency, images/second — tell you the current performance. Profiling tells you the achievable performance and the specific path to reach it. The gap between current and achievable is the optimisation opportunity, and its magnitude determines whether the intervention is worth the engineering investment. We have seen profiling reveal opportunities ranging from negligible (the workload is already well-optimised, within 10% of the hardware ceiling) to transformative (the workload is at 15% of peak, with a clear path to 60% through targeted kernel optimisation and memory layout changes) — an observed range across our GPU Performance Audit engagements, not a benchmarked industry rate. The profiling data determines which situation you are in — and prevents the expensive mistake of planning GPU memory for training or procuring more hardware when the existing hardware is underutilised. The first step is always the same: profile the workload, identify the dominant bottleneck, and quantify the gap between current and achievable performance — which is what a GPU Performance Audit delivers.
