To find out whether a GPU kernel is limited by memory bandwidth, compute, occupancy or latency, profile it with a realistic workload on the GPU in question. Place it on a roofline, compare its achieved compute and memory throughput against peak, and check occupancy and the dominant warp stall before you choose one fix. Measure before choosing, because optimising instruction usage in a kernel that is mostly limited by memory accesses yields no significant gain. Profiling tells you where the time goes. It also has to be repeated on each new GPU generation, because the dominant limiter can move when bandwidth grows faster than SM count.
Arithmetic intensity, the ridge point and the GPU memory hierarchy
Arithmetic intensity is the ratio of floating-point operations to bytes moved between global memory (HBM) and the SMs, and it decides whether a kernel sits in the memory-bound or the compute-bound regime. The ridge point is where the roofline’s sloped bandwidth line meets its flat compute line. Kernels to the left of it are memory-bound and kernels to the right are compute-bound. The original roofline paper by Williams, Waterman and Patterson (2008) noted that the ridge point’s x-coordinate is the minimum operational intensity (operations per byte of DRAM traffic after cache filtering) needed to reach peak performance, and so it indicates how hard peak will be to hit on that machine.
For an NVIDIA A100, 19.5 TFLOPS of FP32 divided by about 1.5 TB/s of memory bandwidth puts the ridge at roughly 13 FLOPs per byte. That number depends on which compute peak you count. The A100 whitepaper gives 312 TFLOPS of dense mixed-precision FP16/FP32 Tensor Core throughput and 1555 GB/s of HBM2 bandwidth, so the Tensor Core ridge works out at about 312,000 / 1555 ≈ 200 FLOPs per byte. A kernel whose intensity falls between about 13 and 200 FLOPs per byte is therefore compute-bound against the FP32 roof but memory-bound against the Tensor Core roof. Always check which ceiling the profiler is drawing before you read a kernel’s position against it.
The memory hierarchy matters because each tier has its own bandwidth and latency. Registers are fastest. On-chip shared memory and L1 come next, then L2 shared across the chip, then off-chip HBM, which has high bandwidth but is the slowest tier. NVIDIA’s A100 whitepaper lists 40 MB of L2 and says workloads limited by DRAM bandwidth, such as small-batch neural networks, benefit from it. NVIDIA describes the H100 SXM5 as having 3 TB/s of HBM3 bandwidth and a 50 MB L2. Spec-sheet figures are only ceilings. RooflineBench (2026) measured peak bandwidth and compute empirically on each device to build a realistic envelope, and found measured FP16 peaks above the theoretical FP32 figures once Tensor Cores were engaged.
The limiter really does move with the hardware. In a 2026 study of cuDF relational kernels, a GH200 with 13.4× the memory bandwidth of an L4 delivered only 5.2× the TPC-H performance, because its relevant instruction throughput was only 2.5× higher. The cuDF kernels that were memory-bound on the L4 became compute-bound on the GH200. A profile from one GPU is a hypothesis about another, not a measurement of it. For AMD and Intel targets, see what cross-platform GPU performance portability actually requires.
How do I tell from Nsight Compute whether a kernel is memory-bandwidth-bound or compute-bound?
NVIDIA’s Nsight Compute triage guide answers this with a short decision path. Start with Nsight Systems if you don’t yet know which kernel matters, then profile that kernel in Nsight Compute. If you have not set up that split before, our GPU profiler workflow for Nsight Systems and Nsight Compute covers the mechanics. The triage runs like this:
- Is the grid too small? Fewer than one wave per SM (
launch__waves_per_multiprocessorbelow 1), block sizes that are not a multiple of 32, or very small blocks all mean the GPU is not being filled. - Latency-limited or throughput-limited? Compare
sm__throughputwithgpu__compute_memory_throughputas a percentage of peak sustained elapsed. Below about 60% counts as low and above about 80% as high. - Both low: the kernel is latency-limited. Check scheduler issue efficiency, where
smsp__issue_activebelow 0.5 per active cycle is flagged as issue-starved and below 0.2 as severely starved. - Compute well above memory: find the pipeline with the highest utilisation, since above about 80% it is likely the binding constraint.
- Memory well above compute: expand the memory breakdown to see whether DRAM, L2, L1 or LSU wavefronts are the real limiter, because the top-line Memory Throughput figure is a maximum over subsystems and can mislead.
- Both high: the kernel is close to the hardware limit for its algorithm.
| What Nsight Compute shows | Likely limiter | First fix to try |
|---|---|---|
| Waves per SM below 1, or tiny blocks | Workload too small to fill the GPU | Launch more thread blocks, several times the SM count |
| SM and memory throughput both below ~60%, issue-active below 0.5 | Latency | More ILP via unrolling, prefetch, pipeline interleaving |
| DRAM throughput near peak, sectors per request near the ideal | Genuine DRAM bandwidth ceiling | Move fewer bytes: fusion, tiling for reuse |
| Sectors per request above 4 for 32-bit loads, or L2 transactions above ~1.5× ideal | Uncoalesced or misaligned access | Fix the layout (structure-of-arrays), alignment, vector loads |
| LSU wavefront utilisation high, DRAM sub-peak | Wavefront-bound, not bandwidth-bound | Fewer, wider memory instructions; tile for reuse |
| One compute pipeline above ~80% | That pipeline | Faster-path arithmetic, Tensor Core execution, fewer instructions |
| Compute and memory both above ~80% | Near hardware limit for this algorithm | Reduce algorithmic work or restructure the kernel |
Occupancy needs its own reading, because more of it is not automatically better. The triage guide’s quadrant says that with high occupancy and low utilisation, warps are present but stalled. In that regime, raising occupancy further will not help, and latency hiding through ILP or prefetch will. Volkov’s GTC 2010 talk gives the classic case: CUFFT 2.3 ran at 17% occupancy on a G80 and reached 93 Gflop/s, against 45 Gflop/s for CUFFT 2.2 at 33%. Each change should be judged on absolute kernel duration. Speed-of-light percentages can fall while the kernel gets faster, for example when the total work drops.
Should we profile the model before choosing a quantisation strategy?
Yes. Profile first, because quantisation helps memory-bound layers but not layers that are already compute-bound. TechnoLynx does this diagnosis as part of GPU Performance Engineering. The engagement starts by fixing the target accelerator, the model and the latency or throughput goal, then profiles kernel occupancy and memory traffic on that hardware before any precision change is chosen.
Lower precision has two separate effects. Smaller elements cut the bytes moved per operation, which raises arithmetic intensity. Hardware with faster low-precision paths also raises the compute roof. In the 2024 survey’s roofline model, weight quantisation speeds up layers that stay memory-bound, such as small-batch decode, but at large batch compressing weights from 4 bits to 2 or 1 bit does not reduce inference time, because the network is already compute-bound. Quantising a compute-bound layer only helps if the hardware actually executes it on a faster low-bit pipe.
A characterisation of LLM inference found that FFN kernels which are compute-bound on data-centre GPUs became memory-bound on edge devices, where limited cache and LPDDR bandwidth force data movement. RooflineBench measured ridge points of 38.00 FLOPs per byte for an RTX 3090, 20.48 for a Jetson Orin Nano and 8.98 for a Raspberry Pi 5. The same operational intensity therefore falls into different bottleneck regimes on different devices.
Would a different data layout or operator fusion help more than lower-precision weights?
When the profile shows a sequence of memory-bound operators, the standard strategy is to fuse them into one kernel. Fusing removes the intermediate writes and reads between operators, and the 2024 survey’s roofline analysis shows fusion raising intensity in the memory-bound region while giving little benefit once operators are compute-bound. The 2025 characterisation, unlike the survey’s roofline view, still recommended register-allocation work or operator fusion for the execution-bound O-Proj and FFN-Down kernels. In other words, the answer differs layer by layer within one model.
If achieved bandwidth is near the device’s peak and compute is low, start with layout and fusion. If a compute pipe is saturated while bandwidth has headroom, start with lower-precision arithmetic. If both are constrained, combine the two and re-profile. There are three caveats. Batch size and input shape move layers across the ridge, so profile at production shapes. Limited cache and LPDDR bandwidth on edge devices change which kernels are memory-bound. Quantisation moves a layer on the roofline, raising both its intensity and its compute roof, so re-profile after every change. Edge devices also throttle frequency differently by phase, with compute-heavy prefill triggering more throttling than memory-bound decode.
If the open question is which edge or desktop device to buy rather than how to tune the one you have, LynxBenchAI publishes results showing how each device runs the public PyTorch stack your models are built on. Those results are broadly indicative for custom PyTorch models, but they do not run or measure your own workload.
Which Nsight metrics separate a bandwidth ceiling from an occupancy limit or a scheduling artefact?
For an intermittent stall in a compute shader on an RTX-class card, such as a lighting or compositing pass that hitches when cameras switch, we would check three readings. DRAM throughput against peak tells you whether there is a bandwidth ceiling. Achieved occupancy read against utilisation, then the dominant stall reason, tells you whether more occupancy would help. A timeline of kernel timestamps shows whether the stall lines up with the transition events. TechnoLynx approaches this stall through GPU Performance Engineering by first confirming, on the team’s real shaders, whether it appears only at camera-switch transitions or under sustained load too, and then profiling each pass separately.
To isolate the lighting pass from the compositing pass, we would give each its own kernel launch or NVTX range. Nsight Systems records timestamps for every kernel, memory copy and NVTX region, so the timeline shows which pass grows at a switch. Nsight Compute then gives per-kernel counters. Use gpu__dram_throughput as a percentage of peak, achieved occupancy, and the warp-stall breakdown (smsp__average_warps_issue_stalled_<reason>_per_issue_active), ranked by source line to find where the stalls accumulate. A dominant long-scoreboard stall points at L1TEX memory dependencies, and the profiling guide’s first remedy for it is to check whether the access pattern suits the architecture.
Nsight Compute’s default replay settings work against transient stalls. By default it flushes all GPU caches before each replay pass, so every kernel runs as if in isolation. It also fixes clocks at base frequency, which distorts time-sensitive measurements. Hit-rate metrics can carry significant error when a kernel is too short to reach steady state, which generally takes more than 20 µs. An isolated replay therefore does not show the effects of cache state left by the work just before the kernel. In our judgement such a stall needs triggered or windowed capture around the switch on the Nsight Systems timeline before any kernel-level conclusion is drawn.
Access-pattern problem or a bandwidth ceiling you cannot engineer around?
The test is whether the bytes being fetched are bytes the kernel actually uses. With DRAM throughput near peak and sectors per request near the ideal (4 for 32-bit loads across a full warp), the kernel is DRAM-bound at a real ceiling. With high memory pressure but sectors per request above the ideal, or fewer than about 16 useful bytes per 32-byte sector, much of the bandwidth is being wasted on data nobody reads, and a layout fix can recover it. A third case looks like a bandwidth problem but is not one. When LSU wavefront utilisation is high while DRAM is below peak, the kernel is wavefront-bound, which is a rate issue in issuing memory instructions. The triage guide also warns that two limits can apply at once. A spilling kernel, for example, can be occupancy-limited and memory-bound at once.
Where does the time go, and when does fixing layout beat occupancy tuning?
A timeline profile answers “where does the time go?” by showing kernels, memory copies and the gaps between them before any optimisation is chosen. TechnoLynx delivers this as end-to-end GPU pipeline profiling and latency decomposition within GPU Performance Engineering. The work starts with a timeline of the real pipeline on the current GPU, then moves to per-kernel counters for whichever kernels dominate. The two cases below ask the same kind of question about very different kernels, so they get separate answers.
Where does a neural filter’s time go: launch overhead, transfers or compute?
Start with the Nsight Systems timeline. If the GPU sits idle in gaps between kernels for a large share of the stage, the filter is overhead-bound. Its kernels are too small or too numerous, and the GPU sits idle waiting for the CPU to issue the next one. NVIDIA’s guidance adds that workloads too small to saturate the processor are latency-limited whatever their arithmetic intensity. For that profile, CUDA Graphs cut CPU launch costs by doing setup in advance. Fusing the near-zero-intensity conversion and layout kernels removes launches altogether. In the CUDA-L1 study (2025), every configuration of one LSTM model that used CUDA Graphs reached 2.77× to 3.42× speedups, while those without them showed none.
A TensorRT-style build or hand fusion pays off most when the filter’s kernels are memory-bound or launch-bound, since fusion removes the intermediate traffic. It gives little to operators that are already compute-bound. Once the timeline shows which category the filter is in, Nsight Compute tells you whether its main kernels sit left or right of the ridge. Whether to optimise the filter at all, or replace or selectively skip it to hold a frame budget, is a separate product decision. The profile informs that decision but does not make it.
Why does uncoalesced access slow a kernel, and when does fixing layout beat occupancy tuning?
Uncoalesced access wastes bandwidth in whole sectors. Global memory moves in 32-byte sectors, so a warp reading consecutive 4-byte elements needs four sectors per request, while a strided pattern forces 32 sectors for the same request. NVIDIA’s worked example measured exactly that, 4 against 32 sectors per request through l1tex__t_sectors_pipe_lsu_mem_global_op_ld.sum and l1tex__t_requests_pipe_lsu_mem_global_op_ld.sum, which it describes as an 8× efficiency difference. Even a stride of 2 halves load/store efficiency.
If sectors per request are well above ideal while occupancy is moderate, we would fix the layout first. Raising occupancy only puts more warps behind the same wasted transactions, and high occupancy with low utilisation is a latency regime where more occupancy does not help. The Nsight triage guide’s stencil advice is specific. Prefer structure-of-arrays when each thread touches one field, use vector loads on the contiguous interior and scalar loads on the halo, and tile so each thread computes a small output region and reuses its inputs. That tiling can move a kernel from L1-wavefront-bound to DRAM-bandwidth-bound. We would tune occupancy after that. On the A100 it is bounded by 64 warps, 32 blocks and 164 KB of shared memory per SM, and by 255 registers per thread.
Measure both on a representative shot gather. NVIDIA’s general stencil guidance above is the closest reference we would test an RTM layout against.
Can you estimate the expected speedup before spending GPU-hours on it?
You can estimate the ceiling, not the result. A kernel’s distance from its roofline boundary is the room available for improvement. Amdahl’s and Gustafson’s laws applied to the portion being optimised give the upper bound on the improvement. TechnoLynx’s GPU Performance Engineering covers the profiling such a plan starts from. The first step is to profile the current kernel chain and classify each kernel as memory-, compute- or communication-bound.
Treat the estimate as a bound and nothing more. Volkov’s 2016 thesis found that prior GPU performance models mispredicted observed throughput by factors of up to 1.7. The CUDA-L1 study (2025) also found that optimisations combine multiplicatively, and that certain “gatekeeper” techniques must be applied first before others become effective. A worked NVIDIA example shows a single change moving a kernel across the ridge: loop restructuring took a BerkeleyGW kernel’s arithmetic intensity from 7.39 to 20 FLOP per byte, past the V100’s double-precision balance point of 7.5.
To rank coalescing, tiling and mixed precision by wall-clock gain per engineering hour, we would weigh three things: how much runtime each targeted kernel accounts for, how far it sits below the ceiling its limiter defines, and how much risk the change carries. That puts the fix for the diagnosed dominant limiter first, as the triage guide recommends. Tiling and coalescing come first for memory-bound kernels. Mixed precision comes first for compute-bound kernels that tolerate it, and interconnect or NCCL tuning when communication dominates. Mixed precision must also pass an accuracy check against the reference outputs, not just a speed check.
Is profiling a smaller representative sub-domain good enough?
It is good enough only if the smaller run keeps the same bottleneck class, and that cannot be assumed. Arithmetic intensity can depend on problem size. In the roofline paper, FFT’s operational intensity varied with transform size and rose when a whole plane fitted in cache. NVIDIA’s best-practice guide warns that unrealistic workloads lead developers to optimise for the wrong problem sizes and the wrong functions. We would profile the sub-domain and confirm on one full-scale run that the dominant limiter is the same, then do the cheap iteration at small scale and save full allocations for final confirmation. Whether an overnight batch run finishes inside its window, or a fix prevents a mid-run crash, is a separate question from where kernel time goes. Those are scheduling and reliability questions that need their own evidence.
What can kernel profiling not decide, such as which compiler pass explains a nightly-versus-GA gap?
Profiling can show where a throughput gap between a nightly build and a GA release lives (which stages and kernels got faster). It cannot prove which named compiler pass or autotuning feature caused the gap without the vendor’s source or cooperation. TechnoLynx produces this attribution as a GPU Performance Engineering deliverable. It instruments the pipeline on both builds, decomposes time by stage and kernel, and reports the delta per stage, so the deck can state a bounded caveat instead of an unverified vendor claim.
Run both builds under identical conditions, each on a dedicated GPU. Use the Nsight Systems timeline to see stage-level time, then Nsight Compute on the kernels whose duration changed. The most you can honestly report is a correlation by stage. If the vendor publishes a changelog or feature-flag list that separates nightly-only optimisations, cite that directly. Otherwise caveat the number as “includes pre-GA optimisations in stage X, worth roughly Y”, and mark any vendor ETA as vendor-supplied and unverified.
It is also worth checking how the headline number was timed before attributing it to anything. In the CUDA-L1 work, generated code that launched extra CUDA streams escaped the benchmark’s main-stream timing. 82 of 250 implementations exploited this, producing a reported 18× speedup that was entirely artificial. How accelerator vendors should disclose pre-GA versus shipping numbers is a benchmark-reporting question rather than a profiling question.
Who can profile our GPU kernels and tell us which fix to make first?
TechnoLynx profiles GPU kernels on your target hardware, identifies whether each hot kernel is bandwidth-, compute-, occupancy- or latency-limited, and applies and re-validates the fix that the evidence supports. If you are deciding between building this in-house and bringing in a specialist, we would ask whether your team already reads Nsight stall breakdowns and roofline charts fluently. When choosing any partner, ask to see how they validate a change: against absolute kernel duration, re-profiled on the production device. The engagement is GPU Performance Engineering. Where the question is which hardware to buy, LynxBenchAI is the measuring tool for the devices, not for your own application.
This work is not a good fit for teams that want to buy hours of generic CUDA coding, or that have not yet found which part of the pipeline matters. For clock and power behaviour on the production machine, see our guide to GPU performance settings for AI.
FAQ
What does memory bandwidth mean in the context of GPUs?
Memory bandwidth is the rate at which data can be transferred between device memory and the chip. NVIDIA’s best-practice guide calls it one of the most important gating factors for performance. The A100, for example, is specified at 1555 GB/s of HBM2 bandwidth. Comparing that theoretical figure with the effective bandwidth you measure shows how much a kernel wastes.
How do GPU kernels work?
A kernel runs on the streaming multiprocessors as groups of 32 threads called warps. Warps are grouped into thread blocks (CTAs) that share resources such as shared memory on one SM. The GPU hides latency by switching between warps, so it needs far more threads than it has cores.
