GPU kernel profiling shows you which part of a kernel is actually costing time before you change any code: low occupancy, stalled warps, divergent lanes or wasted memory transactions. NVIDIA’s Nsight Compute triage guide sets out an iterative workflow. If you don’t yet know which kernel matters, start with Nsight Systems. Then run Nsight Compute on the single largest limiter, make one targeted change at a time, re-profile, and judge every change by absolute kernel duration rather than utilisation percentages. The build you profile also has to be release-optimised and carry line-table information. Metrics are collected at the SASS level, so without line tables they can’t be attributed to source lines.

TechnoLynx runs this loop for teams through GPU Performance Engineering. The first step is an audit of your current build flags. After that, TechnoLynx profiles on your target hardware, restructures the code path the profile identifies, and re-profiles to measure the effect of each change. One practitioner’s warning holds throughout: treat performance advice with suspicion, and always profile before and after an ‘improvement’, because the results sometimes go against common knowledge.

What GPU kernel profiling measures that a timeline does not

A timeline shows when each kernel and memory copy ran on the GPU. A kernel profiler tells you why the kernel took as long as it did. Nsight Systems is a system-wide tool: it draws a timeline across CPUs and GPUs to show where the largest optimisation opportunities are. It traces CUDA runtime and driver calls alongside the GPU work they launch, with kernels and memory copies placed on their streams. Its GPU Metrics sampling adds SM activity, warp occupancy, tensor core activity and PCIe or memory bandwidth over time. That answers questions like “is my GPU idle?” or “are my warp slots full?”.

Nsight Compute looks at a particular kernel in more detail. It is an interactive kernel profiler that collects detailed metrics for each kernel. The metrics come in sections (SM and memory utilisation, the memory units, the scheduler), and you choose sections by how much detail and overhead you can afford. It periodically samples warp program counters and scheduler states, so stall reasons can be tied to individual instructions. Its source page lines metrics up against your CUDA C code when the binary carries line information. The GPU can only collect a limited number of hardware counters at once, so Nsight Compute replays a kernel several times to gather a full set. That replay cost is why you aim it at one kernel, not a whole application.

No single profiler provides complete visibility on its own. KernelPro is a research system for profiling-guided CUDA kernel optimisation. Its authors combine three sources: Nsight Compute for kernel metrics, Nsight Systems for launch overhead and synchronisation, and SASS disassembly for instruction-level detail such as register spills. Their reason is that different bottlenecks show up at different levels of abstraction. The same paper measured something worth remembering. Giving a language model about 50 raw Nsight Compute counters with no interpretation produced a geometric-mean speedup of 1.77×. That is worse than giving it no profiling feedback at all (3.35×). Their interpreted pipeline reached 4.00×. The experiment tested LLM agents, not people, and we read it as a reminder that a counter dump is not a diagnosis, and the skill is reading metrics in context. For using kernel metrics in hardware decisions, see why kernel-level profiling beats vendor specs for workload decisions.

Should I start in Nsight Systems or Nsight Compute?

Start in Nsight Systems whenever you don’t yet know which kernel deserves attention, and move to Nsight Compute once you do. NERSC’s guidance to its users draws the same line: Systems for an overview of the whole code, Compute for a close look at one kernel. The timeline also shows excessive host-to-device transfers, which in one NERSC example slowed the code significantly. You can limit Nsight Systems to the performance-critical region with cudaProfilerStart and cudaProfilerStop, and mark regions with NVTX ranges. That keeps the data manageable and keeps attention on code worth optimising. For the wider optimisation process this kernel work sits inside, see our profiling-first approach to improving GPU performance.

Why the two tools report different kernel durations

Differences between the two tools’ kernel durations have documented causes. Nsight Compute locks SM clocks by default; Nsight Systems does not. Nsight Compute flushes GPU caches between replay passes, while Nsight Systems collects in a single pass and flushes nothing. Nsight Compute also serialises kernel launches unless you use a dedicated replay mode, so kernels that normally overlap on the device report different durations in the two tools. For comparable figures, NVIDIA recommends locking clocks externally with nvidia-smi and passing –clock-control none to ncu. For cache-sensitive workloads it recommends –replay-mode application with –cache-control none. A study of Shader Execution Reordering took this approach: it profiled with Nsight Compute at a locked base clock and measured production wall times without the profiler at boost clock. Use Nsight Compute’s cycle counts as the reproducible lab measurement and a profiler-free run as the production number.

The profile-first loop, step by step

1. Find the kernel that matters on the Nsight Systems timeline.
2. Profile that kernel in Nsight Compute, filtered so each kernel runs only once (NERSC uses –kernel-id :::1, because profiling quickly gets expensive without it).
3. Identify the single largest current limiter, and don’t optimise from one metric in isolation.
4. Apply one targeted refactor.
5. Re-profile with identical capture settings, then keep or revert the change based on its measured impact.
6. Judge by absolute kernel duration. Speed-of-light percentages can drop while the kernel gets faster.
7. Stop when the expected gains are small compared with the complexity and risk.

The bottleneck moves after each successful fix, which is why the loop repeats. In one worked example, spreading independent data sets across blocks cut kernel duration from 2.92 s to 0.0789 s. Restructuring the averaging phase with warp-stride and block-stride loops then took it to 0.0216 s, after which profiling moved the bottleneck to LSU and shared-memory pressure.

Why do kernels go missing or show no metrics in a profiling session?

An empty or failed session has several documented causes, from counter permissions to replay the application can’t tolerate. Work through this checklist:

• ERR_NVGPUCTRPERM. The user running the application has no permission to access NVIDIA GPU performance counters. The restriction applies from driver 418.43 on Linux and 419.17 on Windows. To fix it, have an administrator enable non-admin profiling, or run under sudo or with CAP_SYS_ADMIN on Linux, or use “Run as administrator” on Windows.
• Unsupported GPU or configuration. Nsight Compute supports a listed set of devices per version and may be unable to profile GPUs in configurations such as SLI. NVIDIA’s developer blog notes it does not support devices of compute capability 6.0 and lower.
• Another tool holding the counters. Nsight Systems GPU Metrics can’t run at the same time as Nsight Compute or DCGM.
• Sections or metrics not enabled. Only the sections and metrics enabled in the selection are collected. Check availability with ncu –query-metrics.
• Replay the application can’t survive. Kernel replay saves and restores memory between passes, and application replay relaunches the program, which requires deterministic execution. Kernels that must run concurrently need Range Replay. Range Replay needs start and end markers and contexts that can be synchronised at the start of the range.
• MPS clients. Nsight Compute can profile combined MPS execution but generally can’t isolate one client.
• Data lost at exit. Synchronise all GPU work and call cudaProfilerStop before the application exits, or unflushed data is lost.
• Source page shows only disassembly. The code was compiled without -lineinfo.

Why line-level profiling needs a build that keeps symbols and line info

Source-level profiling needs line tables in the device code and symbols in the host binary. Linking with gcc -s removes the host binary’s symbol tables and debug sections. The fix is not to drop -s from the release build. Instead, add a profiling configuration beside it that keeps -g, leaves out -s and adds nvcc -lineinfo, while staying release-optimised. TechnoLynx starts GPU Performance Engineering work here with a build-flag audit. TechnoLynx finds where -s is applied, defines the profiling target next to the release one, checks that Nsight Compute, CUPTI and perf resolve symbols on it, and documents which target contributors should use.

What gcc -s actually strips

The gcc -s flag passes -s to the linker, which removes the static symbol tables (.symtab, .strtab) and the debug sections (.debug_*) from the executable. The dynamic symbol table (.dynsym) survives because the dynamic linker needs it at runtime. Tools such as perf need those symbols to report which functions are slow. NVIDIA’s Nsight Systems guide also notes that stripping a -g build removes non-exported function names and the unwind information it uses for backtraces. On the device side, the Nsight Compute source view depends on nvcc’s -lineinfo (or –generate-line-info) flag. Without it you get disassembly only.

Profile an optimised build, not a debug build

A profiling build should not be a debug build. NVIDIA’s guidance is to time and analyse an optimised build, because debug builds change code generation and timings, and to add line-table information when you need source-page diagnosis. In practice that means -O2 or -O3 with nvcc -lineinfo rather than -G, which produces unoptimised device code for the debugger. On the host side, -g can be combined with optimisation flags to give an optimised binary that still carries debugging symbols. Nsight Systems backtraces default to frame pointers on Linux, and those only produce full stacks when the binary is compiled with -fno-omit-frame-pointer.

Two targets, or one build with split symbols?

There are two sound layouts. The first is a separate profiling target (release flags plus -g and -lineinfo, without -s) that contributors opt into, while the shipped artefact keeps -s. Its risk is drift. If the profiling target stops matching the release target’s other flags, its profiles stop describing production.

The second layout ships a stripped binary from the same build. You compile release with -g, split the debug information into its own file, then strip the shipped binary. objcopy –only-keep-debug writes the symbols to a file, objcopy –strip-debug removes the debug sections from the shipped binary, and –add-gnu-debuglink lets tools find the symbol file again. GNU strip keeps section headers precisely so that a debug-info file can still be matched to its executable. Linux distributions ship small binaries this way, with debug-symbol packages under /usr/lib/debug. Nsight Systems searches that path by default and uses a debug file when its BuildID matches. NVIDIA ships obfuscated symbols for its CUDA libraries, applied either by unstripping the library or by deploying the .sym file.

An ACCU article on release-mode debugging prefers a single build where possible, so users run the code that was tested. It also recommends adding symbolic information to retail builds without shipping it to customers. Split symbols remove the drift risk, and a separate profiling target is the fallback when your build system makes splitting awkward. Removing -s from the default build is the one option we would rule out. Unstripped binaries are larger than stripped ones. Whichever layout you choose, write it down in the contributor guide: which target to build for profiling, which for shipping, and why.

How do I tell whether a kernel is limited by occupancy, divergence, memory bandwidth or bank conflicts?

Compare compute and memory throughput first, then the stall reasons, then the specific counter for each suspect. No single metric settles the question on its own. TechnoLynx does this breakdown on a named kernel or shader as part of GPU Performance Engineering. TechnoLynx defines the worst-case scenario to capture, chooses the metrics, profiles on your target hardware, restructures the code path the profile points to, and re-profiles to confirm the improvement without regressing correctness.

NVIDIA’s Nsight Compute triage guide classifies a kernel by comparing SM throughput with memory throughput. Below about 60% is the latency-risk regime, and above about 80% is near the limit. If both are low, check what prevents instructions from issuing. If compute is far above memory, find which pipeline is the limiter. If memory is far above compute, look for the memory mechanism that dominates. Scheduler statistics then show whether warps are issuing. An issue-active rate below 0.5 per cycle is flagged as issue-starved, and few eligible warps among many active ones point to dependency or synchronisation stalls.

Reading stall reasons: a decision table

Stall samples record when warps were stalled and couldn’t be scheduled. Two rules frame the table. First, the goal is higher issue-slot utilisation, not every stall counter at zero: reducing stalls mainly speeds things up when low issue activity makes the kernel latency-limited. Second, a sample is charged to the instruction waiting for a result, not the one producing it, so you trace back through source and SASS to find the cause.

Dominant stall reason	What it usually means	First thing to check
Long scoreboard	Waiting on L1TEX data (global, local, texture). The kernel is memory-latency-bound, not necessarily bandwidth-bound	Access pattern and coalescing. If bandwidth isn’t saturated, add ILP, unrolling or pipelining
Short scoreboard	Usually shared-memory operations. Also special math (MUFU) or dynamic branching	Memory Workload Analysis for bank conflicts
LG throttle	Local or global memory instructions issued extremely often	Redundant global accesses, local arrays, register spills. Combine narrow operations into wider ones
Barrier	Warps waiting at a CTA barrier, often after divergent code paths	Uniform work per block. Consider splitting blocks of 512 threads or more
Branch resolving	Waiting for a branch target to be computed and the program counter updated	Fewer branches and less control-flow divergence
Math pipe throttle	One math pipeline oversubscribed. The kernel is usually compute-pipeline-bound	Instruction mix across pipelines. More active warps
Wait	Fixed-latency dependency. Usually tops the list only in already-optimised kernels	Unrolling, more active warps, lower-latency instructions

Divergence and the counter that shows it

Divergence shows up as idle lanes while the warp keeps issuing instructions for the rest. The workload-wide signal is smsp__thread_inst_executed_per_inst_executed.ratio. NVIDIA’s triage guide treats values below about 24 out of 32 as significant divergence, and the same threshold on the predicated-on variant as predication overhead. At line level, the Divergent Branches metric counts divergent branch targets wherever two or more active threads go different ways.

The two counters answer different questions, and the difference matters. A study of GPU optical-photon Monte Carlo separates three SIMT divergence modes. Branch divergence serialises different control paths within one instruction. Execution, or lifetime, divergence is lanes running different numbers of loop iterations, so the warp waits for its slowest lane. Memory divergence is scattered addressing. In that kernel, branch and memory divergence were minor and lifetime divergence dominated. The largest measured improvement was recovering active lanes from 2.64 to 20.38 of 32. Afterwards DRAM throughput sat at 1.98% of peak, which put the kernel in a latency- and occupancy-bound regime rather than a bandwidth-bound one. Advice like ‘avoid branches’ only makes sense in context, and the profile is what supplies that context. If the idle lanes come from loop-length variation, flattening branches will barely move the ratio.

Shared-memory bank conflicts

Shared memory has 32 banks. When two addresses in one request fall in the same bank, the access has to be serialised. NVIDIA’s triage guide counts the cost in wavefronts. Wavefronts per request equals the largest number of distinct cells any one bank has to serve, and bank conflicts equal wavefronts minus one. A 4-byte stride is ideal, a broadcast of one address is conflict-free, and a 128-byte stride puts all 32 threads on one bank for 32 wavefronts. The counters to read are l1tex__data_bank_conflicts_pipe_lsu.sum and the derived L1 Conflicts Shared N-Way, where 1-way means no conflict. High short-scoreboard stalls point to shared-memory operations, which NVIDIA advises checking for bank conflicts. One caveat: a profiler only sees the access pattern of the input you ran. A symbolic-analysis study of bank conflicts points out that input-dependent shared-memory patterns can go unnoticed during profiling.

Memory bandwidth, latency and spills

High memory traffic does not by itself make a kernel bandwidth-bound. Uncoalesced 32-bit loads show up as more than 4 sectors per request. In NVIDIA’s worked example, a matrix-add kernel at 32 transactions per request dropped to 4 after the index order was swapped, cutting elapsed cycles by about 68%. If DRAM throughput is well below peak and nothing else is saturated, the kernel is short of bytes in flight and needs ILP, vectorised loads or async copies. If LSU wavefront utilisation is high while DRAM stays below peak, the kernel is wavefront-bound. More in-flight loads won’t help there, and the access pattern has to change.

Local-memory traffic points to register spilling, or to thread-private arrays the compiler couldn’t keep in registers. Spills show up as long-scoreboard or LG-throttle stalls and often look like a generic memory-bound symptom. The direct counters are derived__local_spilling_requests and sass__inst_executed_register_spilling_mem_local. Not every spill hurts. That is why the KernelPro spill detector only fires when spills cause latency bubbles, bandwidth contention or register-bound occupancy.

Why is my achieved occupancy so far below theoretical?

Occupancy is the ratio of active warps per multiprocessor to the maximum possible. The theoretical figure is the upper bound set by the launch configuration and per-CTA resources. The achieved figure is the active warps measured during execution. Theoretical occupancy is reported as sm__maximum_warps_per_active_cycle_pct, and achieved occupancy appears per kernel alongside it. When the theoretical value is low, the launch__occupancy_limit_* metrics name the resource responsible: registers, shared memory, block size (warps), blocks per SM or barriers. Registers per thread are the most important resource under the compiler’s control.

A gap between theoretical and achieved occupancy points elsewhere. NVIDIA notes that a large discrepancy during execution typically indicates a highly imbalanced workload. In the KernelPro trajectory, a larger GEMM tile underperformed because of wave quantisation at that problem size, and the agent went back to the smaller tile. Capping registers with launch_bounds or -maxrregcount trades occupancy against spill traffic. NVIDIA’s advice is to re-measure and back off if spills increase.

Register pressure limits occupancy on AMD hardware too. AMD’s GPUOpen guide shows that a 1024-thread group needing 40 VGPRs fits only one group per CU, which gives 40% occupancy. Its author reports that cutting one shader from 48 to 32 VGPRs gave a 50% speed-up on one platform with no other change.

Higher occupancy doesn’t always mean higher performance, though low occupancy always reduces the ability to hide latency. NVIDIA’s triage guide flags achieved occupancy below 60%, unless Tensor Cores dominate the workload or throughput is already high. When occupancy is near theoretical but warp latency is high, further occupancy gains give diminishing returns. A graphics practitioner adds a further caution: low occupancy isn’t always bad, and high occupancy bought with a low VGPR count can force the compiler to serialise memory fetches.

Applying it to a clustered lighting compute shader

Take a clustered lighting compute shader that stalls on the worst-case crowd frame. Where are the cycles going: wave divergence, VGPR spill or LDS bank conflicts? And how far is achieved occupancy from theoretical on that frame? Capture the frame that really is worst-case for crowd density. A synthetic stress case can produce a profile that doesn’t match the frame you ship.

If divergence dominates, two candidates map onto the divergence modes above. Per-light culling branches fit the definition of branch divergence above. Variable light-list lengths per cluster tile fit the definition of loop-length divergence above. So the divergent-branch count on the culling lines, compared with the lane-utilisation ratio inside the light loop, tells you which code path to restructure. If spills or bank conflicts dominate instead, fix register pressure or LDS layout before touching branches. AMD’s guide suggests parking data in LDS across a VGPR peak, which can address both at once. The counter names in this article are NVIDIA’s. Once the fix holds, keep the same capture in your performance-regression workflow so the next content change is measured against it.

When is it worth bringing in an outside GPU performance engineer?

Outside help pays off when the profiling and tuning is a bounded piece of work with a deadline attached, and the engineers who could do it are committed to production support or a research cycle. At that point, pulling them off their work costs more than the engagement. TechnoLynx offers this as a fixed-term GPU Performance Engineering engagement. TechnoLynx confirms which kernel chain stages are bottlenecked and why, optimises on your GPU fleet, validates against your sign-off criteria, and deploys the kernels back into your pipeline.

What makes specialists effective is judgement more than tooling. The KernelPro authors note that experts are effective because of the pattern recognition and diagnostic reasoning they apply to profiler numbers, not because they can read the numbers. Three conditions make the case for an outside team stronger. The work can be scoped to profiling, targeted kernel changes and validation. Correctness criteria exist, or can be written, so that no performance change ships unverified. The third condition is that the slowdown sits in the kernels.

That last condition deserves a check before anyone is hired. If an 8-GPU node is now shared with a new workload, Nsight Systems GPU Metrics shows whether the GPUs are already full. On cost, we can’t give a general figure. The comparison that matters is the engagement against the cost of a missed deadline plus the work your own team sets aside, and both depend on your schedule.

Who can take on the profiling without pulling our engineers off production?

TechnoLynx profiles and optimises CUDA kernel chains for teams that can’t spare their own engineers, in a time-boxed GPU Performance Engineering engagement that leaves production and research schedules alone. On disruption, the engagement is scoped to fit your existing CUDA implementation and to leave production and research schedules alone. Ramp-up time depends on how well the kernel chain is documented, and no general figure applies; TechnoLynx starts by framing which stages are bottlenecked and why, before changing any code. On scope, the engagement covers profiling, targeted kernel optimisation and validation sign-off, and the optimised kernels go back into your pipeline.

The engagement doesn’t suit a team that wants open-ended CUDA development hours or the cheapest possible build. It also isn’t the right first move when the slowdown comes from GPU sharing or scheduling, which needs an infrastructure fix. A team that already has a GPU performance specialist with free capacity has a realistic in-house route, as does a team willing to hire a contractor for a defined term.

FAQ

What are debug symbols?

Debug symbols are the function and variable names that debug information keeps in a binary; a symbol table maps such names to addresses. Debuggers such as GDB use them, and profilers like perf need them to show slow functions.

What do the -g and -O flags do in GCC?

-g tells GCC to emit debugging information, and -O with a level such as -O2 sets how aggressively the code is optimised. You can combine them to get an optimised binary that still carries debugging symbols.