GPU Profiler Tools and Workflow: Nsight Systems and Nsight Compute

The Nsight Compute CLI does no GPU or API tracing; Nsight Systems covers that. We would use three kinds of tool, in order. First, an Nsight Systems timeline shows CUDA API calls, GPU kernels and memory transfers, with optional NCCL tracing. Next, Nsight Compute profiles the specific kernel that timeline points to. Finally, communication-level tools such as NCCL Inspector analyse collective performance across GPUs. Nsight Systems puts CPU, GPU, CUDA library and network activity on one timeline, and Nsight Compute is the interactive kernel profiler for CUDA applications. You find multi-GPU scaling loss the same way: you measure it, rather than reading it off a topology diagram.

TechnoLynx profiles your training or inference run end to end and breaks its time into compute, communication and I/O. This is part of its GPU Performance Engineering work, and the first step is a profiling run of the workload as it runs today. In a published project for SIM registration software, frame capture on low-end Android devices turned out slower than the inference itself. No amount of model compression would have closed that bottleneck. When porting V-Nova’s OpenCL codebase to Apple Metal, a replay tool that recorded kernel inputs and outputs let the team compare behaviour side by side. After the port from OpenCL to Metal, the case page reports a 229% average encoding speedup and a 340% average decoding speedup. Neither project was a multi-GPU training job. What carries over is the habit of measuring kernel execution before deciding what to optimise.

That habit runs through every section below: profile before you optimise.

Which GPU profiler answers which question

NVIDIA has deprecated nvprof and the Visual Profiler. It recommends Nsight Systems for CPU and GPU sampling and tracing, and Nsight Compute for kernel profiling. Each remaining tool answers a different question, and using the wrong one gives you a confident answer to a question you did not ask.

Question you are asking	Tool	What it gives you	What it cannot tell you
Which device in the fleet is underused or imbalanced?	DCGM profiling metrics	Low-overhead interval averages of engine, SM, tensor-pipe, device-memory, PCIe and NVLink activity	Which kernel, source line or instruction caused a value
Where does wall-clock time go in one job?	Nsight Systems (`nsys`)	A unified timeline of CPU, GPU, CUDA libraries, network communication and multi-node activity	Why a specific kernel is slow; for that you open the kernel in Nsight Compute
Why is this kernel slow?	Nsight Compute (`ncu`)	Throughput against theoretical peak, roofline position, warp stall sampling and source-level metrics	Any GPU or API trace; Nsight Systems covers that
How are collectives performing per communicator in production?	NCCL Inspector plugin	Algorithmic bandwidth, bus bandwidth, execution time and message size per collective	Actionable insight without further analysis of its high-volume raw records
Is the link hardware delivering expected bandwidth?	nvbandwidth, ib_write_bw, nccl-tests	Pairwise GPU bandwidth, link checks and bus bandwidth comparable with hardware peak	The larger collective-bandwidth variation seen in production training

NVIDIA’s DCGM documentation recommends starting with DCGM to locate an underused or imbalanced device, then switching to Nsight Systems or Nsight Compute when you need application-level attribution. These tools compete for the same hardware counters. Nsight Systems’ GPU Metrics feature cannot run at the same time as Nsight Compute or DCGM, and NVIDIA recommends pausing DCGM metrics collection while the developer tools are in use.

Nsight Systems: the timeline that shows where time goes

Nsight Systems needs no application changes to profile a run, but a few annotations make the result far easier to read. NVTX ranges added to the code show up on the CPU thread rows and are projected onto the GPU timeline, so you can see which GPU work each phase of your step launched. Collection can be limited to the interesting region with cudaProfilerStart()/cudaProfilerStop() or an NVTX capture range, which reduces the data you and the tools must process.

A typical capture for a PyTorch training step on Linux looks like this:

nsys profile --trace=cuda,nvtx,osrt,nccl \
  --gpu-metrics-devices=all \
  -o step_profile python train.py

With nccl in --trace, the default NCCL events recorded are API calls, copy-engine collectives, groups and GPU operations. The advanced NCCL tracing mode introduced in Nsight Systems 2025.6.1 needs NCCL 2.28 or later, and it gives more detail on GPU operations and runtime scheduling than the legacy NVTX-based trace. GPU Metrics sampling uses dedicated hardware to capture PCIe, NVLink and DRAM throughput, SM activity, tensor core activity and warp occupancy in a single pass with minimal overhead. Its default sampling rate is 10 kHz.

After the capture, post-collection recipes generate visualisations and summaries from the profile. One recipe lists periods of GPU inactivity with a probable cause for each, and another summarises NCCL operations per communicator and rank with counts and costs. For multi-node jobs, the nsys CLI can be prefixed to launchers such as srun or mpirun. Once the timeline points at a kernel, Nsight Systems can open that kernel directly in Nsight Compute.

Nsight Compute: explaining the kernel that dominates

Nsight Compute collects metrics in section sets, and the number and type of metrics chosen has a large effect on profiling overhead. The CLI collects the basic set unless you pass --set full or name specific sections. The Speed of Light section reports achieved compute and memory throughput as a percentage of the theoretical maximum, and the roofline chart shows whether the kernel sits in the memory-bound or compute-bound region and how far it is from the ceiling.

NVIDIA’s guidance is that the target is issue-slot utilisation, not driving every stall counter to zero. Stall samples are attributed to the instruction that is waiting, not the one that produced the data. A kernel dominated by long-scoreboard (L1TEX) stalls is memory-latency-bound, which is not the same as memory-bandwidth-bound. When bandwidth is not saturated, NVIDIA’s guide says to prefer latency-hiding changes such as unrolling or pipelining. Occupancy follows the same logic: higher occupancy does not always raise performance, but low occupancy always weakens latency hiding.

This is where best-practice dogma meets evidence. Nsight Compute counts divergent branch targets per source line, and divergence can cause warp stalls. “Avoid branches” only makes sense in that context, and the context comes from profiling. The rule engine’s speedup estimates come in global and local forms. Both are approximations, not measured results. NVIDIA’s profiling guide says the same: validate an optimisation against absolute workload duration, because utilisation percentages can move either way when total work changes. Our guide to profiling GPU kernels to find the real bottleneck goes further into memory-bound kernels.

Why Nsight Compute and Nsight Systems report different durations

Nsight Compute adjusts how metrics are collected to make them deterministic, so its kernel durations differ from an Nsight Systems timeline. Before comparing the two, check these four effects:

Clock control. Nsight Compute locks SM clocks by default and Nsight Systems does not. NVIDIA recommends locking clocks externally with nvidia-smi and running ncu --clock-control none when comparing.
Cache flushing. Nsight Compute flushes all GPU caches between replay passes, while Nsight Systems collects in one pass. For cache-sensitive workloads, --replay-mode application --cache-control none lets the application prime the caches itself.
Kernel serialisation. Nsight Compute serialises kernel launches unless a dedicated replay mode is used, so concurrent kernels report different durations in the two tools.
Multi-pass metric error. Not every metric fits in one pass. A ratio such as hit rate can carry significant error when its inputs come from different passes and the kernel runs for less than about 20 µs, too short to reach a steady state.

Other GPU engines sharing L2, DRAM, PCIe and NVLink can also distort the results for small kernels. The guide recommends profiling on a GPU with no active display and no other processes using it.

ERR_NVGPUCTRPERM: the Linux counter-permission failure

NVIDIA drivers require elevated permissions to read GPU performance counters by default. Without them, every Nsight Compute report contains only an error saying the user does not have permission to access NVIDIA GPU performance counters on the target device. There are two fixes. A system administrator can enable non-admin access to the counters, or the profiled application can be launched with sudo or as a user with the CAP_SYS_ADMIN capability. On shared clusters, the first option is an administrator’s policy decision, so ask for it before the profiling day rather than during it.

Why does GPU utilisation look high while throughput stays low?

A high utilisation number does not show that the GPU was actually computing. In DCGM’s SM activity metric, a warp waiting on memory still counts as active, and NVIDIA describes a value of 0.8 or more as necessary but not sufficient for effective use of the GPU. The DCGM_FI_DEV_GPU_UTIL field is only a high-level indicator of GPU activity. NVIDIA’s own fleet analysis planned to move from it to DCGM_FI_PROF_GR_ENGINE_ACTIVE for a more precise view of engine utilisation.

The evidence that separates busy from productive sits in three places: gaps on the Nsight Systems timeline, issue-slot use and stall mix in Nsight Compute, and the step duration itself. In ByteDance’s training cluster, Python garbage collection paused workers for hundreds of milliseconds, during which no new kernels could start. If you are profiling inference rather than training, see what profiling tools for AI inference measure and how to read their output.

Where is the time going once we go past 16 GPUs?

When scaling drops off past a given GPU count, the measured studies below each trace the loss to a different place. The fabric may be congested on shared links. The communication transport may be inefficient in software. The data pipeline may not be feeding the GPUs. Or compute imbalance may be leaving fast ranks waiting for slow ones. Per-rank, per-link measurement at each GPU count tells these apart.

TechnoLynx runs this diagnosis as a GPU Performance Engineering engagement. It profiles the run end to end at the GPU counts where scaling drops, instruments NCCL and InfiniBand per rank and per link, names the ceiling, and re-measures scaling efficiency after the fix, before anyone asks for more hardware or quota.

Is the InfiniBand fabric saturated, or is it a software-side bottleneck?

NCCL’s own troubleshooting guide starts by separating an NCCL performance problem from a configuration or hardware issue. If nvbandwidth reports lower than expected GPU-to-GPU bandwidth, the problem is in inter-GPU communication. If nvbandwidth and ib_write_bw match the hardware’s expected figures but NCCL still underperforms, the NCCL configuration is the likely culprit. Process placement belongs in the same check, because each rank should use CPU cores, host memory and a NIC close to its GPU.

Bus bandwidth is the number to compare against link peak. nccl-tests derives it from algorithm bandwidth so that the result can be compared with hardware peak whatever the rank count. For AllReduce, the correction is busbw = algbw × 2(n−1)/n. With 16 ranks that factor is 2 × 15/16 = 1.875, and with 64 ranks it is 2 × 63/64 ≈ 1.97.

“The network is the bottleneck” is a common belief, and it deserves a measured test. A measurement study of distributed training on AWS found only 60% scaling with 64 workers (eight servers with eight GPUs each) for VGG16. Yet the communication phase used no more than 32 Gbps of the 100 Gbps available per server. The authors traced the root cause to the network transport implementation, which could not use the available bandwidth, not to network speed. In that study, communication was the bottleneck while the network ran at low utilisation.

What is the per-GPU compute-to-communication ratio at 16, 32 and 64 GPUs?

A temporal-breakdown analysis splits GPU time into idle, compute and non-compute (communication and memory) categories. It counts communication that overlaps compute as compute time. Running that breakdown at each GPU count lets you compare the communication share across scales. For production runs, the NCCL Inspector plugin records every collective operation per communicator. The NIXT study measured its overhead at under 2% end to end on training workloads from 16 to 2,048 GPUs.

That same study compared microbenchmarks with production training. Mean bus bandwidth in nccl-tests agreed with production training to within 4–8%, but the variation was two to five times larger in production. The authors point to the ML framework calling NCCL as a source of that variation. In their Nemotron-4 15B runs, adding GPUs mainly increased the amount of AllGather communication rather than degrading the throughput of individual collectives.

Scheduler-level dashboards add per-job telemetry. CoreWeave’s Slurm job dashboards report NCCL bus bandwidth and latency by process-group size and identify the rank that fell furthest behind in each interval. NVIDIA researchers combined DCGM telemetry with Slurm job metadata and cut idle GPU waste from roughly 5.5% to about 1% across large clusters.

Is the data pipeline starving the GPUs?

A study of nine models on servers in a Microsoft production cluster found that in many cases DNN training time was dominated by data stall time, the time spent waiting for data to be fetched and preprocessed. On the Nsight Systems side, the file_access_sum recipe analyses file I/O patterns to find I/O bottlenecks.

Stragglers, synchronisation skew, and why the published studies disagree

The published measurements do not agree on which cause dominates. ByteDance’s five-month trace of jobs using at least 128 GPUs found that 42.5% of jobs were at least 10% slower because of stragglers. Computation slowdowns caused stragglers more often than communication did, mainly through uneven pipeline-stage partitioning, sequence-length imbalance and garbage-collection pauses. The authors attribute the small role of communication to ample network bandwidth, dedicated training clusters and in-house network tuning. The MegaScale work found specific hosts taking about 10% longer on the same forward computation. Removing them improved MFU by about 0.7%.

An empirical study across several production clusters reached a different conclusion. It found that topology, congestion, collective synchronisation and GPU locality frequently dominate performance once workloads move beyond a small number of nodes. It also found these effects are often invisible to standard profiling tools and get misdiagnosed as framework inefficiencies. That study describes synchronisation amplification, where small per-rank delays become cluster-wide idle time, and says it typically dominates once clusters exceed a few dozen nodes.

We read the two as compatible rather than contradictory. On ByteDance’s well-provisioned dedicated clusters, compute stragglers dominated. On oversubscribed fabrics, the cluster study reports contention causing throughput plateaus. Either way, the 2026 study’s advice holds: watch iteration-time variance and tail latency, not just mean throughput.

What scaling efficiency should we expect, and when do more GPUs stop paying?

Analytical models built on compute-to-communication ratios tend to treat the network as uniform and miss the congestion and synchronisation effects that dominate at scale. We expect the diminishing-returns curve to differ between model types. Designing a cluster topology that keeps near-linear scaling as it grows is a separate question from diagnosing this one, and it builds on this diagnosis.

The order we would follow before any request for more quota is:

Measure step time and scaling efficiency at two or three GPU counts that bracket the drop.
Check link hardware with nvbandwidth and ib_write_bw, and compare bus bandwidth with link peak.
Break each rank’s time into compute, communication and idle at each scale.
Look for GPU idle gaps and check whether file I/O lies behind them.
Localise stragglers per rank, and check variance as well as the mean.
Only if the run is genuinely compute-bound at scale does adding GPUs or quota follow from the evidence.

Is our scaling wall the network, the availability-zone split, or our all-reduce?

Test one layer at a time, keeping the rest of the setup stable. The node count alone will not tell you. TechnoLynx frames which layers could be the bottleneck before instrumenting anything. It then runs controlled experiments on the target hardware, varying interconnect, placement, scheduling and kernel behaviour one at a time, and hands over before-and-after measurements with the recommended change, as part of GPU Performance Engineering. These findings belong to the sharding strategy, topology and provider that were tested, and may not transfer elsewhere.

Scaling capped past eight GPUs on rented nodes split across availability zones

The cheapest experiment is to run the same job inside one availability zone and then split across two, with link hardware checked separately first. If nvbandwidth and ib_write_bw meet the hardware’s expected figures and NCCL still falls short, the communication pattern or NCCL configuration is the candidate. In hierarchical or oversubscribed fabrics, traffic can concentrate on shared links even when average utilisation looks low. Moving into one availability zone only helps if the fabric itself is not already the limit.

A sharp throughput drop from one node to four

When throughput falls the moment a job spans nodes, locate the stragglers before choosing between topology, scheduling and sharding. MegaScale used a per-machine heat map of computation latency, which showed a small fraction of machines running much slower and holding back the whole job. The NIXT study localised stragglers to specific ranks through spatial correlation of NCCL Inspector records. Uneven pipeline-stage partitioning is a common cause of stragglers. In a four-stage test job the ByteDance authors ran, the last stage, which carries the loss layer, was 2.07 times slower in forward computation than the average stage.

Drift or infrastructure on a shared inference cluster

Poor predictions in one market can come from data or concept drift, or from serving problems on a shared cluster. Separating the two means instrumenting model-quality metrics and latency or capacity metrics as independent series and seeing which one moved first. We read a drop in accuracy with flat latency and queue depth as pointing at the data. Timeouts, truncation or fallback paths during load peaks point at the infrastructure. To tie serving latency back to GPU work, see how kernel-level profiling of inference requests matches each measurement to a question.

A kernel overrunning its window on a shared node

Telling contention apart from an inefficient kernel takes two measurements: a per-kernel profile of the kernel in isolation, and a time series of device activity while other tenants run. Nsight Compute can profile work from all Multi-Process Service clients at once, but generally cannot isolate one client’s performance. Other engines and processes sharing L2, DRAM and PCIe also distort small-kernel results. If the isolated kernel meets its budget and the shared run does not, the cause is contention. Because other tenants’ load changes from night to night, a single profile is not enough to settle a contention diagnosis.

Which nodes in our rented pool produce different output for the same input?

TechnoLynx’s protocol rules out code and weights by checksum, then runs the same input, model and seed repeatedly on each node and diffs the outputs. It respects the live service’s SLA, adds stage-level timing across the node pool, and re-runs under controlled version and determinism settings to confirm the root cause, as GPU Performance Engineering work.

Each step of the decision narrows the cause:

If the weights and code checksums match, the model is not the cause.
If the recorded driver, CUDA and library versions differ between nodes, a version mismatch is the first suspect.
If the versions match and divergence disappears once deterministic kernels are enabled (for example torch.use_deterministic_algorithms(True) in PyTorch), the cause was numerical non-determinism.
If divergence persists even then, suspect execution order or scheduling.

We would run this testing on a shadow or replica path, not on the production serving stack. Some divergence may be floating-point variation within acceptable tolerance, not a bug. Cross-zone network variance can also look like compute divergence.

Is the slowdown past four nodes NVLink, the inter-node network, or the application?

Compute, intra-node NVLink transfer and inter-node network time can each be measured directly, so the answer comes from a profile rather than from blaming the application or the hardware. TechnoLynx runs this as GPU Performance Engineering work. It characterises utilisation against node count, captures Nsight and NCCL diagnostic traces across node counts and before and after a firmware change, and compares them with the pre-upgrade baseline to confirm the cause.

A scaling cliff as node count grows

Nsight Systems GPU Metrics report NVLink bytes received and transmitted as a percentage of the peak possible in each sample period. NIC metrics show busy and idle network time, and congestion notification packets and send waits help explain network latency. Quantum InfiniBand switch metrics can be placed on the same timeline. When this evidence clears both links, the next suspect is application configuration, such as a RELION or cryoSPARC job layout, or the scheduler’s placement. Scheduler contention and interconnect limits can occur together, so change one variable at a time.

A regression that appeared after a firmware upgrade

After a firmware change, we would first check whether NCCL still detects the same topology and picks the same algorithms. Setting NCCL_DEBUG=INFO with NCCL_DEBUG_SUBSYS set to include GRAPH (topology detection), TUNING (algorithm and protocol selection) and NET filters the log to exactly those subsystems. nvbandwidth then checks whether each GPU pair still reaches its expected PCIe or NVLink bandwidth. NCCL’s LL protocol choice for P2P operations depends on per-channel message size and channel count, not a strict cutoff. That is why the comparison needs to be against a pre-upgrade baseline rather than against expectation. Profiler overhead also matters at scale, which is one reason a low-overhead collector such as NCCL Inspector suits this kind of before-and-after comparison.

Will more GPUs help before we spend more of our funding on them?

We would add GPUs only if the profile shows the run is compute-bound. If communication, data loading or parallelism inefficiency is the limit, extra GPUs can leave throughput flat. TechnoLynx profiles the current pipeline on whatever cluster is available, models how each candidate fix would scale, and designs a small pilot that isolates the chosen fix before the larger spend, as GPU Performance Engineering work.

The decision follows the profile:

Compute-bound, near-linear on a small trial: more GPUs of the same type are justified.
Communication-dominated: better interconnect, or fewer but larger nodes, comes before raw GPU count.
Parallelism inefficiency: changing the sharding or pipeline split is usually cheaper than any purchase.

A pilot only predicts the full-scale run if it reproduces the production interconnect, data pipeline and topology, and bottlenecks can shift as scale grows. When the question is which device to buy, LynxBenchAI publishes how each device runs the public PyTorch stack that teams’ own models run on. Those results are broadly indicative for custom PyTorch models, though they do not measure your own application.

Who can profile our multi-GPU job and tell us what to fix first?

TechnoLynx profiles your training or inference run on your hardware and tells you what is capping it, whether that is the fabric, NCCL configuration, the data pipeline, stragglers or a single kernel, with before-and-after measurements for the fix. Building this capability in-house makes sense if you have engineers who already read Nsight and NCCL traces fluently and expect a steady flow of these problems. A specialist makes sense when the question is urgent, one-off, or crosses kernel, communication and cluster layers at once. When choosing a partner, look for one that will show you the raw traces and the measurement method, re-measure after every change, and hand the workflow over to your team. The work runs as GPU Performance Engineering and starts with a profiling run of the workload as it stands.

If your problem sits entirely inside one kernel and you have CUDA engineers, NVIDIA’s Nsight documentation and a profiling-first approach to improving GPU performance may be enough. For the in-house route, the trade-offs of staffing and build configuration for finding and fixing GPU bottlenecks are covered separately.

What one profile still cannot settle

A profile gives you a diagnosis for the scale, software versions and tenants present when it ran, and those all change. Even in the ByteDance traces, most steps showed a slowdown close to the job’s overall figure. That suggests persistent causes and makes sampling a few steps cost-effective. We would not assume the bottleneck fixed at 32 GPUs is still the one at 256. For teams running AI infrastructure and production SaaS, the practical consequence is to keep always-on DCGM and NCCL Inspector telemetry, both built for low overhead. Then the next scaling question starts from a baseline instead of a guess.

FAQ

Is NVLink scale up or scale out?

NVLink is mainly a scale-up interconnect between GPUs inside a node, while NICs such as InfiniBand carry scale-out traffic between nodes. Multi-node NVLink extends NVLink bandwidth between nodes, and nvbandwidth can measure it when built with multinode support.