Whether a GPU bottleneck is hardware or software should be settled by profiling, not by argument. Profile the kernel and compare what it achieves against the sustained peak of each resource. If one resource, arithmetic throughput or memory bandwidth, is running within about 80% of its peak, that resource is the limiter. If no resource is near its ceiling, because warps are stalled waiting, the GPU is underfilled, or time goes to launch overhead and host-side scheduling, the limit is in software. A published comparison only deserves trust when every row says which of the two it was, using a method an outsider can re-run.

Hardware vs software bottleneck on a GPU: where the line falls

AMD’s HIP documentation sorts GPU kernel bottlenecks into three categories. Compute-bound kernels are limited by arithmetic throughput. Memory-bound kernels are limited by how fast data moves between HBM and on-chip caches or LDS. Overhead-bound kernels are limited by latency from host-side scheduling, kernel launch cost or small array operations. The first two count as hardware limits only when that resource is actually running near its sustained peak.

The authors of the Operator Profiler paper point out that framework profilers such as torch.profiler can tell you which operator is slow but not which hardware resource it is contending for. Without that, optimisation choices stay heuristic.

The HIP documentation says occupancy measures how fully a compute unit is loaded, not how efficiently it is used, and that high-performance MFMA-based GEMMs often run at low occupancy because a few warps are enough to saturate the pipelines. Low occupancy on its own therefore proves nothing about software quality.

How do I tell whether a performance gap is hardware-bound or software-bound?

Measure achieved throughput against each resource’s peak, then check whether the scheduler has eligible warps to issue. Best practices like “avoid branches” only make sense in the context that profiling reveals.

The roofline model gives a quick first filter. It plots a kernel by arithmetic intensity: points left of the ridge point are memory-bound and points right of it are compute-bound. Nsight Compute’s Speed of Light section reports, for compute and memory separately, the achieved percentage of each unit’s theoretical maximum, with breakdowns that name the largest contributor. NVIDIA’s Compute Triage Guide counts throughput below roughly 60% as low (a latency-risk regime) and above roughly 80% as near the limit, the level at which it treats a resource as the binding constraint.

When the top-level numbers are ambiguous, NVIDIA’s guide uses occupancy against pipeline utilisation as a 2x2 frame, with 60% as the practical threshold on both axes.

Decision table: occupancy vs utilisation

Achieved occupancy	SM pipeline utilisation	What the guide says it indicates	Disclosure tag we would assign
Low	Low	Underfill or a launch-limiting resource; check whether the SM can accept more warps	SW-bound (parallelism)
Low	High	Active warps already saturate the busy pipe; more warps may hurt	HW-bound (pipeline)
High	Low	Warps are resident but stalled: a latency-bound regime	SW-bound (latency hiding)
High	High	SM or memory saturated; reduce work or restructure the algorithm	HW-bound

The right-hand column is our mapping onto a publishable tag.

A first-pass label only identifies where to begin the investigation, not the conclusion. The MLC appendix on GPU kernel benchmarking works through a kernel with a Speed of Light reading of 77.74% compute, 38.71% memory and 12.88% DRAM, which looks compute-bound. The scheduler statistics then showed 1.44 active warps per scheduler but only 0.04 eligible, with no eligible warp in 96.11% of cycles. About 87% of warp-state cycles were Long Scoreboard waits on memory handled by L1TEX. As the authors note, calling that kernel compute-bound would have hidden the key clue. NVIDIA’s profiling guide makes the same distinction: a kernel dominated by Long Scoreboard stalls is memory-latency-bound, which is not necessarily memory-bandwidth-bound.

Is there a kernel-level profiling breakdown that separates hardware limits from software inefficiency?

Yes. Read the reports in order, so that each step points to the next section to collect. See also our piece on diagnosing GPU bottlenecks with kernel-level profiling.

1. Check the metrics can be trusted. Compare kernel duration in Nsight Compute against Nsight Systems timing. NVIDIA treats divergence beyond about 2x as a reason to investigate clock control, cache control or replay mode, and expects a release build compiled with line-table information.
2. Check the launch fills the GPU. Grid size and waves per SM show whether there is enough device-wide parallelism before anything else is worth reading.
3. Read Speed of Light. Compute is the busiest SM compute path, Memory the busiest memory-side path, and DRAM the external memory interface on its own. Each uses its own peak as denominator, so the three cannot be added together.
4. Read the scheduler. NVIDIA flags issue activity below 0.5 per active cycle as issue-starved and below 0.2 as severely starved. A low ratio of eligible to active warps is a strong latency signal.
5. Read warp stalls, but only if the scheduler is starved. Nsight Compute’s guidance is to focus on stalls only when schedulers fail to issue every cycle. Stall samples land on the instruction waiting for a result, not the one producing it.
6. Change one factor and re-measure without the profiler. If the counters move as predicted but unprofiled latency does not improve, another bottleneck has emerged or the hypothesis was incomplete.

The same order works on AMD hardware. rocprof’s documentation starts with application tracing to find the kernels that take the most time, then moves to counter collection for hardware metrics during kernel execution. Counters vary by GPU, so check what can be collected before profiling. rocprofv3 gathers achieved FLOPs, memory transactions and operational intensity for roofline analysis.

In a study on counter-guided autotuning, a kernel running in 10,475 µs showed texture cache utilisation at 9 out of 10 against FP utilisation at 3. After one tuning parameter was changed to improve locality, it ran in 1,531 µs, with texture utilisation down to 3 and FP up to 8. At operator level, the Operator Profiler authors found more than 85% of an LSTM’s runtime unattributed, coming from a cuDNN-opaque dispatch path. That identified the LSTM body as unreachable by any FX-level transformation, rather than a candidate for direct kernel optimisation. The fix that followed, re-dispatching to cuDNN, gave a 3.63x speedup in the attributed region.

Nsight Compute replays kernels across several passes because the GPU can only collect a limited number of hardware counters at once, and software-patched counters can skew hardware counter results. Nsight Compute locks SM clocks by default while Nsight Systems does not, so NVIDIA recommends locking clocks externally with nvidia-smi before comparing the two. The authors of the Valk study (arXiv, 2026) point out that profiling at a fixed base clock lowers instruction throughput while memory bandwidth stays the same, which distorts the roofline.

Is the gap between peak TOPS and realised throughput coming from bandwidth, sparsity or the compiler?

Before blaming any of them, measure against a denominator the chip can actually reach, such as max-achievable FLOPs. The HIP documentation names clock frequency, compute unit count, instruction throughput per clock and memory bandwidth as what sets theoretical peak. It attributes the shortfall in real applications to imperfect resource use, inefficient memory access, control-flow divergence, synchronisation and launch costs.

AMD’s ROCm blog says the ratio of max-achievable to peak FLOPs no longer measures software efficiency on modern hardware, and estimates the gap from preliminary testing at 44-70%. For most cases, AMD recommends max-achievable FLOPs as the optimisation target instead. Stas Bekman’s ml-engineering benchmarks separate a boost-clock burst ceiling (MAMF) from a sustained rate under full load (MSMF), and treat MSMF as the realistic bar for training throughput. On an H200, repeated runs gave MSMF of 743-746 TFLOPS against MAMF of 843-850. Input data matters too: high-entropy bit patterns draw more power, which can make the clock step down and slow an identical kernel. A re-run of the same GPU can move by up to 2.2%, so a smaller gain is noise.

The Valk authors measured a GH200 with 13.4x the memory bandwidth and 2.5x the instruction throughput of an L4, yet it ran TPC-H only 5.2x faster. The extra bandwidth made the kernels compute-bound, and they then failed to use the available instruction throughput. On the GH200, kernels ran at or above 75% of any of four resources only 19% of the time, compared with 49% on the L4.

For the compiler, AMD MI300X measurements attribute the gap between theoretical and observed FLOPs to two factors: software stack efficiency (compiler and kernel maturity) and dynamic frequency scaling. For sparse acceleration, treat it as a configuration fact: record whether the sparse path was exercised at all, and do not attribute a gap to missing sparsity without a run that toggles it.

What test harness lets you hold the compiler fixed and vary the workload, and the reverse?

The harness should record the whole software stack and change one factor at a time. The MLC appendix lists what to record: GPU, driver, CUDA, framework and compiler versions, dtype, shape, clock and power settings, plus library version, selected algorithm and workspace for any library baseline. Implementations are only comparable when they do the same work inside the same timing boundary. Alternate the measurement order so that no implementation always runs on a colder or hotter device, and use one cache policy throughout. Operator Profiler locks both captures to the same probed sustained clock, which makes their duration ratio clock-immune by construction. NVIDIA also asks for an identical profiling configuration between compared runs unless that configuration is the variable under test.

Experiment	Held fixed	Varied	What a change in the limiter tells you
Workload sweep	Compiler, driver, library, clocks	Shapes and input sizes across the ridge point	If the bound follows arithmetic intensity as the roofline predicts, the limit is hardware
Stack sweep	Workload, shapes, clocks, hardware	Driver, compiler or framework version	A throughput change beyond run-to-run noise points to the software stack
Input sweep	Everything above	Value distributions and sizes	An optimisation that helps one input can hurt another

Shape moves the classification on the same hardware. In the 2021 autotuning study, a 2048×2048 square matrix multiplication is compute-bound, a 128×128 one is latency- or strong-scaling-bound, and highly rectangular products are memory-bound because of low arithmetic intensity. CANN Bench (arXiv, 2026) sizes its test cases across three regimes: small cases expose tiling and launch overhead, large ones bandwidth-bound behaviour, and a medium band covers most production shapes.

The Milabench authors (arXiv, 2024) note that PyTorch version can have a major impact on performance. The MI300X study saw a B200’s measured memory bandwidth improve by 10% when benchmarks were re-run a month later, which the authors say points to ongoing software optimisation by NVIDIA. Each sweep also needs a correctness check against the same reference and tolerance. Numerical parity across backends is a separate question, with its own acceptance criteria, and should not be folded into the performance verdict.

How much of a CUDA vs ROCm gap comes from kernel maturity rather than the hardware?

In the MI300X measurements, both software efficiency and clock behaviour contribute, and separating them required recording the clock. Ambati and Diep (arXiv, 2025) measured the MI300X at an average of 45% of theoretical peak FLOPs across FP8, BF16 and FP16, while H100 and B200 sustained up to 93%. Recalculating peak at the clock the MI300X actually held (1,083-1,217 MHz against a 2,100 MHz boost), software efficiency came out at 80-85%. In FP16, for example, measured throughput was 573 TFLOPs against a calculated 674.

Memory and collective bandwidth also fall short of NVIDIA’s utilisation. The MI300X reached about 81% of its theoretical bandwidth against up to 90% on A100, H100 and H200. RCCL reached about 70% of theoretical peak interconnect bandwidth against about 85% for NCCL. End to end, the MI300X delivered 37-66% of H100/H200 LLM inference performance. Those runs used TensorRT-LLM on NVIDIA and vLLM on AMD, so they measure hardware and stack maturity together. The Milabench authors attribute the gap they saw between synthetic and real-world results mainly to software stack support, given more than a decade of CUDA kernel tuning.

Vendor-specific APIs are not inherently faster than standard ones. The same code performs essentially the same through either. What a vendor-specific API offers is access to lower-level features, and using that access well takes real commitment that does not always pay off.

When should a result say “as currently supported” rather than “intrinsic hardware capability”?

Call a result intrinsic only when the limiter is a hardware resource near its sustained peak, on a pinned and disclosed stack, and the classification holds across shapes. Everything else is “as currently supported”, stamped with the stack version and date. NVIDIA’s triage guide tells engineers to re-evaluate the dominant limiter on each new GPU generation, because memory bandwidth tends to grow faster than SM count. A kernel that saturated the bus on an A100 may fall well short on a B200 without modification. The Kernel Tuning Toolkit benchmarks showed performance is not portable in general, and code tuned for a different GPU ran at 31% of the best result in the worst case.

CANN Bench’s hardware-anchored limit depends only on an operator’s FLOP and byte counts and the chip’s published peak parameters, so it stays constant across software releases. A software baseline, by contrast, moves with CANN releases, which is why CANN Bench versions its baselines. SPEChpc’s rules state the reporting principle plainly: estimates must be clearly distinguished from results.

Can we publish an evidence pack alongside the comparison table?

Yes. Keep the table short and give each row a reference to an evidence pack that holds the profiling data behind its bound classification: utilisation, stall reasons and roofline position.

TechnoLynx builds that pack as an engagement. It designs the hardware-versus-software classification method and the profiling instrumentation behind it, builds a reproducible benchmark harness with documentation that an external auditor can re-run, and defines the annotation format that links each table row to its evidence. This is part of its AI Governance and Trust work, and the first step is agreeing the classification method for the rows you intend to publish.

SPEC’s run rules set out disclosure requirements of this kind. The organisation holds that a reported performance level means something only when the conditions it was observed under are stated, and requires a description of all performance-relevant conditions. Its SPEChpc 2021 rules expect later testers to be able to reproduce the claimed result within a small range allowing for run-to-run variation. A pack built on that model contains:

• hardware identity, driver, compiler, framework and library versions, plus dtype, shape, clock and power settings
• logs of the software stack, shapes, power and clocks, plus any system settings the logs do not capture
• the reported statistic, for example SPEChpc’s median of an odd number of runs or lower median of an even number
• per-row profiler output, which can be machine-readable: Operator Profiler’s profile.json records each operator’s attribution method and confidence tier alongside its metrics
• anything modelled rather than run, marked as an estimate

How to make such numbers stand up beyond a benchmark, across regulated pipelines, is covered in our piece on turning GPU performance numbers into auditable evidence.

An annotation format that keeps the table readable but honest

The tags below are our proposed vocabulary, not an industry norm. Use a fixed, small set of tags so readers can scan them quickly. Put each tag in its own narrow column, and keep the full traces in the pack.

Tag	Use when	Evidence the pack must show
HW-compute	A compute pipeline is within about 80% of its sustained peak	Pipeline utilisation, Speed of Light compute, roofline position
HW-memory	A memory tier (DRAM, L2 or L1) is the dominant near-peak unit	Tier throughput breakdown, hierarchical roofline
SW-bound	No resource near peak; issue-starved, launch-limited or overhead-bound	Scheduler eligibility, stall breakdown, timeline gaps
Mixed	Candidate limiters on both sides, or a classification that changes with shape	Both sets of evidence plus the one-factor experiment
Est.	The figure comes from a model, not a run	The model and its inputs

Three caveats belong in the pack itself. Bound classification can shift between GPU architectures, so every pack needs a hardware, version and date stamp. Forcing a mixed case into one tag misleads as much as having no tag, which is why “Mixed” is a tag in its own right. Third-party reproduction needs access to the same hardware: SPEChpc requires that a user can obtain the described system and reproduce the result. Not every auditor will have that access.

Who can help us prove where our GPU bottlenecks come from?

TechnoLynx profiles your kernels, classifies each limiter as hardware-bound, software-bound or mixed with counter evidence, and builds the reproducible harness behind the verdict through its GPU Performance Engineering work. On whether to do this in-house or bring in a specialist: you need outside help mainly when the classification has to convince people outside your team, or when nobody internally has time on the target hardware to work through the profiling sequence properly. When choosing a partner, ask for a sample evidence pack and check that it pins versions and clocks and labels its estimates. If the real question is which hardware to buy, LynxBenchAI’s published results show how each device runs the public PyTorch stack your own models run on, so they are broadly indicative for custom PyTorch models. LynxBenchAI does not run or measure your own workload.

This engagement does not suit a team that wants one headline number with no audience to convince. Other realistic routes are an in-house performance engineer with dedicated time on the target hardware, or the GPU vendor’s support for questions about its own stack, bearing in mind that a vendor is not a neutral classifier of its own results.

Counter names drift, and so do the classifications built on them

The counter names that classifications depend on change between GPU generations. The authors of the 2021 autotuning study note that NVIDIA’s performance counters have been changed or extended several times as GPUs evolved. Operator Profiler needed name fallbacks for seven of its twenty counters because some were renamed in Blackwell silicon. A tag like “HW-memory” stays comparable across vendors and generations only as long as the underlying counters are mapped between them. That mapping belongs in the pack, as a judgement open to challenge rather than a measurement.

Sources

1. Nsight Compute Profiling Guide, docs.nvidia.com
2. Nsight Compute – Compute Triage Guide, docs.nvidia.com
3. Understanding GPU performance — HIP 7.2.53211 Documentation, rocm.docs.amd.com
4. Hardware-Attributed Operator Profiling for PyTorch, arxiv.org
5. Measuring and Analyzing GPU Kernel Performance — Modern GPU Programming For MLSys, mlc.ai
6. Using rocprof — ROCProfiler 2.0.0 Documentation, rocm.docs.amd.com
7. Using hardware performance counters to speed up autotuning convergence on GPUs, arxiv.org
8. Over the Memory Wall, Into the Instruction Wall: The New Bottleneck in GPU Data Processing, arxiv.org
9. Understanding Peak, Max-Achievable & Delivered FLOPs, Part 1 (AMD ROCm Blogs), rocm.blogs.amd.com
10. ml-engineering: Maximum Achievable Matmul FLOPS (MAMF) finder benchmark, github.com
11. AMD MI300X GPU Performance Analysis Chandrish Ambati and Trung Diep, arxiv.org
12. CANN Bench: Benchmarking Agent Generated Kernels against Real NPU and Algorithmic Limits, arxiv.org
13. Introducing Milabench: Benchmarking Accelerators for AI, arxiv.org
14. A Benchmark Set of Highly-efficient CUDA and OpenCL Kernels and its Dynamic Autotuning with Kernel Tuning Toolkit, arxiv.org
15. SPEChpc 2021 Run & Reporting Rules, spec.org
16. SPEC CPU®2017 Run and Reporting Rules ..., spec.org