A TOPS figure on a GPU or NPU spec sheet is a theoretical peak. It counts the trillions of operations per second the silicon could perform at one stated precision, under conditions no real workload meets. Delivered throughput is a different quantity. It belongs to the AI Executor: the device together with the backend, driver, framework and runtime it ran through. That is why two parts with the same TOPS rating can deliver very different throughput. A TOPS GPU rating gives you the compute ceiling, calculated from the hardware alone. It does not tell you how fast the whole stack runs.
The rest of this page explains the gap and shows where the usual “hardware versus software” framing stops working. It ends with what has to be written down next to any number so that someone else can still use it later.
What a TOPS figure on a GPU spec sheet counts
Vendors report peak arithmetic throughput in operations per second, counting one multiply-accumulate as two operations. Nearly all neural-network compute is multiply-add, so that peak describes an ideal rather than an observation. Dense TOPS reflects the number of multiply-accumulate units available at a specific precision format such as INT4, INT8 or FP16. Qualcomm says its TOPS figures usually refer to dense TOPS.
Three qualifiers change the number without changing the chip:
- Precision. NVIDIA’s Hopper documentation states that FP8 doubles throughput compared with FP16 or BF16.
- Dense or sparse. NVIDIA quotes Hopper Tensor Core throughput for both dense and sparse matrix maths.
- Operation type. The A100’s 312 teraflops apply to matrix multiplication on Tensor Cores. Horace He’s first-principles analysis puts other operations at most at about 19.5 teraflops.
Form factor matters too. The same GPU name can ship as different boards, and the 26Q3 results score them as separate devices. The NVIDIA H100 80GB HBM3 scores 1678.1 in inference and the NVIDIA H100 PCIe scores 1039.6 in the same category. Before comparing two TOPS figures, we would check that both state the same precision, the same dense or sparse basis and the same board.
Worked example: two 40-TOPS chips, one delivering more than twice the other
A CSEE explainer gives an illustrative comparison of two chips. Both are rated at 40 TOPS INT8.
| Property | Chip A | Chip B |
|---|---|---|
| Peak INT8 TOPS | 40 | 40 |
| Memory bandwidth | 120 GB/s | 40 GB/s |
| Mature software tools | Yes | Limited |
The effective figures are the peak multiplied by utilisation: 40 × 0.65 = 26 and 40 × 0.30 = 12. The label is the same, but in practice the two chips end up more than twice apart. The chips also differ in memory bandwidth, software tools, power and typical utilisation, and the TOPS label shows none of these.
Why delivered throughput falls below peak: the roofline
The roofline model shows where the bottleneck lies. A workload’s performance is capped either by peak memory bandwidth or by peak compute. Which cap applies depends on arithmetic intensity, the number of operations performed per byte moved from memory. Below the device’s ridge point the workload is memory-bound. Memory cannot feed the processor fast enough, so the compute units sit idle whatever the TOPS rating says. Williams, Waterman and Patterson introduced the model to relate processor performance to off-chip memory traffic.
NVIDIA’s matrix-multiplication guide gives a concrete case. On a V100 with FP16 Tensor Core inputs, the GPU’s ops:byte ratio is 138.9 FLOPS per byte. One example GEMM has an arithmetic intensity of 124.1, so it is memory-limited. Enlarging it to 8192 × 8192 × 8192 raises the intensity to 2730, and on the same GPU it becomes maths-limited. Only the size of the GEMM changed. The hardware stayed the same.
Published measurements show how large the gap gets:
- Jain et al. reported in 2019 that ResNet-50 on a V100 reached less than 25% of peak at interactive latencies, and that larger batches struggled to reach 40% of NVIDIA’s advertised 15.7 TFLOPS.
- A 2024 study of convnet efficiency ran an NVIDIA A5000 at FP16 and batch size 128. EfficientNet’s computational efficiency was 5% to 8%, while a model built on block-fusion kernels reached 47% to 55% on the same GPU.
- A 2025 roofline study of the Jetson Orin AGX found that achievable peak compute sits far below the theoretical specification.
Overhead is a third regime alongside compute and memory. It covers Python interpretation, framework dispatch and kernel launch time. When GPU operators are too small for the CPU to queue work ahead of them, the GPU spends most of its time waiting.
Software moves the number on unchanged silicon
A 2026 review of LLM accelerators reports that, in independent benchmarking, the AMD MI300X reached about 620 BF16 TFLOPS against a marketed 1,307. The review attributes the shortfall to the relative immaturity of AMD’s ROCm communication stack. It also argues that this gap between nominal and realised performance explains why the CUDA ecosystem remains the field’s centre of gravity.
Software can also raise the number. NVIDIA reports that its MLPerf Inference v3.0 submission improved H100 performance by up to 54% over its v2.1 submission six months earlier, and credits software optimisation for the gain. That is a vendor describing its own results, but the direction is what matters here: the H100 gained through software. A 2022 benchmark of inference frameworks found that, for the same network on the same hardware, the choice of framework can significantly change prediction speed, memory and compute use. NVIDIA’s own documentation notes that the requirements for using Tensor Cores depend on cuBLAS and cuDNN versions.
The wider argument is set out in why performance emerges from the hardware × software stack. In practice it means that changing the runtime changes the executor. Treating the inference engine as a first-class choice makes it part of the hardware decision.
Language models: memory bandwidth, not TOPS, sets the ceiling
For local language-model generation, memory bandwidth sets the speed ceiling, not TOPS. Each generated token requires reading roughly the whole model from memory, so single-stream speed is bounded by tokens per second ≈ memory bandwidth ÷ model size in memory. CSEE’s example is a 4 GB model on 60 GB/s memory: at most about 15 tokens per second, whether the chip is rated at 40 TOPS or 400. The 2026 accelerator review applies the same bound to an 8B model at INT4 on about 77 GB/s of LPDDR5X. It gets roughly 19 tokens per second, which is in the range observed on flagship handsets.
Prefill processes the prompt in parallel and is compute-bound. Decode generates one token at a time, uses each loaded weight once and leaves the arithmetic units idle. A spec sheet that headlines TOPS describes the phase that matters less for interactive serving.
Where does a hardware-versus-software framing stop being useful for reasoning about AI workload performance?
The framing stops being useful once you need to explain a measured number. Hardware sets the roofline and software decides where on it the workload lands, and you cannot read either one off without the other. “Specs on one side, code on the other” is still a useful way to list what a system contains. It fails as a way to predict performance, because the interactions between layers set the outcome.
Each of these shows the split breaking down:
- Memory-bound decode is a hardware limit reached through software. Even an optimised attention kernel such as FlashAttention-2 reaches only about 35% utilisation on the H100. Kernel design took FlashAttention from 25–40% of peak to 50–73% on the A100. The bandwidth limit comes from the hardware. How close a run gets to it depends on the software.
- Software techniques pay off only on hardware that can exploit them. The 2026 review states that each software technique changes arithmetic intensity and memory access, so its benefit appears only when the hardware is co-designed for it.
- Configuration moves the roofline itself. On Jetson Orin, each power mode produces a different roofline, and GPU frequency changes memory bandwidth as well as FLOP/s. Going from FP32 to FP16 doubled ResNet’s arithmetic intensity. The balance point moved by a similar amount, so the workload stayed memory-bound.
- Layer-by-layer reasoning misleads. The roofline overestimates attainable performance for a sequence that mixes memory-bound and compute-bound kernels. Fusing layers into one kernel changes their operational intensity.
None of this makes hardware choice irrelevant. Bandwidth, memory capacity and peak compute are real limits. The point is that you cannot reason about AI performance one layer at a time and then add the answers together.
Same GPU, different stack: what a side-by-side actually compares
When two machines share a GPU but differ in driver, framework, runtime or platform, their numbers describe two executors, not one device. The 26Q3 results show this directly. The NVIDIA GeForce RTX 3080 on native Linux scores 240.5 in inference and 64.8 in compute. The same card under WSL scores 231.7 in inference and 72.5 in compute. Neither platform leads across the board; each is ahead in a different category. The WSL executor scores higher on FP16 dense matrix multiplication (106.2 against 71.2), and the native executor scores higher on INT8 DistilGPT2 inference (235.2 against 190.1). The GeForce RTX 5090 on native Linux scores 985.8 in inference against 891.8 for the same card under WSL.
Precision also changes which results exist. The RTX 3080 executors have no FP8 inference result in 26Q3 (scored 0.0), while the RTX 4090 executor does. Even the silicon is not one fixed point. A 2022 study of large GPU clusters found performance variation averaging 8%, and reaching 22%, across GPUs of the identical SKU. PyTorch’s documentation states that results are not guaranteed to be reproducible across PyTorch releases or platforms. It also says cuDNN’s benchmark mode may pick different algorithms on later runs, even on the same machine.
A LynxBenchAI score never belongs to a device on its own: every result is bound to the device, the backend, and the driver, framework and runtime present on the machine. Binding the result this way is what makes the stack claim checkable rather than rhetorical. The 26Q3 results list the platform that produced each device’s numbers. Scores are comparable only within one release name: 26Q3 against 26Q3, never across release names.
What to record beside a TOPS or benchmark number
CSEE suggests asking for a figure in the form “___ TOPS, dense, INT8, NPU only, sustained”, and treating it as a marketing headline if any blank is missing. That covers the spec sheet. A measured number needs more. MIT’s Vivienne Sze argues that peak TOPS alone gives insufficient insight and that metrics must be reported together for fair evaluation. From v4.0, MLPerf Inference rules require submitters to describe the architecture and software behind their I/O bandwidth. The MLPerf paper concludes that the number of options at every level of the inference stack makes auditability necessary for reproducibility.
Diagnostic checklist for any number you intend to share:
| Record | Why it changes the number |
|---|---|
| Device and form factor | Board variants of one GPU score differently |
Backend (cuda, cuda via ROCm, xpu, CPU) |
Part of the executor the result is bound to |
| Driver, framework and runtime versions | Library versions affect which kernels run and whether results reproduce |
| OS platform (native Linux, WSL) | Same card, different scores in 26Q3 |
| Precision, dense or sparse | Peak and delivered figures both shift with format |
| Workload, batch size and scenario | Batch size controls utilisation; latency constraints cut throughput |
| Power mode and clock settings | Each power mode yields a different roofline |
| Release name, for published results | Comparability holds within a release only |
Batch size needs separate attention. Why batch size is part of a throughput figure explains how it changes what the number means.
Keeping optimisation effort from becoming the hidden variable
In cross-vendor comparisons, unbounded or opaque optimisation effort undermines comparability. MLPerf accepts any implementation that is mathematically equivalent to the reference model, and submitted results need optimisation to perform well. The results therefore reflect the optimisation work behind them, and that is not the same as how fast an executor runs a fixed model.
LynxBenchAI measures the fixed-model case by bounding the effort. The benchmarked models are prepared once, before any target device is known, and the identical prepared artefacts ship to every vendor’s hardware. Where two vendors’ kernels disagree about what is permissible, the stricter constraint applies to all of them rather than being set per vendor. Correctness criteria are enforced per test type and precision, and generic benchmark logic is kept separate from platform-specific adaptation. A workload that fails or times out is recorded as a zero, not dropped. In the 26Q3 tables, a 0.0 means that precision produced no result on that executor.
Measuring your own executor
The published 26Q3 results cover LynxBenchAI’s own benchmarked models on the 48 executors that ran them. They do not cover your model on your machine. To measure your own stack the same way, install the LynxBenchAI Personal Edition with pip install lynxbench-ai. It records the device, the backend and the software present on the machine alongside each score, so the result stays tied to your executor. One limit applies. Each test is one timed window after a discarded warm-up, not a median of repeats, and the run does not capture thermal state or settled clocks. If those conditions matter to you, log them yourself. The Press, Pro and Enterprise editions have not shipped and are contact-gated. More on the product is on the LynxBenchAI page.
Measuring the executor and changing it are different jobs. If the numbers show the software half holding the hardware back, porting inference across runtime and hardware without swapping the model is the next step. TechnoLynx does that kind of engineering through its GPU performance engineering work.
Any comparison you inherit leaves one question to answer: which executor produced each number, and under which release? If that is missing, the figure describes a machine you cannot rebuild.
