You’re comparing two GPUs for an inference cluster
The spec sheets are open side by side. One card advertises higher peak TFLOPS, the other has wider memory bandwidth. The numbers feel decisive — concrete, comparable, clearly pointing to a winner. You could make a spreadsheet, sort by the metric that matters most, and call it done.
We’ve watched this process play out many times, and the pattern is remarkably consistent: the spreadsheet makes the decision feel safe, and then deployment tells a different story. Not because the spec sheets lied, but because they were answering a question nobody actually asked.
A spec sheet describes the theoretical ceiling of a component in isolation. AI performance lives in the gap between that ceiling and what actually happens when a real workload executes through a real software stack, on a real system, under sustained load. That gap is where most of the interesting — and most of the expensive — surprises live.
Theoretical limits and executed behavior are different things
A GPU spec sheet is a statement about capability under idealized conditions: maximum theoretical compute throughput, peak memory bandwidth, supported data types, clock domains and power envelopes. These values define the outer boundary of what the hardware could do in carefully constructed, short-duration scenarios.
What they don’t capture is the interaction between the workload’s structure, the framework that lowers it to device operations, the runtime that schedules and synchronizes those operations, and the physical system that sustains the whole thing over time. Spec sheets describe the envelope; AI workloads live inside it, usually far from the boundary, and rarely at the same point for different models or configurations.
This distinction matters because people routinely treat peak metrics as predictions. When someone says “this GPU does 300 TFLOPS and that one does 200, so the first is 1.5× faster,” they have made at least four assumptions — about the workload being compute-bound, about the software stack hitting optimal paths, about data movement keeping pace, and about sustained thermal and power behavior — without examining any of them. In our experience, at least one of those assumptions breaks in every real deployment. Often several break at once.
How is a peak figure actually derived?
A theoretical FLOPS number is arithmetic, not measurement. Take the number of arithmetic units of the relevant type, multiply by the operations each retires per clock, multiply by the boost clock, and you have the headline figure. Peak memory bandwidth is derived the same way: bus width multiplied by effective transfer rate. Neither calculation contains a model, a batch size, a precision choice, or a duration. That is why the figure is stable across every workload and useless for distinguishing between them — it is a property of the silicon’s specification, not of anything the silicon did.
What an AI workload achieves sits below that ceiling, and the distance is not a constant. It widens with memory-boundedness, with kernels that map imperfectly onto the hardware, with launch and synchronization overhead, and with a sustained clock regime under the boost figure. We have seen well-tuned workloads land a long way under the headline number and poorly matched ones land far further; quoting a single universal efficiency percentage would misrepresent an observed range as a benchmark. The useful move is to name which of those factors dominates in your environment.
Peak FLOPS: the metric that misleads most often
FLOPS are the metric everyone reaches for first, probably because they look like the most direct proxy for “raw speed.” For AI workloads, treating peak FLOPS as a performance predictor only works if all of the following hold simultaneously: the workload is compute-bound, the compute units are saturated, data movement keeps up with compute demand, the software stack exploits the hardware’s fast execution paths, and the workload stays in a stable regime over time rather than bouncing between phases.
When a transformer inference workload is memory-bandwidth-bound — which it frequently is for autoregressive decoding — more FLOPS buys you nothing. The execution spends its time waiting on memory, not on arithmetic. Conversely, a workload that is compute-bound might still not saturate the device if the kernel doesn’t map well onto the hardware, or if synchronization and launch overhead eat into useful cycles. The relationship between the peak number on the spec sheet and the achieved throughput is contingent on so many intermediate factors that treating one as a proxy for the other is, in practice, a bet you’re making without seeing the odds.
A second, quieter trap is conflating the units. FLOPS (floating-point operations per second) is a rate; FLOPs (floating-point operations) is a count of work in a model or a layer; and TOPS describes integer or low-precision operations per second, typically INT8. A spec sheet that advertises a big TOPS figure and a comparison framed in FLOPS are not measuring the same thing — they live in different precision regimes and answer different questions.
Why a TOPS figure cannot be converted into FLOPS
Vendors report TOPS for NPUs and accelerators because those parts are built around low-precision integer arithmetic, and the number is derived the same way any peak figure is: unit count × operations per clock × clock. There is no conversion factor that turns it into a comparable FP16 or FP32 FLOPS figure, for two reasons. First, the operations counted are not the same operations — an INT8 multiply-accumulate and an FP16 one are different work on different units. Second, whether your model can even run in the precision the TOPS figure assumes is a property of the model and its accuracy requirements, not of the accelerator. A part rated in INT8 TOPS may have a far lower floating-point ceiling, or no fast floating-point path at all. Comparing across the two units inflates one device or deflates another for reasons unrelated to what your model will run, which is why conflating the three makes spec-sheet comparisons misleading before the workload is even considered.
Memory bandwidth has the same problem, just dressed differently
Memory bandwidth is often treated as the more “realistic” spec, especially by people who’ve already been burned by FLOPS comparisons. And it’s true that bandwidth matters more than peak FLOPS for many inference workloads — but it matters in context, not as an absolute.
Effective memory throughput depends on access patterns, operator fusion, cache hierarchy behavior, and runtime scheduling decisions. Two GPUs with similar advertised HBM bandwidth can deliver very different effective throughput depending on how the software stack organizes memory accesses. A PyTorch model with one attention implementation might achieve a large share of theoretical bandwidth; switch to a different kernel (say, FlashAttention versus a naive implementation) and the effective bandwidth utilization changes substantially, even on the same hardware with the same advertised spec.
Bandwidth is not consumed directly by models — it’s mediated by execution. And that mediation is where the divergence lives.
The peak-vs-sustained mismatch
Spec sheets quietly blend two different regimes: burst behavior and sustained behavior. Boost clocks, peak throughput numbers, and turbo specifications describe what the hardware can reach for brief windows under favorable conditions.
AI workloads are rarely brief. Training runs last hours to weeks; inference services run continuously under variable traffic. Under sustained load, GPUs settle into operating regimes defined by power limits, thermal constraints, and clock states that can sit meaningfully below the advertised peak. If you sized your capacity plan around the boost-clock number, you may find the system delivering 15–25% less sustained throughput than you expected (an observed range across the deployments we have worked through, not a published benchmark), with no defect present — just physics doing what physics does.
We pay close attention to this distinction because it’s one of the most common sources of “the benchmarks said it would be faster” complaints. The benchmark was probably correct for the regime it measured; it just measured a regime the production system never stays in. That is also why a measurement has to declare what it observed rather than what it implies: a timed continuous window after a discarded warm-up is a defensible statement, whereas “the settled thermal state of the card” is a claim the run did not make.
What actually determines AI performance?
Once you accept that spec sheets describe limits rather than outcomes, the natural question is: what does determine performance? The honest answer is that it’s the interaction between hardware, software stack, and workload — operating as a coupled system over time.
How spec-sheet metrics relate to real performance
| Spec-sheet metric | What it promises | Why it fails as a predictor |
|---|---|---|
| Peak TFLOPS | Raw computational speed | Only relevant if the workload is compute-bound, compute units are saturated, and the software stack hits optimal execution paths |
| TOPS (INT8) | Accelerator throughput | Counts different operations in a different precision regime; not convertible to a floating-point figure, and assumes your model runs in that precision |
| Memory bandwidth | Data throughput capacity | Effective bandwidth depends on access patterns, operator fusion, cache behavior, and runtime scheduling — not the advertised HBM number |
| Boost clocks / TDP | Sustained operating speed | Describes transient burst behavior; sustained AI workloads settle into lower thermal and power-limited regimes |
The hardware provides capability and constraints. The software stack (drivers, runtime, framework, kernels) determines which execution paths are taken and how efficiently the hardware is used. The workload determines what gets stressed, for how long, and in what pattern. None of these are separable in the outcome. You can’t point at the GPU and say “that’s where the performance lives,” because a different framework version, a different kernel library, or a different batch configuration can move the bottleneck to a completely different subsystem.
This is why benchmarks measure execution, not hardware — and it’s why reducing GPU performance to a single number loses the information you actually need to make a decision.
When a specification-based estimate is all you have
Sometimes execution isn’t available. The part hasn’t shipped, the procurement window closes before an evaluation machine can be booked, or the cloud instance type you’d need isn’t in your region. Estimating from specifications is the legitimate fallback in that situation, and it has a defensible scope: it can bound an upper limit, it can rule a part out when even the theoretical ceiling is below what you need, and it can rank two parts that differ by a wide margin on the axis your workload is known to be constrained by.
Where it must stop is anywhere the estimate would be quoted as a result. A specification-derived figure cannot support a throughput commitment, a capacity plan, a cost-per-token model, or a claim that one part outperforms another when the margin is narrow. The honest form is explicit: “estimated from published specifications; execution not performed.” Empirical execution of the real software workload remains the reference standard, and the estimate is the placeholder that gets replaced the moment a machine is available.
Better questions than “which GPU has better specs?”
The fix is not to find a more clever single metric. It’s to change the shape of the question. Instead of “which GPU has better specs?”, the questions that actually survive contact with deployment are:
What is the workload actually doing — is it compute-bound, memory-bound, or limited by something outside the device entirely? How does behavior change under sustained load versus the first few minutes? What software stack is being used, and does it exploit the hardware’s strengths or work around its limitations? What assumptions are embedded in the comparison, and are those assumptions true in your environment?
Performance conclusions that don’t state their assumptions aren’t conclusions — they’re guesses wearing a lab coat. Spec sheets make it easy to skip the assumptions, which is precisely why they keep leading to surprise.
The uncomfortable implication
None of this means spec sheets are useless, or that hardware selection doesn’t matter. Both obviously do. The point is narrower and harder to dodge: spec sheets are not performance measurements, and treating them as if they were is one of the most expensive mistakes teams make in AI infrastructure decisions.
Real performance is an execution property, not a static attribute. If you care about what your system will actually do — in production, under load, over time — the spec sheet is where the conversation starts, not where it ends.
The argument is also testable rather than rhetorical now. LynxBenchAI ships a free Personal Edition: pip install lynxbench-ai on Python 3.11 or later, on Linux or on Windows via WSL2, runs a fixed model catalogue on the machine in front of you and reports what that machine achieved, typically in 15–30 minutes. Every result carries the name of the release that produced it — currently 26Q3 — and is read only against results from the same release name, because a number produced under one catalogue and one stack is not comparable to one produced under another.
Frequently Asked Questions
Why do GPU spec sheets fail to predict AI workload performance?
Spec sheets describe theoretical ceilings under idealized, short-duration conditions — not what executes when a real model runs through a real software stack under sustained load. They omit the interactions between framework, runtime, kernels, memory access patterns, and thermal behavior that actually shape achieved performance. Treating them as predictors silently bundles four or five assumptions about the workload that almost never all hold at once.
How is a peak figure such as theoretical FLOPS or memory bandwidth derived, and how far below it does the performance an AI workload actually achieves typically sit?
A peak figure is arithmetic: unit count × operations per clock × boost clock for FLOPS, bus width × transfer rate for bandwidth. Neither calculation contains a model, a precision, or a duration, so the number cannot distinguish between workloads. What a workload achieves sits below that ceiling by a margin that varies with memory-boundedness, kernel mapping, scheduling overhead, and the sustained clock regime — an observed range in our engagements rather than a fixed published percentage.
Why can two GPUs with very similar spec sheets behave very differently on the same model?
Effective throughput depends on how the software stack organizes memory accesses, fuses operators, and schedules kernels — and those interactions land differently on two devices even when their advertised numbers look close. One stack may hit fast paths the other doesn’t; one device may hold its clocks while the other settles lower under sustained load. The spec-sheet similarity hides architectural and software-path differences that only show up under execution.
Which interactions between compute, memory, scheduling, and software shape the gap between theoretical peak and observed performance?
Compute saturation depends on whether kernels map well onto the hardware and avoid launch and synchronization overhead. Memory behavior is mediated by access patterns, operator fusion, and cache hierarchy decisions made by the runtime. Scheduling determines whether compute and data movement overlap or stall each other, and the software stack — drivers, runtime, framework, kernel libraries — decides which execution paths are taken at all. None of these are separable from the others in the final number.
What is the practical difference between FLOPS, FLOPs, and TOPS, and why does conflating them make spec-sheet comparisons misleading for AI workloads?
FLOPS is a rate — floating-point operations per second — while FLOPs (lowercase s) is a count of operations in a model or layer, and TOPS describes integer or low-precision operations per second, usually INT8. A rate and a count answer different questions, and a TOPS figure lives in a different precision regime than an FP16 or FP32 FLOPS figure. Comparing a device advertised in TOPS against one framed in FLOPS distorts the comparison before workload behavior is even considered.
What is TOPS as vendors report it for NPUs and accelerators, and why can a TOPS figure not be converted into a comparable FLOPS figure for an AI workload?
Vendors derive TOPS the same way as any peak figure — unit count × operations per clock × clock — but over integer or low-precision units, typically INT8. There is no conversion factor to floating point, because the operations counted are different work on different hardware paths, and because whether your model can run in that precision at acceptable accuracy is a property of the model, not the accelerator. A part with a large INT8 TOPS rating may have a much lower floating-point ceiling.
When a specification-based estimate is the only option available because execution is not feasible, what does that estimate legitimately support and where must it stop?
It can bound an upper limit, rule out a part whose theoretical ceiling is already below requirement, and separate two parts that differ by a wide margin on the axis the workload is known to be constrained by. It must stop short of throughput commitments, capacity plans, cost models, and narrow-margin orderings. Label it explicitly as estimated from published specifications with execution not performed, and replace it with a measured result as soon as a machine is available.
Methodology anchor — workload selection is the K1 primitive
This hub owns one decision in the LynxBenchAI methodology graph: what makes a workload benchmark-worthy in the first place. Spec sheets do not predict AI performance because they cannot specify a workload — and a workload that is not specified cannot be a benchmark, only an anecdote with decimal places. K1’s job is to push every downstream benchmark question through a workload gate before it is allowed to become a measurement: which model, at which input shape, in which precision, over which window, against which accuracy criterion. The hubs that follow (executor specification, sustained-load measurement, precision regimes, procurement decisions) all assume a workload that K1 has already declared admissible.
On the applied side, that same workload-first instinct decides whether a deployment leans on a public leaderboard or builds its own evaluation — the practical shape of the K1 question, worked through in public leaderboards versus task-specific evals. So the right question to put to any benchmark cited in your environment is the K1 one: which workload was measured, under which release name, and is it the workload you intend to deploy?
