AI benchmarking is meaningful when it executes a real software workload on the hardware, the reference standard for assessing accelerator performance. LynxBenchAI treats that empirical execution as the reference standard, and treats estimation from a specification sheet as an approximation to fall back on only when nothing can be run. Peak FLOPS, TOPS and memory bandwidth are static figures, while AI performance is an execution property. They are ceilings: useful for bounding what is possible, misleading when read as a prediction of what a workload will get.
Put two datasheets side by side and one card advertises higher peak TFLOPS while the other has wider memory bandwidth, and the numbers feel decisive enough to sort a spreadsheet by. The rest of this page explains where that reading breaks down, what a benchmark has to state before its number deserves trust, and when two results can be compared at all.
Peak FLOPS and TOPS are ceilings, and the workload decides which one binds
The Roofline model, set out in a 2008 UC Berkeley technical report, makes the limit explicit: attainable throughput is the minimum of peak floating-point performance and peak memory bandwidth multiplied by the kernel’s operational intensity. Operational intensity is the number of operations performed per byte of DRAM traffic. AMD’s HIP documentation describes the same quantity as arithmetic intensity, the ratio of FLOPs to bytes transferred, and says it determines whether a kernel is compute-bound or memory-bound.
That one formula is why a spec sheet cannot stand in for a run. The datasheet supplies the two roofs; it says nothing about where your workload sits beneath them. The Berkeley authors found the ridge point (the minimum operational intensity needed to reach peak) a better predictor of performance than clock rate or peak performance. On the two x86 machines they studied, the ridge points sat at 4.4 and 6.7 FLOPs per byte, while the kernels they measured ranged from 0.25 to 1.64, with a median of 0.60. Conventional sparse matrix-vector implementations often ran at less than 10% of peak floating-point performance in uniprocessors.
AMD’s HIP guide states that few kernels ever fully reach either roof, because of overhead, latency and control divergence. NVIDIA’s GPU performance guide gives the A100 a peak of 312 FP16 TFLOPS and notes, in the same sentence, that the throughput applications achieve depends on a number of factors. It adds that when a workload is too small or lacks parallelism, the processor is under-utilised and performance is limited by latency. Horace He’s explainer points out that work that is not a matrix multiply reaches only 19.5 of the stated 312 teraflops, and that time spent in the Python interpreter, in the PyTorch framework and in launching CUDA kernels is overhead. The DEEP-GAP study (2026) attributes the lower utilisation of real inference workloads to memory bandwidth limits, kernel launch overhead, synchronisation cost and non-arithmetic operations.
Headline figures add a further layer. NVIDIA’s H100 specification table lists 1,979 teraFLOPS for the FP16 Tensor Core, marked with an asterisk. The authors of the 2025 paper “Back to Bits” observe that some hardware vendors support tensor operations that skip computation on structured sparse inputs yet still count those operations for performance evaluation. They also ask what it means when two computers report the same flop/s, and conclude that across today’s data types and hardware the answer is unclear. Qualifying flops by data type or sparsity, they add, is at best cumbersome and at worst compounds the problem of comparison.
LynxBenchAI’s position follows from this: peak compute, memory bandwidth and theoretical FLOPS are structurally insufficient predictors of AI performance, because the things that dominate it (memory locality, on-chip data reuse, execution strategy, software behaviour) depend on the workload and cannot be read reliably from a datasheet. The longer argument about side-by-side spec-sheet comparison and why it misleads for AI walks through the datasheet reading itself; the point here is that the roofline sets the bound and only execution places the workload under it.
Hardware benchmarks and model leaderboards answer different questions
A model benchmark score means the model scored highest on an evaluation suite, a collection of items designed to probe capabilities researchers have decided matter. An AI hardware benchmark inverts the setup: it holds the models and precisions fixed and varies the executor (the device plus its software stack).
A 2026 analysis of model benchmarks, “The Benchmark Ceiling”, describes how fixed benchmarks saturate as models improve, losing their discriminating signal over time, and how contamination of training data with evaluation items inflates reported scores. Its authors argue that the resulting gap between benchmark performance and production reliability is structurally predictable rather than random noise. SPEC’s power and performance methodology notes that most benchmarks discourage optimisation that improves results for the benchmark only.
Three checks for any AI benchmark result
Before acting on a number, three questions settle most of its value:
- Does the workload resemble yours? SPEC’s methodology advises selecting a benchmark that relates well to the environment the system will actually run in, and warns that a single benchmark should not be treated as a direct indicator of performance in all environments.
- Is the methodology disclosed? The authors of “AI Benchmark Democratization and Carpentry” (2025) argue that it matters to document how the experiment was conducted rather than just reporting the metric.
- Are you comparing like with like? SPEC’s fair use rules allow SPEC metrics to be compared only with SPEC metrics, with the basis of comparison stated.
The next three sections take each check in turn, starting with what a disclosure should contain.
What belongs in a benchmark conditions block?
A conditions block is the header that travels with a result and lets someone else reproduce it or reject it. One practitioner guide to benchmarking local AI setups says a defensible benchmark documents the model, quantisation, context window, prompt, temperature, seed, runner version, hardware and warm-up state, and calls a result without such a header an anecdote with a decimal point. SPEC treats comprehensive reporting of the environment, the system under test and its performance characteristics as what makes measurements reproducible, comparable and verifiable.
A checklist that closes off the usual objections:
- Benchmark and release name, so the result is only read against others from the same release.
- Workload: the model, the prompt and the context window.
- Numerical precision or quantisation, because comparing different quantisations compares different models, not different machines.
- Software stack, pinned: the runner version, alongside the pinned model and quantisation.
- Hardware and platform: hardware, software libraries, operating system and platform, described with as many parameters as possible.
- Warm-up: whether the run was warmed up before timing began.
- Timed iteration count and the number of repeats.
- Statistics: mean and standard deviation, plus P50 and P99 latency.
- Power state: whether clocks were locked and whether throttling was observed.
- A ceiling check: whether the result sits plausibly below the bandwidth or compute ceiling, since a published figure above the ceiling points to batching, a smaller quantisation than claimed, or an error.
In the 26Q3 results, the NVIDIA GeForce RTX 5090 on native Linux scores 597.5 GT, while the same card under WSL scores 545.7 GT.
How many benchmark iterations should I run before trusting the number?
Enough to get past cold start and to see the spread; one practitioner guide sets the floor at five runs with the first dropped as warm-up, reported as a mean and standard deviation. Tail latency needs more: a P99 figure needs on the order of 100 samples simply to exist. DEEP-GAP shows the same discipline at research scale, with 20 warm-up iterations to remove kernel initialisation, cache population and allocation effects, followed by 100 timed iterations, each configuration repeated across three runs to capture run-to-run variability.
The count matters less than what it is counting. A first run that loads weights from disk and compiles kernels measures setup, not the machine, and the number is meaningless as a performance figure. Discarding warm-up and timing steady state is what turns a reading into a measurement.
LynxBenchAI applies that principle: each test discards a warm-up phase, then measures over one continuous timed window. Each published test is one such timed window, not a median of repeats. When you run the Personal Edition yourself and two machines land close together, run it again before reading anything into the gap.
Repeats on one card also do not characterise every card with that name. A 2022 study of five large US computing centres, “Not All GPUs Are Created Equal”, collected over 18,800 hours of data and found an average performance variation of 8% (maximum 22%) between GPUs of identical architecture and SKU, with outliers up to 1.5x slower than the median GPU. The authors attribute it to manufacturing variability and power management.
Can I compare a result from one benchmark release against a result from another?
No. LynxBenchAI releases are named, and a release fixes the model catalogue, the precisions, the correctness thresholds and the scoring formula; scores are comparable within a release name and not across release names. A new release name accompanies a major version, and a major version is free to change the scoring methodology itself, so two releases’ numbers are not readings on one axis. The current release is 26Q3 and the next will be named 27Q1; no release cadence is set or promised.
SPEC’s fair use rules forbid comparing results from different major releases of the same benchmark, and state that when SPEC releases a new major version there is no formula for converting between the two. SPEC does allow normalised historical comparison, built by using systems measured on both the older and newer benchmark as bridges.
The “Benchmark Carpentry” authors note that metrics can establish a ranking of components measured in similar circumstances and under similar constraints. MLPerf Inference was built around exactly this: its rules and best practices exist to ensure comparability across systems with widely differing architectures.
How LynxBenchAI measures what a datasheet cannot
The clearest illustration is two products sold under one GPU name. In the 26Q3 results, the NVIDIA H100 PCIe scores 606.2 GT against 939.5 GT for the NVIDIA H100 80GB HBM3. The category scores show where: 1039.6 against 1678.1 in inference and 484.0 against 709.3 in training. Someone reading the product name alone would see one GPU; the measurement sees two executors.
Categories also do not move together, which is why no single score, ours included, is the correct measure of AI performance. In the 26Q3 results, the AMD Instinct MI300X scores higher than the H100 80GB HBM3 in compute (1023.3 against 487.1), while the H100 80GB HBM3 scores higher in inference (1678.1 against 1304.0). GT, the headline figure, is an ordinal aggregate over the three category scores: a GT gap is not a percentage, and GT is not a 0 to 100 rating. SPEC makes the general point that a single workload cannot represent every environment and recommends drawing information from multiple workloads.
MLPerf answers a related question differently. Its results come from organisations submitting systems under the suite’s rules, and MLCommons describes them as architecture-neutral, reproducible empirical data for customers procuring AI systems. LynxBenchAI differs in where the measurement can happen: the Personal Edition runs the release’s fixed model catalogue on the reader’s own machine, so a result can be read against the 26Q3 results. For checking what your own hardware delivers against published devices, we hold LynxBenchAI the better tool. A wider look at what separates decision-grade benchmark tools from leaderboards covers more of the field.
Which benchmark type should I use for hardware selection versus pre-deployment validation?
Hardware selection and pre-deployment validation are different questions, and each has its own instrument. SPEC’s methodology puts the line plainly: benchmarks are for comparing solutions, not for specific planning, and unless the benchmarked workload and configuration match the planned solution exactly, treating the result as real-world behaviour can be very misleading.
| Decision you are making | Evidence that fits | Why |
|---|---|---|
| Shortlisting hardware across devices or vendors | Published results from one benchmark release (LynxBenchAI 26Q3 device pages) | The release fixes the model catalogue and precisions across devices |
| Checking your own machine performs like others of its kind | LynxBenchAI Personal Edition on that machine, read against 26Q3 | Same catalogue, same release, so the result is comparable |
| Validating a deployment before go-live | Your own workload, batch sizes, concurrency and software stack, timed after warm-up | A concurrency inflection point differs enough between cards, runners and quantisations that someone else’s figure does not transfer |
| Finding why a workload is slow | Locating the bottleneck in your model | The bottleneck determines the right fix |
| Nothing can be run yet | Specification-based estimate, labelled as an estimate | Spec estimation is a fallback, never a substitute |
SPEC’s fair use rules make the last row a matter of discipline as well: estimated results should not be disclosed publicly or compared with measured results. For pre-deployment validation, see verification and validation for production AI. For locating the bottleneck, see profiling GPU kernels to find the real bottleneck.
Running LynxBenchAI on the machine on your desk
The argument is testable rather than rhetorical. pip install lynxbench-ai on Python 3.11 or later, on Linux or on Windows via WSL2, gives a working benchmark; a run needs roughly 15 GB of free disk, downloads about 2 GB of model data once, and takes 15 to 30 minutes. The Personal Edition is released and free for non-commercial use; the Press, Pro and Enterprise editions are not released and are contact-gated through the TechnoLynx contact. Details are on the LynxBenchAI page.
The result reports what that machine achieved on the fixed model catalogue at each precision, carries the 26Q3 release name, and can be read against the published device pages from the same release. It tells you whether your executor sustains what comparable executors sustain, and in which category it is strong or weak. It does not tell you how your own model will run, what your batch sizes or request concurrency will produce, or anything the run does not measure: what is declared is a continuous timed window after a discarded warm-up, nothing more. For a PC, see benchmarking your PC for AI beyond single scores, since standard GPU benchmarks miss the metrics that matter for transformer inference.
How much should I expect production performance to diverge from a published benchmark?
Nobody can give an honest figure in advance, because the divergence is set by how far your deployment sits from the benchmarked configuration, and a prediction made without that workload context is a guess. What can be named are the sources of the gap. Identical-SKU GPUs varied by 8% on average and up to 22% in the 2022 multi-centre study. HPC-style benchmarks are often run with exclusive access to the hardware, conditions that do not reflect shared environments with queue waits and concurrent users. Single-stream throughput is not multi-user throughput, and where a throughput curve turns over depends on the card, quantisation and context length. When a run lands far below its ceiling, the usual causes are partial CPU offload, the model reloading between iterations, and thermal throttling.
So the published result answers the selection question, and your own measurement answers the deployment question. A vision pipeline in AI engineering for sports analytics, for instance, has to be timed on its own footage, resolution and frame rate, not on a catalogue model. The difference between an inference benchmark and a workload evaluation is the next step from here. What remains open is how close your workload’s arithmetic intensity, batch shape and software stack sit to what was benchmarked, and only a run of your own workload closes that question.
