Benchmarking AI hardware measures a running system, not a chip. The number you get describes the device together with the backend, driver, framework and runtime that drove it. That is why LynxBenchAI produces cross-vendor numbers with one instrument under one set of rules. As Wang, Wei and Brooks put it, no performance conclusion is possible without benchmarks.
How far apart can “the same GPU” land? A 2022 SC22 study of large accelerator clusters found an average performance variation of 8% (and up to 22%) between GPUs of identical architecture and vendor SKU, with outliers up to 1.5 times slower than the median GPU, regardless of application, vendor or cooling method. Earlier work traced similar variation on CPU-based HPC systems to manufacturing variability and the chip’s power management. If the silicon alone cannot hold a number steady, a benchmark score cannot be a property of the silicon alone.
Does a GPU benchmark measure the hardware or the system that ran it?
The system. Mature benchmark bodies state this outright: SPEC describes CPU 2017 as intentionally depending on the processor, the memory hierarchy and the compilers together, not just the processor. MLPerf’s inference rules define the system under test as a set of hardware and software resources, explicitly including the operating system, compilers, libraries and drivers that significantly influence running time. Research on OpenCL portability treats the device, compiler and runtime together as one complete environment.
LynxBenchAI treats the AI Executor as the unit of measurement: the device plus the backend it ran through (cuda for NVIDIA, cuda via ROCm for AMD, xpu for Intel, or CPU), and the driver, framework and runtime present on that machine. No score is attributed to silicon on its own. The backend names are not cosmetic: PyTorch’s own documentation for Intel GPUs tells developers migrating code to change device references from cuda to xpu.
Why do two identical GPUs give different benchmark numbers?
Because the conditions around the silicon differ, such as clock state and thermal throttling. NVIDIA’s TensorRT documentation puts it bluntly: two runs of the same engine on the same GPU can disagree by tens of percent if the clock floats during one run and locks during the other, or if thermal throttling starts halfway through.
By default a GPU clock floats, idling with no work and boosting when a workload starts. Floating clocks or active throttling mean every kernel may run at a slightly different frequency, which makes measurements less stable; locked clocks give more consistent numbers but lower average performance than a floating or maximum-locked clock. Power throttling is by design, and it hits GPUs with low power limits, such as the T4 and A2, hardest. Even input values matter: all-zero inputs draw less power than realistic data because fewer bits flip in DRAM and L2, which changes the measurement. On Windows, NVIDIA’s TensorRT guidance notes that a GPU in WDDM mode tends to give worse and less stable inference results than one in TCC mode; Linux has no such distinction. NVIDIA’s recommendation is to record and monitor GPU status alongside the workload so unexpected results can be traced to a cause.
A worked example: one kernel setting, months of confusion
A report on benchmarking the NEST simulator describes an extended, slow warm-up phase that appeared on one system and not on another with nearly identical hardware; it took several researchers months to trace the cause to different NUMA-balancing settings in the Linux kernel. In the same project, researchers using supposedly identical benchmark specifications obtained significantly different results, and some of the configuration discrepancies took months to find. Their conclusion is that repeatability is only possible when the experimental setup, including the executing hardware, stays identical. What let them investigate the second machine immediately was a shared configuration that guaranteed the exact same benchmarking setup on both systems.
The lesson we draw is that a result without its configuration attached is an anecdote, however carefully it was timed.
The same card model, two published entries
The LynxBenchAI 26Q3 results carry two entries for the NVIDIA GeForce RTX 5090, one on native Linux and one under WSL: GT 597.5 against 545.7, inference 985.8 against 891.8, training 515.3 against 470.3. For the RTX 3080 the pattern does not run in one direction: the native Linux entry scores higher on GT (160.5 against 158.8), while the WSL entry scores higher on compute (72.5 against 64.8), with FP16 dense matrix multiplication at 106.2 against 71.2. The results record the platform, not what else differed between the entries, so the platform is not a proven cause. Each result is bound to the device and the software stack it ran on.
How do software stacks, drivers and runtime configuration shape results?
Materially. In a cross-platform study, Wang, Wei and Brooks could not reproduce NVIDIA’s ResNet-50 figures, which relied on unreleased libraries; they measured a 6.2x speedup with Google’s ResNet-50 against 4.2x with NVIDIA’s version, which they note suggests software optimisation can significantly change performance.
Hardware and stack together shape how a system behaves under a latency bound. In the MLPerf Inference paper (Reddi et al.), one system lost about 50% or more of its throughput when a latency bound was imposed, while another lost as much as 40% on translation but roughly 10% on vision models; the authors attribute the gap to hardware tuned for small batches, more effective dynamic batching in the software stack, or most likely both. They state that software can leave hardware short of its potential when the matching framework is not used.
PyTorch documents that cuDNN’s benchmark mode may select different convolution algorithms on subsequent runs, even on the same machine. DAWNBench measured TensorFlow running faster than PyTorch on CPUs but slightly slower on GPUs, partly because of data-layout support. TensorRT notes that kernel-launch overhead varies by CPU, driver, operating system and capture mode, so Enqueue Time should be compared with GPU Compute Time on the deployment system.
This is where the choice of benchmark matters. MLPerf deliberately lets submitters optimise reference models and run them through their preferred toolchain on their chosen hardware. In our view that makes an MLPerf result a report of what a submitter’s own optimised stack achieves within the rules, not of the stack your model will run on.
When are AI benchmark numbers safe to compare across machines?
When every result is run and reported under the same set of rules. MLPerf requires the same system and framework for every result reported in one context, and its “Available” submissions must specify software versions, hardware configuration, software stacks, containers and settings for all components. SPEC requires a tool-generated, contemporaneous record of the system under test for public results.
Comparability checklist
| Condition | Why it matters | Safe to compare when |
|---|---|---|
| Same instrument and rules | Vendor-chosen tests on vendor-chosen workloads are not one measurement | One methodology was applied to every executor in the comparison |
| Same release | LynxBenchAI scores are compared within one release only | Both numbers carry the same release name, such as 26Q3 |
| Software stack recorded | Operating system, libraries and drivers significantly influence running time | Versions and settings are attached to each result |
| Same workload and input sizes | Fixed problem sizes can under-utilise some GPUs, while memory-scaled sizes give larger devices larger problems | Input sizing rules are declared and identical |
| Same measurement boundaries | Pre- and post-processing belong in a system number but not an inference-only number | Start and stop points match |
| Same precision support | A precision a device cannot execute is scored as zero, not skipped | Both executors ran the precision being compared |
The input-size row is harder than it looks. The authors of a 2025 cross-architecture solver study found few clear, singular points of comparison across heterogeneous platforms. Comparing each architecture against its own optimised baseline avoids much of that bias, but costs considerable extra effort, and peak-normalised metrics ignore the attainable algorithmic peak that only architecture-specific implementations reveal.
Version boundaries are just as firm. SPEC states there is no formula for converting CPU 2006 results to CPU 2017 results, because code, data sets, hardware stressed, metrics and run rules all differ. LynxBenchAI applies the same rule: 26Q3 numbers are compared within 26Q3 only.
CPU and GPU results need extra care. PyTorch does not guarantee reproducible results across releases, commits or platforms, or between CPU and GPU executions even with identical seeds. The CPU and GPU comparison has its own failure modes, covered in our piece on where CPU and GPU benchmark metrics differ.
Accurate, correct, and still misleading
A benchmark can be run flawlessly and still answer the wrong question. DAWNBench’s authors observed that most deep learning benchmarks measured proxies such as time per minibatch, while larger batches and reduced precision can stop training from converging or slow it down. In their experiments, cumulatively enabling optimisations reduced time to 93% accuracy, but the combination never reached 94% until the larger minibatch was removed.
Task choice distorts conclusions in the same way. Dehghani and colleagues showed that the relative performance of methods can change significantly simply by picking different benchmark tasks. Wang, Wei and Brooks found that the Transformer trains 3.5x faster on TPU than GPU, yet that single model hid a severe TPU memory-bandwidth bottleneck in fully connected layers above 4k nodes. And a speed number ignores qualities such as reliability, availability and serviceability, which often trade off against raw performance.
Some limits no amount of care removes. Vendors tune products for industry-standard benchmarks, and SPEC itself warns that no standardised benchmark can perfectly model a particular system and user community. The governance side of this, where a reviewer needs evidence about one workload rather than a leaderboard position, is argued in inference benchmark versus workload evaluation.
Synthetic, micro, application and suite benchmarks: what each one isolates
Each type moves the measurement boundary to a different part of the executed system.
| Type | What it isolates | What it misses |
|---|---|---|
| Microbenchmark | One capability, such as memory bandwidth or a single GEMM shape | The complexity of full models and dependencies between operations |
| Synthetic or parameterised | Sensitivity to model attributes across a wide space; ParaDnn spans almost six orders of magnitude of parameter size | The specific behaviour of the models a team actually deploys |
| Application (real model) | End-to-end behaviour of a real workload, including the stack | Only the sparse points chosen; popular models can age out |
| Suite | A spread from low-level characteristics to end-to-end applications, as in ALTIS levels | Coverage depends on diversity; Rodinia’s applications were found highly correlated |
LynxBenchAI’s catalogue combines real models with dense and sparse matrix-multiplication tests. Its 26Q3 catalogue covers MobileNetV3, DistilBERT, DistilGPT2, Flan-T5 Small and GraphGPS on Peptides.
Category scores diverge. In the 26Q3 results the AMD Radeon RX 9060 XT scores higher on compute (112.7 against 61.7) while the NVIDIA GeForce RTX 5060 Ti scores higher on inference (240.4 against 122.7). Reading those categories against each other is developed in why GPU performance is not a single number.
What do hardware performance counters tell you that a benchmark score cannot?
Counters show what happened in a kernel and how close it came to peak; a timed end-to-end run measures the whole system. They answer different questions, and neither substitutes for the other.
ALTIS used nvprof utilisation metrics to show that many GPU components sat underused by the Rodinia suite, and they can separate compute-bound convolution from memory-bound batch normalisation. Their limits are just as concrete:
- Not portable. Stress counters such as cache or floating-point utilisation differ significantly between GPUs and input sizes, counters changed completely from NVIDIA’s Volta generation onward, and the equivalence of old and new counters is incompletely documented.
- Not freely aggregable. AMD’s GPUPerfAPI documentation notes that percentage counters cannot be summed or simply averaged across a group of draw calls and still reflect actual performance.
- Not always deterministic. Weaver and colleagues found only a small minority of x86-64 counter events to be deterministic and free of overcount.
- Not free to collect. Hardware limits mean kernels may need replaying across passes; metrics such as hit rate can carry significant error when their parts come from different passes on a short kernel; and kernels run slower while counters are collected.
- Not a verdict. Nsight Compute warns that high peak utilisation can show a bottleneck but does not necessarily indicate efficient use.
We would use counters after a timed run has shown where the gap is, never in place of it.
Measuring your own executor with LynxBenchAI
LynxBenchAI measures AI Executors with one catalogue and one set of rules across NVIDIA, AMD, Intel and CPU backends. The Personal Edition installs with pip install lynxbench-ai on Python 3.11 or later, on Linux or Windows via WSL2; a run needs about 15 GB of free disk, downloads about 2 GB of model data once, and takes 15 to 30 minutes. It is free for non-commercial use; the Press, Pro and Enterprise editions are not released and are contact-gated.
It suits anyone who wants to see the executor argument with their own eyes: two runs on the same card under different stacks are the cheapest proof of it you can produce. The getting-started guide covers installation and the first run, and our practical protocol for benchmarking a PC for AI covers the surrounding discipline.
The limits are declared, not implied. A score reflects the fixed model catalogue of one named release, not your application; each test is one timed window, not a median of repeated trials. GT is an unbounded aggregate with no reference-device normalisation, so it implies no 0 to 100 scale. A precision the device cannot execute scores zero rather than being skipped: in the 26Q3 results the RTX 3080 and the A100-SXM4-80GB both show 0.0 for FP8 inference on every model.
The benchmark that finally decides
When performance is critical, the only benchmark that matters is the target environment’s application suite. For sports workloads, our page on AI engineering for sports analytics explains how we build and evaluate systems in that target environment. Public benchmarks earn their place before that point, comparing architectures that specifications alone cannot. They do not model your own system perfectly, and vendors have tuned for benchmarks unrepresentative of real use.