You can only compare two GPU benchmark numbers when the conditions behind them match and are written down. That means documenting what was compared, on what hardware, and at which batch size. A raw tokens-per-second figure means little without a stated batch size. Precision and runtime together can move the result further than a hardware generation does. With the same driver version on both GPUs, the DEEP-GAP study (2026) measured the newer L4 at up to 4.4 times the T4’s throughput, while INT8 on the T4 reached up to 14 times its FP32 speed.
Profiling comes first, because the context that decides what to optimise comes from a profile, not from best-practice rules. Normalisation expresses each result on a single, comparable metric. Runs with different configurations stay in separate rows, and any “A beats B” sentence covers only what was actually measured. Whatever GPU benchmark software produced the original number, these steps decide whether the number can be ranked at all.
How to phrase “GPU A beats GPU B” so it covers only what was tested
“GPU A beats GPU B” should not be generalised beyond the configuration that was benchmarked. It tells you nothing about gaming, fine-tuning or any batch size that was not run. A study of TPU, GPU and CPU platforms found that large batches favoured the TPU and small batches favoured the GPU. So the TPU’s large-batch throughput lead did not hold at small batch sizes, and calling either platform the overall winner overstates the result.
Researchers who compared CPU and GPU runs of a climate-model component (GMD, 2026) showed that code structure and optimisation can change which device looks faster. The AlgoPerf authors (2023) warn of a related trap: it is tempting to claim a general improvement for a method that only helps on one specific workload.
A scoped claim that can survive a hostile follow-up question names six things: the model and its version; the precision, including what was quantised; the batch size; the metric, and whether it was sustained; the date and software stack; and the basis of comparison. In practice it reads like this:
On [model, version] at [precision], batch size [n], GPU A sustained [x] tokens per second against GPU B’s [y], measured on [date] with [driver, CUDA, framework, container image]. Other models, precisions, batch sizes and workloads such as fine-tuning were not tested.
The second sentence is the caveat that keeps the claim from misleading when it is quoted on its own. Leaving it out turns a measurement into a verdict.
How do I normalise tokens/sec across FP16, FP8 and INT8 runs?
Tokens per second from an FP16 run and an INT8 run can only be compared once each figure carries its batch size, precision and workload. A claim of real generational uplift also needs either a same-precision rerun on both parts or a documented adjustment for precision. TechnoLynx checks that a claimed generational uplift still holds once precision is matched, starting from the configuration list behind each number. LynxBenchAI’s published cross-device results serve as the reference.
The T4 and L4 figures in the opening show why. Set an INT8 result on the newer card against an FP32 result on the older one, and you mix a generational effect with a much larger precision effect. The headline ratio then describes neither.
FP8 adds another trap, because no standard defines which operations actually run in FP8. Van Baalen et al. (2023) point out that the NVIDIA, Arm and Intel FP8 whitepaper ran only convolutional and linear layers in FP8 for its ImageNet results. Intermediate values, and operations such as Softmax, stayed in FP16. The same authors note that the choices made in a number-format comparison strongly affect both accuracy and on-device performance, so an “FP8” row needs its quantisation scope listed before it can sit next to an INT8 row.
In the HPC AI500 tests, mixed precision raised FLOPS while accuracy dropped as system scale grew. MLPerf Inference keeps that trade-off visible by requiring almost all implementations to reach within 1% of the FP32 reference model’s accuracy.
What’s a fair common axis for a consumer GPU and a datacenter accelerator?
No axis works for every case. Google Cloud’s accelerator benchmarking guide recommends tokens per second per chip with a representative model held constant, divided further by chip cost to give tokens per second per chip per dollar. The authors of a review of a decade of GPU research software (2020) add a warning: normalising by cost, energy or theoretical peak only means something for some applications, does not solve the comparison problem in general, and can make published results harder to interpret. Whatever axis you pick has to be stated on the chart and applied the same way to every result on it.
Worked example: how the choice of axis flips the ranking
Google Cloud’s guide gives an example in which Chip_A averages 0.79 of Chip_B’s throughput and 0.60 of Chip_C’s, which on raw performance makes Chip_C the best. The guide then prices the chips at $100, $180 and $200. Set Chip_A’s throughput to 1 and the per-dollar figures follow:
| Chip | Relative throughput | Price | Relative throughput per $100 of price |
|---|---|---|---|
| A | 1.00 | $100 | 1.00 |
| B | 1 / 0.79 ≈ 1.27 | $180 | 1.27 / 1.8 ≈ 0.70 |
| C | 1 / 0.60 ≈ 1.67 | $200 | 1.67 / 2.0 ≈ 0.83 |
Chip_A goes from last place on raw throughput to first place per dollar. The axis decides the ranking here, so it must be stated.
Per-watt figures need extra care. One 2026 edge LLM study measured the Jetson at the socket. On the server it combined GPU and CPU-package telemetry, which leaves out memory, fans and power-supply losses, so its authors treat cross-platform energy values as indicative only. Gernot Heiser’s list of benchmarking crimes adds two rules: the baseline belongs in the denominator, and absolute numbers belong next to every ratio.
SPEC’s fair-use rules describe a technique for historical comparisons: results are normalised through “bridge” systems measured on both benchmark generations, and the result must be presented as approximate, because different bridges give different ratios. Two leaderboard entries reported against different reference baselines become comparable once both are re-expressed against one shared baseline, measured on the same stack, using the configuration each entry disclosed. If the driver, CUDA or container details for either entry are missing, the resulting speedup is indicative only and should not be published as a number.
MLCommons likewise treats results from different rounds as comparable only when the benchmark has not changed between them. Baselines drift between releases, so every comparison across releases has to be normalised again.
Should results be averaged into one leaderboard row or kept split by configuration?
Keep them split. Community results that differ in driver, power limit or quantisation format belong in separate rows, each tagged with what varied. Only rows whose controlled variables match should be merged, because averaging across them blends independent variables into a number that no real configuration produced. TechnoLynx lists which of those variables actually move throughput in the leaderboard’s own data, builds the bucketing and tagging logic against that list, checks it against known variance patterns, and runs it as a standing rule for new submissions, adapting the disclosed-controls cross-vendor methodology.
Industry benchmarks keep differing implementations apart through separate divisions. MLPerf’s closed division requires implementations equivalent to the reference so that systems can be compared directly, while the open division lets submitters change the model. MLPerf results may only be compared with compatible MLPerf results. Any difference in version, division, scenario or chip count must be identified in the main text, table or figure. Every submission must contain system metadata, code, setup scripts and result logs for each system-implementation combination tested. SPEC requires the basis for comparison to be stated, and the SPECjbb2015 rules forbid comparing results across run categories.
Agreeing which runs count as the same release identity (same model version, precision and cluster size class) is a separate rule that is best settled before any ranking. Two caveats apply to bucketing. Without a cap, it can split a leaderboard into near-duplicate rows. And self-reported driver or power-limit metadata may simply be wrong.
A gap smaller than the run-to-run spread is not a win
A 2025 review of AI benchmarking practice reports GPU performance variability of 8% on average, up to 22%, with outlier GPUs up to 1.5 times slower than the median. In the DAWNBench analysis (2018), the coefficient of variation for time-to-accuracy was at most 4.5% for reproduced entries without novel statistical optimisations, and 12.2% across all entries. SPEC lets a published SPECjbb2015 result be challenged as non-compliant by showing its average metric is more than 5% lower than published, using a Student’s t-test on at least 10 consecutive runs. MLPerf HPC requires a specified number of training runs to convergence, drops the fastest and slowest, and reports the mean of the rest. Heiser’s list treats a raw average with no indication of variance as misleading. Two rows whose gap sits inside their measured spread should be shown as a tie, with the run count beside them.
Can I cite an independent benchmark next to a vendor’s figure?
Yes, provided the independent result discloses enough for a third party to rerun it. A disclosed figure placed next to an undisclosed one shows readers exactly what the vendor left out.
LynxBenchAI publishes results on how each device runs the public PyTorch stack, broadly indicative for custom PyTorch models but not a measurement of your own application; the first step is to check its stated configuration against the one behind the vendor’s claim.
TechnoLynx pins down the workload, precision and configuration behind the vendor figure. The team then runs a matched, scaled-down version of that workload on the relevant accelerators and checks the run against a disclosure-sufficiency standard, so it can be cited next to the vendor number.
Independent cross-vendor reports exist and can be cited. MLPerf Inference v0.5, in 2019, cleared 595 valid submissions from 14 organisations. Milabench (2024) lists the hardware, PyTorch version and CUDA or ROCm version of each of the four systems it compares. A buyer’s guide from BERI defines a benchmark as independent when someone other than the model’s maker chose the prompts, decoding settings and scoring, and published enough for a third party to rerun it. The same guide notes that most vendor claims publish the tasks but leave out the harness and the exact configuration.
Before you cite a result, check that it discloses:
- the workload or task set, and its version;
- the harness and full software stack, including driver and framework versions;
- the exact configuration: precision and quantisation method, batch size, scenario;
- the run count and spread;
- whether it was independently reviewed or self-reported.
SPEC puts the principle plainly: anything that affects performance must be described, so that an independent party could reproduce the result without further help. Under MLCommons rules, a comparison that sets an unverified result next to a verified one must carry a footnote saying the unverified result has not been through MLPerf review. When you benchmark a competitor yourself, Heiser recommends describing every configuration parameter and taking particular care if your numbers disagree with published data for that system.
Why does the same stream score differently on Jetson and x86 appliances?
Precision, batch size, software framework and operating conditions each shift measured performance on edge hardware. Only profiling each appliance type shows which effect dominates.
LynxBenchAI publishes device results for each hardware class on the public PyTorch stack, which sets expectations before any profiling; the first step is to look up the appliance types in your fleet.
TechnoLynx catalogues the appliance types, precisions and driver versions in the fleet, and profiles each type with Nsight Systems and Nsight Compute to separate quantisation, driver and thermal effects. The team then implements normalisation with controlled batch size, precision and workload, and validates the normalised scores against ground-truth quality before deploying a single calibrated threshold.
Profiling intrudes on what it measures, which is why a 2025 study of concurrent inference on Jetson split its method in two. A lightweight first phase collected throughput and power. The second phase used Nsight Systems, which by itself cut throughput by 50%. Throughput for a calibration therefore comes from the light tool, and the heavy profiler is used only to explain it. The same study found that GPU utilisation could reach 100% while SMs and Tensor Cores often ran at only 15% to 30%. Once power draw passed a threshold, the device cut throughput through dynamic voltage and frequency scaling, which made the behaviour nonlinear.
Precision effects can also depend on the backend rather than the format. In a 2026 study of small vision-language models on Jetson Orin, the INT4 token-latency penalty came with the BitsAndBytes execution path and did not appear with AWQ. In the same study, accuracy rankings held across the edge platforms whatever the quantisation configuration.
If variance remains after normalisation, treat it as a real capability difference and set separate thresholds per appliance class, rather than forcing one threshold to fit. Keeping driver and TensorRT versions under control across hardware refreshes is therefore an ongoing job. Any requantisation has to be validated for quality as well as speed, because INT8 and FP16 change accuracy too.
Explaining normalised GPU results to readers who are not engineers
A plain-language explainer holds up if it does three things. It names the configuration behind each number (batch size, precision, peak or sustained, hardware generation) before drawing any comparison. It uses one analogy throughout. And it is reviewed by a technical lead for accuracy and by communications for clarity before it goes out.
LynxBenchAI publishes results on how each device runs the public PyTorch stack, giving an explainer a sourced number to anchor on; the first step is to pick the device your readers already know.
Why mixing releases or configurations is unfair, in terms anyone accepts
On a single T4, the DEEP-GAP study measured INT8 inference at roughly 2.5 to 14 times FP32 throughput, depending on the model and batch size. The gains rose fastest between batch sizes 1 and 32, then levelled off as the hardware saturated. In plain terms, batch size 1 is one passenger per trip and batch size 64 is a full coach, while INT8 carries each number in less detail than FP16. The analogy works as long as it does not suggest that performance scales in a straight line, because the T4 results show it does not.
One sentence that engineering and communications can both sign
Engineering should draft the scoped sentence from the template, communications simplify the wording, and engineering check that no claim-changing caveat was lost. SPEC’s fair-use rules explain why the caveat has to sit inside the quotable unit: diagrams, tables and abstracts are often quoted separately, so each must carry enough context on its own not to mislead. Under the same rules, an estimate must be marked as an estimate wherever it appears, including inside a chart.
Putting a new chip in terms of the consumer GPU readers already know
For a trade audience who know the consumer GPU but not the new accelerator, the clearest framing is an index. Set the familiar card to 100 on one normalised axis, with the previous model and the new chip expressed against it under the same disclosed configuration. LynxBenchAI’s published results are one source for that consumer-GPU anchor, since they show how each device runs the public PyTorch stack. A quotable summary then reads: “Running [model] at [precision] and batch size [n], the new chip sustained about [k] times the throughput of [consumer GPU]; other workloads were not tested.” Any index across releases stays approximate and should say so.
Why does a GPU’s peak throughput differ from its sustained throughput, and which one should a comparison use?
Peak throughput is the theoretical ceiling vendors quote. Sustained throughput is what the chip delivers in real workloads once power management and frequency scaling reach steady state. Comparisons should use sustained figures, with peak shown only for context.
LynxBenchAI publishes results on how each device runs the public PyTorch stack, a reference for measured rather than advertised throughput; the first step is to set its results beside the vendor’s peak figure.
Google Cloud’s guide quotes SemiAnalysis on this: what matters is effective FLOPS, because peak numbers are almost never reached in real workloads. NVIDIA’s NVML documents the clock event reasons behind the gap. They include a software power cap, software thermal slowdown and hardware slowdowns that cut core clocks by a factor of two or more. Logging those flags during a run shows whether a result was throttled. MLPerf Inference requires every benchmark to run for at least 60 seconds, so that it measures the steady state of power management and frequency scaling.
The original TPU paper (2017) shows how much configuration and latency targets matter. Running the K80 without Boost mode, and per die rather than per board, lowered its peak from 8.7 to 2.8 TOPS. Under the application’s 7 ms 99th-percentile response limit, the Haswell CPU and the K80 ran at 42% and 37% of their best possible throughput on one workload. One study measured average Tensor Core utilisation of 3.9% to 9.2% across six networks in end-to-end training.
For a journalist or ministry official, one analogy is a car’s top speed against its average speed on a real journey. A simple chart with two bars per device, advertised peak and measured sustained, and the test conditions in the caption, makes the same point. “Sustained execution beat peak TOPS” reads as fact rather than marketing only when it is tied to the organisation’s own reproducible measurements and their stated conditions. Without that evidence, an analogy invites accusations of spin.
Which driver and container variables must be pinned before a cross-vendor or cross-cloud comparison is valid?
Before a throughput gap can be put down to hardware, the software versions of every component in the stack, including driver, toolkit, framework and container, have to be specified.
LynxBenchAI publishes results on how each device runs the public PyTorch stack, a reference point for cross-vendor comparisons; the first step is to match its stated configuration against the one behind your own numbers.
TechnoLynx starts with an inventory of the stack each existing number was produced on and defines which confounders matter for the comparison. The team then chooses between a full rerun on a pinned reference stack and a calibrated correction factor, runs the protocol across the SKUs, clouds or vendors in scope, and documents it for the audit or tender record so it can be repeated.
MLPerf defines the system under test as including the operating system, compilers, libraries and drivers, because each of these significantly affects runtime. The 2017 TPU paper disclosed that turning off Boost mode and enabling ECC cut the K80’s memory bandwidth from 240 to 160 GB/s, which shows that clock and memory settings belong in the same disclosure.
The pin list for a cross-cloud or cross-vendor comparison:
- GPU driver version, recorded per host;
- CUDA or ROCm toolkit, plus cuDNN or the equivalent vendor libraries;
- framework version (for example PyTorch) and inference runtime (for example TensorRT);
- container image, pinned by digest rather than by a moving tag;
- virtualisation layer and instance type;
- power limit, application clocks, boost behaviour and ECC state;
- precision and quantisation method;
- batch size, sequence lengths, warm-up, run duration and run count.
How much of a gap could come from driver, CUDA or container versions?
It can be large. The Milabench authors (2024) report that new PyTorch versions often bring gains from 5% to more than 100% on specific benchmarks. They also note that not every vendor in their own study could run the same PyTorch version, and that version can have a major impact on performance. In the 222 days between TensorFlow 1.7.0 and 1.12.0, one platform study measured 90th-percentile TPU speedups from software stack improvements of 7 times for fully connected models and 9.7 times for RNNs. Milabench attributes part of the gap between synthetic FLOPS and real workload performance to differences in software-stack maturity.
Rerun everything on a pinned image, or correct statistically?
When the stakes are high, as in tenders or decisions that must survive a challenge, a full rerun on one pinned image is the defensible choice. When rerunning everything is impractical, a sampled rerun on a fixed reference stack can calibrate a correction factor for the remaining claims. The HPC AI500 methodology states the underlying rule: test one layer while keeping the others intact. AlgoPerf requires wall-clock comparisons on a standardised system with consistent software versions. Containerised environments help by providing a stable operating system and software stack across machines.
A statistical correction is only as good as the version-delta data behind it. Even on a pinned stack, interactions between hardware, interconnects and software still shape the result. Turning those controls into an evidence package that holds up to independent scrutiny is covered in our piece on turning GPU performance numbers into auditable evidence.
Who can run a defensible cross-vendor GPU comparison for us?
TechnoLynx runs cross-vendor and cross-cloud GPU comparisons for AI infrastructure and SaaS teams, documented for an audit or tender record. The team pins the stack, reruns each configuration at matched batch size and precision, checks the numbers are statistically defensible and writes the methodology down. Doing this in-house works when your team controls the reference hardware and can hold the software stack still for the length of the study. A specialist earns their place when the result must survive an outside challenge or spans vendors and clouds you cannot pin yourself. When choosing a partner, ask to see a previous methodology disclosure, how they reported run-to-run variance, and whether they kept differing configurations in separate rows. LynxBenchAI provides published results on how each device runs the public PyTorch stack, as a hardware reference rather than a run of your own application.
A full engagement does not suit a team that needs one quick sanity check on a single card, or a buyer looking for the cheapest possible test hours. If all you need is a public reference point, published MLPerf results or LynxBenchAI releases may be enough. If the real decision is which model to deploy rather than which accelerator, the LLM Selection Pack fits better.
Which metric belongs in the headline, and which in the footnotes?
For non-specialist readers, the headline should be a normalised, workload-level metric: tokens per second at a disclosed batch size and precision, or latency-bounded throughput for serving. Utilisation percentage and the gap to vendor peak go alongside or in footnotes. For a technical audience diagnosing efficiency, utilisation and latency breakdowns move into the main report.
LynxBenchAI publishes cross-device results for the same PyTorch stack, a reference for the headline metric; the first step is to compare its configuration with the one behind your headline figure.
TechnoLynx defines the metric set and normalisation protocol against your actual PyTorch workloads and runs them on the target architectures under matched conditions. The team then computes the headline and supporting metrics and writes the methodology disclosure that answers an accusation of cherry-picking.
Google Cloud’s guide frames inference as maximising throughput without breaking latency requirements, compared as tokens per second per chip for a fixed model. The MLPerf Inference authors (2019) argue that latency-bounded throughput reflects realistic deployment better than raw throughput or latency alone. Their data shows why. Compared with the offline scenario, the latency-constrained server scenario cut throughput by 39% to 55% for the NMT translation model and by 3% to 35% for ResNet-50. An unconstrained figure therefore has little bearing on a latency-bound deployment.
Utilisation percentage measures the share of time a process runs on the GPU, not how busy its SMs and Tensor Cores are. A 2025 review of benchmarking practice calls the choice of metric the most crucial part of the process, because the metric defines what the benchmark is for. A ratio in the headline still needs the absolute numbers next to it.
For forum-sourced numbers with inconsistent configurations, tokens per second at batch size 1 in FP16 is a fair canonical point only where that configuration was actually measured. Otherwise the honest label is “not comparable”. Batch size 1 in FP16 may also misrepresent a production setup that serves batched INT8 traffic. Normalisation does not remove the bias of the workload chosen. Google Cloud notes that gpt-oss-120B’s attention head dimension of 64 suits GPUs but wastes cycles on TPU matrix units, so using it as a TPU benchmark can signal poor hardware when the real problem is an architecture mismatch. HPC AI500’s Valid FLOPS is one example of a single composite score reflecting both performance and accuracy.
What a defensible GPU comparison has to disclose
| Disclosure field | Why it matters |
|---|---|
| Model, version and workload | Small pipeline details change which method or device looks better |
| Precision and quantisation scope | “FP8” can mean very different things; there is no standard definition |
| Batch size and scenario | Throughput is only meaningful together with a stated batch size |
| Sustained condition and run duration | Peak figures are almost never reached in real workloads |
| Driver, toolkit, framework, container | New PyTorch versions alone have moved specific benchmarks by more than 100% |
| Normalisation axis | The axis can reorder the ranking |
| Run count and spread | Without variance, a gap gives no indication of significance |
| Date and verification status | Unverified results must be marked as such |
Every field in that table can be filled in correctly and the comparison can still mislead, because someone picked the workload and the workload carries their priorities.
FAQ
Is INT8 faster than FP8?
In dedicated hardware, van Baalen et al. (2023) estimate that FP8 multiply-accumulate units are 50% to 180% less efficient than INT8, depending on accumulator size, which would make compute-bound workloads slower in FP8. On accuracy, INT8 beat FP8 for most well-behaved networks they tested. FP8 came out ahead mainly on outlier-heavy layers in naive post-training quantisation.
Is FP8 slower than FP16?
Not in the proposed FP8 implementation. FP8 is faster than FP16 for networks dominated by weights, memory or arithmetic, but where large activation tensors stay in FP16 they remain the bottleneck and the speed-up shrinks.
