In computing, a benchmark is a procedure: a process for comparing metrics for programs, models or systems to assess their relative performance, usually against a baseline. The score is the numerical estimate that procedure produces. Whether the score can carry a decision depends on things the number alone never shows: which workload ran, under what conditions, with how much tuning allowed, and under which version of the rules.
What is meant by benchmark: a procedure, with the score as its output
A common definition from computing and scientific software evaluation holds that three dimensions structure most benchmarking efforts: runtime, accuracy, and efficiency (the ratio between resources used and the quality of the outcome). A reliability audit of eight cybersecurity LLM benchmarks takes a broader view of the term. It argues that a benchmark is not a static dataset paired with a fixed metric but a measurement pipeline, so the score depends on every stage, from dataset construction through inference to scoring and aggregation. SPEC’s run rules for SPEChpc 2021 make the same point from the hardware side: a published result is an empirical report of performance observed on defined tasks, and it means something only if the conditions of observation are stated.
For AI hardware, a LynxBenchAI release fixes the model catalogue, precisions, correctness thresholds and scoring formula, adapting only batch size. Change any of them and you are measuring something else, even if the hardware stays the same.
A benchmark result is a claim about one procedure on one executor, and it reaches only as far as that procedure is disclosed.
The longer argument for why comparability comes from methodology and not from the metric is in our explainer on what an LLM benchmark measures and what it cannot. The protocol view of the same idea is set out in the comparability protocol for AI benchmark comparison.
Four tests that separate decision-grade results from rankings
A leaderboard and a decision-grade benchmark can show the same kind of table. What separates them is whether the procedure behind the table holds up when someone is about to spend money on its ordering.
The workload resembles the decision
The CARAML authors (Jülich Supercomputing Centre, 2024) state the problem bluntly: when buying heterogeneous accelerators, comparing core counts, TDP, theoretical bandwidth or peak FLOP/s is not sufficient, and performance data from the intended workloads, collected independently of vendor interests, is what carries information. Dehghani et al. (2021) went further and called it a “benchmark lottery”: on several prevalent ML benchmark setups, the relative performance of methods could be altered significantly just by choosing different tasks.
Hardware results show the same effect. In the LynxBenchAI 26Q3 results, the AMD Instinct MI300X scores 275.4 on fp32 MobileNetV3 training against 564.1 for the NVIDIA H100 80GB HBM3, while on fp32 Flan-T5 Small training the order reverses, 1482.9 against 1096.0. Which device comes “first” depends on which test you read. A benchmark whose workloads look nothing like your deployment can rank devices correctly, yet the ranking may not transfer to your deployment.
The methodology is disclosed down to the pipeline
A disclosed methodology gives others enough information to reproduce the result. The 2026 audit recommends that a benchmark release specify the full measurement pipeline, not only the dataset and metric: provenance, inference settings, the executable scorer, denominator policy, aggregation rules and reliability statistics. The audited benchmarks fell short of this. Across eight benchmarks and nine configuration fields, 44 of 72 pipeline fields (61%) could not be reproduced from the benchmark documentation alone. BetterBench (Reuel et al., 2024) found that 17 of 24 assessed AI benchmarks gave no easy-to-run script to replicate their published results. It also asks for statistical significance reporting, such as variance bounds, or an explicit statement where a benchmark is deterministic.
SPEC sets the bar where it belongs for hardware: a full disclosure must contain enough information for the result to be reproduced independently, and any configuration choice the tester knows affects performance must be documented. Precision is a comparison axis in its own right, so LynxBenchAI reports each precision separately and gates every result on a per-precision correctness threshold before it counts.
Optimisation effort is bounded, and the bound is stated
Optimisation effort is part of the methodology, so a benchmark has to bound and disclose it. SPEChpc 2021 handles this with two tiers. “Peak” lets each benchmark be built with its own optimisations, while “base” requires one common set of optimisations and environment settings across the whole suite. Base is meant to reflect what a performance-minded user gets without tuning each program individually. NIST’s January 2026 draft AI 800-2 makes the matching point for model evaluation: for results to be comparable on one scale, execution costs should be held uniform across systems, or else costs have to be reported next to performance.
The MLPerf Training authors (Mattson et al., 2020) note that ML systems vary so much that fair benchmarking with the same binary, code and even hyperparameters is difficult. CARAML’s answer was to roll Megatron-LM back to a commit that can run on all the devices it tests. Cross-vendor comparability is constrained by design, and an honest benchmark says where its constraint sits instead of claiming to have removed it.
A version scopes what may be compared
A benchmark’s rules change over time, so its results should carry the version that produced them. NIST’s draft recommends tagging benchmark versions and suggests semantic versioning, so that a major increment marks the point where results before and after the change are no longer properly comparable. SPEC is stricter: when it releases a new major version of a benchmark, the metrics are generally not comparable to the previous version, and there is no formula for converting between them. Results also may not be compared across different benchmarks or different major releases of the same one.
Where leaderboard orderings lose their meaning
A leaderboard is one way of presenting benchmark output, and nothing is wrong with the format itself. The trouble is that an aggregate ranking is easy to read as a verdict when the procedure behind it would not support one. The research gives several separate reasons:
- Aggregation noise. An analysis of the Hugging Face Open LLM Leaderboard (2026) found that naive rankings conflate underlying capability with ecosystem artefacts, with the biggest consequences at the decision-relevant margins of the ranking.
- Designer priorities. Jung et al. (FAccT 2026) show that leaderboard outcomes are largely set by data composition and aggregation choices made by others, then inherited by users whose goals may differ substantially.
- Selective disclosure. Singh et al. (2025) found that undisclosed private testing let a handful of providers try several variants and choose which scores to show, which biased Chatbot Arena scores.
- Pipeline sensitivity. The 2026 audit found that a single pipeline choice could move a model’s score by more than 80 percentage points and substantially change rankings.
- Missing costs. Ethayarajh and Jurafsky (2020) point out that an inefficient model costs the practitioner but not the leaderboard, because only the practitioner pays for that inefficiency.
Eriksson et al. (AIES 2025) reviewed about 110 studies and describe systemic flaws, including misaligned incentives, construct validity problems and gaming. They also argue that commercial and competitive dynamics shape benchmark practice. That point matters, because every benchmark makes a statement about what it perceives to be important. Even a careful quality checklist has limits. BetterBench describes its own assessment as minimum quality assurance, not enough to judge whether a benchmark suits a specific use case. Our longer treatment of what public ML leaderboards do and don’t tell you goes through these failure modes one by one.
Peak TOPS is a specification, not a benchmark result
A headline TOPS or peak-FLOPS figure has no workload, no conditions and no tuning bound, so by the definition above it is not a benchmark result at all. A 2026 survey of LLM accelerators notes that standardised benchmarks such as MLPerf Inference exist because peak specifications do not predict delivered performance. Google Cloud’s accelerator benchmarking guide likewise points out that vendors often highlight single-chip peak FLOPS while real AI workloads are a distributed-systems problem. Exactly what has to be disclosed before a peak figure can mean anything is answered in the LLM benchmark explainer.
How LynxBenchAI applies the bar
LynxBenchAI runs one model catalogue at fixed precisions on every device in a release, across NVIDIA GPUs (cuda), AMD GPUs (cuda via ROCm), Intel GPUs (xpu) and CPUs. Only the batch size changes, and it rises until throughput stops improving within a defined noise band, so a laptop GPU and a data-centre accelerator are each measured at their own saturation point. Google Cloud’s vendor-neutral guide also raises load for inference: with the model fixed, it increases concurrent requests until the P99 latency SLA breaks, then records TPS/chip.
LynxBenchAI bounds optimisation effort. Models are prepared once, before any target device is known. The same prepared artefacts go to every vendor. Where vendors’ kernels disagree on what is permissible, the stricter constraint is applied to all of them. A workload that fails or times out is recorded as zero and kept in the aggregate. That aggregate is a shifted geometric mean that is monotone by construction, so running a workload slowly always beats not completing it. In the LynxBenchAI 26Q3 results, the NVIDIA GeForce RTX 3090 produced no fp8 result (0.0) on any of the four inference tests, and under the scoring rules those zeros stay in its inference score of 294.1. Its fp16 results stand alongside them, for example 625.5 on Flan-T5 Small.
A LynxBenchAI score has stated limits:
- It reflects the fixed catalogue of one named release, not your own application.
- Each test is one timed measurement window after a discarded warm-up, not a median across trials.
- GT is an ordinal aggregate, not a 0–100 rating, and it is not normalised against a reference device.
- Scores compare only within a release name. The current release is 26Q3 and the next is 27Q1. A new name accompanies a major version, which is free to change the scoring methodology, and no release cadence is promised.
The difference from other tools is one of purpose. MLPerf sets rules and best practices so that systems with very different architectures can be compared. In MLPerf, vendors port and optimise a reference code for their own architecture, showcasing its capabilities. CARAML sizes networks to fully use each system, so not every accelerator trains the same model. LynxBenchAI holds model architecture and precision constant and adapts only batch size to each device’s saturation point. The catalogue runs on stock PyTorch. It does not run or measure your workload.
Published 26Q3 results for 48 devices are on the LynxBenchAI benchmark and leaderboard.
A checklist to run against any benchmark tool before its numbers count
| Question to ask | A decision-grade answer | A warning sign |
|---|---|---|
| What workload ran, and how close is it to mine? | Named models or tasks, chosen for a stated reason | Unnamed “AI performance” or a single synthetic kernel |
| Can I reproduce the result? | Code, configuration and a replication script available | Score published without the pipeline behind it |
| How much tuning was allowed, and was it equal? | A stated bound applied to every system, like SPEC base | Each vendor tuning its own submission with no stated limit |
| Is precision reported per result? | Each precision shown separately with a correctness gate | One number with the precision unstated |
| How was variation handled? | Variance bounds, or an explicit statement of the run protocol | No mention of noise or repeats |
| Which version produced it? | A release or version name on every result | Numbers mixed across versions in one chart |
| Who chose the aggregation, and what does it hide? | A published formula with per-test results visible | Only an overall rank |
A tool that answers all seven can still rank devices on workloads you do not run. A tool that cannot answer them gives you little basis for trusting its ranking. For organisations treating these answers as procurement evidence, see our piece on benchmarks as decision infrastructure, not marketing material.
What no public benchmark can settle for you
A decision-grade public benchmark narrows the hardware shortlist. It cannot tell you how your own model, data pipeline and serving stack will behave on that shortlist, because those were never in its catalogue. The gap between the leaderboard number and your workload’s number is where an evaluation of your own workload starts. TechnoLynx builds that for teams as a re-runnable evaluation harness for production AI. When the question gets down to a single slow kernel, the same discipline continues with profiling GPU kernels to find the real bottleneck.
