Someone else can reproduce a benchmark result only if it comes with enough detail to rebuild the run. That means the exact hardware, the driver, firmware and software versions, the dataset or workload, every setting held fixed (precision, batch size, context length, seeds), how many runs were made and how much the results varied between them. SPEC puts the principle plainly: a report that certain performance has been observed is meaningful only if the conditions of observation are stated. Everything else on a reproducibility checklist serves three jobs: making that disclosure complete, keeping the comparison fair and letting someone outside check the number.
What a full benchmark disclosure contains
SPEC and MLPerf ask for much the same list. SPEC CPU 2017 requires a public result to include the tester’s configuration file, a flags disclosure, a description of the hardware, firmware/BIOS and software needed to reproduce it, and every configuration choice that differs from the default. MLPerf Inference asks for the same kind of content in its own terms: performance scores, the benchmark code, a system-description file covering accelerator count, CPU count, software release and memory system, and the LoadGen logs for the performance and accuracy runs. In MLPerf’s rules, results that cannot be replicated are not valid results.
Hardware and software versions are the items most often missing
A How2Bench survey of code benchmarks (2025) found that only 24.6% documented the devices used in their experiments, and only 6.3% documented the operating system. The gap matters because performance metrics are highly sensitive to hardware specifications, system configuration and environment, so small differences in conditions can move results significantly. For GPUs and other accelerators, the HPC reproducibility report (2025) notes that matching an author’s setup can mean matching microarchitecture, model number, firmware and OS device-driver version.
SemiAnalysis ran into this while building InferenceMAX. A vLLM container built on a PyTorch image shipped a version of NCCL without prebuilt Blackwell SM100 code, so the instance stalled for up to 30 minutes on their bare-metal B200 machines while other users on the same image saw no problem. Those users had the compiled code cached in a home directory that their cluster mounted. The same container image stalled on one cluster and not on another, because a mounted home-directory cache hid the missing code. For that reason we would put the container digest, the library versions inside it and the driver on the host into the checklist as three separate items.
Comparing a new model or GPU against the one it replaces
We would run a candidate and its incumbent on the asker’s own workload. The two are measured under settings fixed in advance and disclosed alongside the results, and they are judged against an acceptance threshold agreed before anyone sees the numbers. TechnoLynx builds that kind of harness: it frames the comparison (representative workload, acceptance metrics, confounds to hold fixed), runs the candidates on the target stack and documents the setup so it can be rerun. For the hardware side of the question, LynxBenchAI publishes results showing how each device runs the public PyTorch stack, a useful reference before a team commits to its own runs.
LynxBenchAI publishes results showing how each device runs the public PyTorch stack, so a team weighing a new GPU against the one it replaces starts by checking the candidate’s published LynxBenchAI results before running its own workload.
The decision splits by what is being compared. When the question is model quality (summarisation, speech recognition, a replacement risk model), we would build the evaluation set from the reader’s own documents, audio or scenarios. Domain metrics are then backed by human review wherever automatic scores miss hallucinated figures or edge cases. When the question is hardware, the method is a normalised cross-vendor protocol: precision, batch size and workload are fixed, and any quantisation difference is reported as an explicit accuracy-versus-throughput trade-off.
Which settings must be held fixed for an apples-to-apples comparison?
Batch size, concurrency, input and output length and sampling parameters all change measured throughput and latency, so they need to be held consistent across a comparison. Batch size and sequence shape alone move the number a lot. In the LLM-Inference-Bench study, batch size 64 produced 26.6 times the throughput of batch size 1 at a token length of 2048 on an A100. An input/output shape of {1024, 128} had 14.6 times the throughput of {128, 1024}. Compare two cards at different batch sizes or sequence shapes and the gap you measure is mostly the configuration.
Some settings are less obvious. NVIDIA’s benchmarking guidance says ignore_eos should be set to true so that every request reaches its intended output length, and that sampling choices such as greedy, top-p or temperature should stay consistent within a setup because they change generation speed. Tools also define their metrics differently. NVIDIA notes that LLMPerf’s tokens-per-second figure includes prompt generation, request preparation and response storage, overheads sometimes accounting for 33% of the benchmark duration at single concurrency. So the measurement tool and its version belong on the checklist too. Prompt wording counts as a fixed setting as well. EleutherAI’s lm-eval team reported performance swings above 20% from prompt choice, and one model scoring 72.4% or 26.5% on ARC-Easy depending on the evaluation format.
The same principle applies when the output is power. A study of MFU as a proxy for GPU power found that fitting one model per GPU, precision and batch size cut prediction error from about 10% to about 1%. That makes the error statistically indistinguishable from the measurement-noise floor, so in that study precision and batch size were the variables that changed the result.
How many runs or queries before a result is statistically solid?
The answer depends on the size of the difference you are trying to detect and on how noisy the system is. SPEC CPU 2017 runs each benchmark three times and takes the median, or twice and takes the slower run. The MLPerf Inference paper sets query counts by the latency percentile. It requires 1,024 queries for single-stream, about 270K queries for a 99th-percentile guarantee in server and multistream vision tasks, and a 60-second minimum run so that power management and DVFS settle into steady state. The run count has changed over time. The 2019 paper required five server runs, while the current rules require one submitted run per scenario, though in the server scenario LoadGen searches across several test runs to find throughput. We would follow the current rules text over the 2019 paper. MLPerf Training requires 5 runs for vision tasks and 10 for the others, drops the fastest and slowest, and reports the mean of the rest.
LLM evaluation is noisy even at temperature 0. Across ten runs on SWE-Bench-Verified, single-run pass@1 estimates varied by 2.2 to 6.0 percentage points even at temperature 0. Detecting a 2% improvement at 80% power needed about 9 runs per agent at median variance, and detecting a 1% improvement needed 36. Miller (2024) adds that when evaluation questions come in related groups, clustered standard errors can be more than three times larger than naive ones. For a pass/fail reading on production-representative queries, we would therefore run a power analysis before collecting data. That means stating the smallest regression that would matter and sizing the query set and repeat count to detect it.
Few published cloud performance studies do any of this. A systematic review of cloud performance papers from 2012 to 2017 found that more than 63% reported no measurement variance and more than 90% performed no statistical evaluation.
When MLPerf or a public suite is enough, and when you need your own workload
A public suite is the right tool when the question is “how does this system compare with others on a shared, audited task”. MLPerf fixes the dataset, LoadGen and accuracy script for every submission, so its closed-division results can be compared across very different architectures. A public suite does not answer how your traffic mix, documents or audio will behave. The Benchmark Lottery study showed that choosing 4 of SuperGLUE’s 8 tasks in its 70 possible ways produced 6 different top-ranked models. A broader review of optimisation benchmarking reached the matching conclusion: problem instances should be representative of the real-world scenario, or general conclusions cannot be drawn from them.
The practical answer is to use both. We would reuse MLPerf’s methodology (query counts, minimum run time, the caching ban, logging) as the measurement protocol and swap in the reader’s own model and data as the workload. That mix is not “MLPerf-comparable” in the closed-division sense, and the next section covers how to say so honestly.
Ground truth when the legacy system is also in doubt
Two systems disagreeing is not enough to say which one is wrong. We would run both over the same historical and synthetic cases and agree a disagreement tolerance in advance. Every case above that tolerance should go to engineering review instead of being averaged away. Synthetic scenarios should come from a generator that neither system’s assumptions shaped, and they should cover more than one operating regime, because a comparison that holds in one regime says little about the others. There are limits. A scaled-down or synthetic set can miss rare production edge cases, so the evaluation report should say what the set does not cover. Numerical parity at a defined tolerance (bit-for-bit or basis-point agreement between a legacy kernel and its port) is a related but separate acceptance test, and it deserves its own protocol.
A harness you can rerun when the next GPU generation arrives
A benchmark you can rerun keeps version-controlled, pinned build manifests for its environment and puts its parameters in configuration files instead of hardcoding them. InferenceMAX reruns its whole suite every night on hundreds of chips. It orchestrates jobs through GitHub Actions with Docker or SLURM launch scripts and fixes three input/output length pairs (1024/1024, 1024/8192 and 8192/1024). The HPC reproducibility report recommends version-controlled build manifests and explicit version pinning instead of defaulting to “latest”. One benchmark-quality assessment framework scores a build status check, which shows the evaluation code ran at the latest commit. Storing raw measurements also lets the analysis be recomputed without rerunning the experiment.
A harness that reruns on every model, driver or GPU change is the same discipline as keeping production AI reliable: a change triggers a comparison against a recorded baseline. The harness also needs maintenance. When the workload or model versions drift, the comparisons go stale.
One harness every bidder or listed host must run
We would let a score carry a “verified” label, or be compared across tender bids, only when every host or bidder has run the same disclosed harness, container specification and workload mix unmodified and someone independent has spot-checked the submissions. TechnoLynx can help a procuring team or marketplace operator define the representative workload mix (an OCR plus RAG pipeline, for example), specify the harness, container and submission protocol, and run or validate submissions on the target hardware to support the spot-checks. The assurance side of that work belongs under AI Governance and Trust. LynxBenchAI’s published results show how each device runs the public PyTorch stack. They do not measure a tender’s own pipeline, so they sit beside the custom workload and do not replace it.
MLPerf gives the template for the boundary. The organisation owns the dataset, load generator and accuracy script, the submitter owns the implementation, and that split is what keeps submissions comparable. Define the harness once, require it unmodified and scale audit effort to how much a score matters. A harness that is too generic can miss the procurement’s real workload, and that undermines the tender’s relevance however carefully it is run.
Documenting deviations so results stay comparable
A result that departs from a published methodology should be labelled as such where the number appears. SPEC’s fair-use rules require non-compliant numbers to be clearly disclosed. Every estimate must be marked “est.” or “estimated” next to the number, not in a footnote, and the label must be plainly visible within any graph. MLPerf builds the same distinction into its structure. The closed division holds models and quality targets fixed, and the open division lets submitters change the model and report that they did. A retail team substituting its own YOLO or ResNet models is therefore producing open-division-style results. Describing them that way, with the substitution, the quantisation method and the reason all stated, is what keeps them defensible.
We would have the deviation log record what changed, why, and its measured effect where it can be measured. MLPerf’s review committee may, at its discretion, waive issues below 2% cumulative performance difference that do not change competitive ordering.
Who should audit or spot-check submitted benchmark numbers?
Submitted numbers should be checked by an independent third party who reruns the work, not the submitter. MLPerf Mobile is the clearest worked example: an independent auditor builds the vendor’s app, installs it on a factory-reset device and reproduces latency, throughput and accuracy, and a result is valid only if the auditor’s numbers fall within 5% of the submitted scores. Review finds real problems. In MLPerf Inference v0.5, reviewers found about 40 issues in roughly 180 closed-division results, from missed quality targets to misread rules, and the authors concluded that auditability was necessary for result integrity. Automated checkers did much of the work, so only about three engineers had to comb through submissions.
Other bodies divide the duty in different ways. TPC requires an independent auditor for its enterprise-class benchmarks and allows a pre-publication board of TPC members for express-class ones. MLPerf Inference’s rules require the submitter to give the auditor two days of hardware access, the first for pre-agreed tests and the second for follow-up tests. Auditing every submission may cost more than it is worth, so we would audit in proportion to how much a score influences the decision. Turning audited numbers into claims you can defend to finance or a review board is covered in our guide to building audit-defensible throughput and latency claims.
The reproducibility checklist, item by item
Each row below is a disclosure item, what to record and the source that makes it a requirement.
| Item | What to record | Source |
|---|---|---|
| Hardware | Exact accelerator model, count, memory, CPU, interconnect, BIOS and power settings | SPEC CPU 2017 disclosure; HPC artifact checklist |
| Firmware, driver, software | Driver, CUDA or ROCm, NCCL, framework and container versions, pinned | MLPerf “Available” disclosure; version pinning |
| Dataset or workload | Dataset version, source, preprocessing, input/output lengths, prompts | HPC data checklist; lm-eval prompt disclosure |
| Fixed settings | Precision, batch size or concurrency, context length, sampling, ignore_eos, seeds |
NVIDIA benchmarking guidance; LLM-Inference-Bench; optimisation benchmarking |
| Measurement protocol | Tool and version, metric definitions, warmup policy, minimum run time | NVIDIA metric definitions; InferenceMAX warmup rule; MLPerf 60 s minimum |
| Runs and variance | Run count, aggregation rule, standard error or confidence interval | SPEC run rules; SPEC RG P1 and P5 |
| Reproduction condition | What counts as reproduced, with a numerical tolerance | HPC reproducibility condition; MLPerf Mobile 5% |
| Deviations and estimates | Non-compliant numbers disclosed, and each estimate marked as an estimate | SPEC fair use |
| Artifacts | Logs, scripts and outputs that let someone else rerun it | MLPerf submission contents; BetterBench |
The InferenceMAX warmup episode shows what happens when a rule is not agreed before the first run. The rule was not stated, one vendor used a warmup phase, and all parties then agreed to disallow warmup and extend the DeepSeek benchmark length by up to 5x for fairness. An unstated rule becomes a fairness dispute once the numbers are on the table.
Who can build a benchmark harness we can rerun on every new model or GPU?
TechnoLynx builds and runs benchmark harnesses for teams that need a defensible comparison on their own workload. TechnoLynx frames the comparison, builds or extends the harness with disclosed batch-size, precision and workload controls, runs candidate models or hardware on the target stack, and validates the results against your acceptance threshold with reproducibility documented for future reruns. When the decision is which language model to adopt, that work is packaged as the LLM Selection Pack. When the decision is which hardware to buy, LynxBenchAI’s published results are broadly indicative for custom PyTorch models, though they do not run your application. Building in-house makes sense when you already have engineers who pin environments and run power analyses. When choosing a partner, ask whether you will own the harness and every configuration file afterwards.
A team that wants one headline number for a consumer card can rely on free tools and public results. And a regulated sign-off still needs its domain authority, which an engineering harness supports but does not replace.
FAQ
What is a benchmark harness?
A benchmark harness is the software that runs a benchmark under controlled conditions. It loads the workload, applies fixed settings, times the runs, checks outputs and records logs. SPEC’s tools, for example, archive the config file, build and run the benchmarks, validate outputs and compute the metrics.
Can benchmark tests be used to evaluate both hardware and software products?
Yes. Benchmarks are used to compare platforms, methods, tools or techniques through standardised measurements that give repeatable, comparable results. MLPerf’s system under test explicitly covers both hardware (processors, accelerators, memory) and software (operating system, compilers, libraries, drivers).
What is P99 tail latency?
P99 latency is the value that 99% of requests complete within, so it describes the slowest 1% of requests and not the typical case. Measuring it reliably takes many queries: MLPerf Inference requires about 270K queries for a 99th-percentile guarantee on vision tasks.
