A throughput or latency claim holds up under audit when another team can rerun it and tell from the numbers whether a gap is real. A reproducible benchmark methodology needs four things for that. The first is the full software stack, disclosed down to driver, CUDA toolkit, cuDNN, NCCL, container digest and precision settings. The second is raw, untruncated logs kept next to the summary figure. The third is enough repeated runs to state a noise band. The fourth is a versioned rulebook that says how comparisons are made and who may change them. Without stack disclosure, the number can hardly be reproduced. Without the last two, nobody can say whether a rerun that came out differently is a failure or just noise.

MLPerf and SPEC CPU, the audited industry rulebooks, already require stack disclosure, logs and multiple runs before a result can carry their name, and MLPerf also states numeric tolerances. What follows takes each requirement in turn, gives the measured evidence behind it, and covers the two governance questions that come up as soon as more than one party submits numbers: independent verification, and who owns the rules.

What a reproducible benchmark methodology has to disclose

Disclose every layer between the model code and the silicon. The kernel driver, user-mode CUDA driver, CUDA toolkit, cuBLAS, cuDNN, NCCL, framework wheels, container images and container toolkit each have their own version and release cadence. If any one of these factors goes undescribed, along with the input, compiler, runtime environment, machine and measurement method, results can hardly be reproduced and may even mislead.

LynxBenchAI publishes cross-vendor results showing how each device runs the public PyTorch stack, giving a disclosed hardware reference before you pin your own layers; the first step is to check whether your target device already appears in its results.

TechnoLynx’s AI infrastructure and production SaaS work covers throughput and latency claims that have to hold up under audit. TechnoLynx identifies which stack elements a particular committee, customer or release gate needs pinned and logged, configures the benchmark harness with pinned versions, logging and accuracy-delta gates on the target hardware, then runs repeated trials to set the noise band and puts change control and log archival in place so later submissions stay auditable. The first step is framing: working out which of driver, CUDA, cuDNN, TensorRT-LLM, NCCL, container image, quantisation and batch size actually matter for the requirement in front of you. Where the question is how a device behaves rather than how your application behaves, LynxBenchAI publishes results showing how each device runs the public PyTorch stack your own models run on. Those results are broadly indicative for custom PyTorch models, but LynxBenchAI does not run your workload.

Disclosure checklist for a GPU throughput or latency claim

Item	What to record	Why it matters
Host driver and firmware	Driver, VBIOS and GSP firmware versions as nvidia-smi -q reports them	The user-mode driver comes from the host, so an image SBOM never shows it
CUDA libraries, individually	cuBLAS, cuDNN and runtime versions, not just “CUDA 13.4”	A toolkit number is not a component version, and each cuDNN version needs a particular cuBLAS
NCCL	Exact release	NCCL ships on its own release stream and is the collective library for multi-GPU training
Container image	Digest, not tag	NVIDIA refreshes CUDA image tags for CVE fixes and deletes them on a published schedule
Versions actually loaded	cuDNN and NCCL as the framework reports them at runtime	An image can hold two cuDNN copies, one from the base layer and one from a wheel
Precision and quantisation	Lowest precision per compute path; the quantisation method	MLPerf Training v6.1 logs lowest precision in linear, attention and communication, and MLPerf Inference requires the quantisation method to be described well enough to reproduce
Serving configuration	Engine version, flags, max context length, resolved dtype	One published protocol disclosed vLLM 0.9.1, CUDA 12.4, PyTorch 2.4.0 and that --dtype auto resolved to bfloat16
Run lifecycle	Warm-up, restarts, gaps between runs	The same protocol excluded a 30-second warm-up and restarted the server for each context size
Hardware topology	Interconnect, accelerator bus, memory bandwidth	Readers who cannot access the machine need these to interpret the result

Disclosure is not paperwork for its own sake. SemiAnalysis’s InferenceMAX team reported that a vLLM container image left B200 instances stalled for up to 30 minutes. The cause was an NCCL build without prebuilt Blackwell SM100 code, which forced JIT compilation. Other users of the identical image saw no stall, because their cached compiled kernels sat in a mounted home directory. The same image gave two very different results, and the difference lay in the environment.

Quantisation backends need the same care. The llama.cpp perplexity documentation warns that its numbers are not directly comparable with other projects, because exact values depend strongly on implementation details. If a tool does not record the quantisation level and backend build automatically, record them by hand and treat any unrecorded field as a reason the result cannot be reproduced.

Run-to-run variance on identical GPUs

Identical GPUs running identical work produce measurably different execution times. Across five GPU-rich clusters, Sinha et al., as reported in the AI Benchmark Carpentry paper, found an average performance variability of 8%, a maximum of 22%, and outliers up to 1.5× slower than the median GPU. Across 128 NVIDIA L40 GPUs, inference throughput differed by up to 27.7% between fastest and slowest, and a 7.7% spread persisted within one 8-GPU node over a week. Dynamic voltage and frequency scaling reacts to each GPU’s thermal state, so GPUs in the same node can run the same workload at different clocks.

In batched GPU simulation, fixed seeds do not remove that variability. The GPUSimBench authors fixed random seeds, disabled task-level randomisation and still measured run-to-run variability across ten runs, which they attribute to floating-point operation order and solver-path differences. On a consumer stack with an unpinned driver, quantisation level and backend build, we would treat those three as the first suspects. We would also measure a noise band on that exact machine before calling any rerun a reproduction failure. Some variance on consumer stacks may be irreducible without vendor-level pinning support, and the honest response is to bound it and report the bound. Thermal control belongs in the protocol too: MLPerf Mobile requires a room temperature of 20 to 25 °C and recommends cooldown pauses between tests.

How many runs separate a real difference from run-to-run noise?

No run count is universal. You need enough runs to support a statistical test of the difference you intend to claim, and that number depends on how noisy your stack is. A re-evaluation of LLM code-performance benchmarks ran each of 1,538 tasks 30 times and applied a Mann-Whitney U test. Only 6.11% of the “performant” implementations were significantly faster, and 93.89% could not be distinguished from the baseline. The authors cite the long-standing recommendation of at least 30 executions when comparing runtime performance.

The ETH Zurich scientific-benchmarking rules add a warning that sits alongside that advice. Their experiments showed that the 30 to 40 samples some textbooks suggest are not enough to assume normality, so they recommend either testing for normality or using nonparametric methods. Computer performance data is usually right-skewed and multi-modal because of scheduling, congestion and cache misses. The two positions fit together. Thirty runs with a nonparametric test, as the code-benchmark study used, is defensible; thirty runs fed into a t-test on an assumed normal distribution is not.

Smaller counts can still work for regression testing when the protocol is tightly controlled. A published Sequential Isolation protocol for vLLM on an A100 cut the average coefficient of variation from 15.2% to 2.2% using five repetitions per configuration, with 113 of 144 configurations under a 3% threshold. The same authors reported their between-model differences descriptively and made no significance claims at n = 5. The MLPerf Training paper set 5 runs for vision tasks, so that 90% of entries from one system fall within 5%, and 10 runs for other tasks, for 90% within 10%; current rules set the count per benchmark. It drops the fastest and slowest runs before averaging. SPEC CPU 2017 runs each benchmark three times and takes the median, or twice and takes the slower result.

Skipping the statistics has a measured cost. In Georges, Buytaert and Eeckhout’s 2007 study of Java performance evaluation, prevalent methods produced misleading comparisons in up to 16% of cases. A 2026 variance study of LLM code evaluation found that pooling all runs into one interval misstated uncertainty by roughly 2× in either direction, and recommended a two-to-three-run pilot to identify the variance structure.

Procedure for setting a run count on your own stack

1. Pin the stack from the checklist above and record it before any timed run.
2. Run a pilot of two to three repetitions to estimate the variance components.
3. Choose the comparison test for the data you have. Use a nonparametric test such as Mann-Whitney U unless a diagnostic check supports normality.
4. Work out the run count from the confidence level and allowed error you need, the approach the ETH Zurich rules describe.
5. Report the noise band with the result, for example by collecting measurements until the 99% confidence interval is within 5% of the mean.
6. For LLM release gating, pair the speed measurement with an accuracy metric. llama.cpp’s KL-divergence statistics, for example, show whether quantisation adds symmetric noise or genuinely degrades the model.

Accuracy tolerances and the fixed workload suite depend on the workload and the risk, so they belong in the pre-registered protocol rather than in a general rule.

When does a gap from a published result count as a reproduction failure?

We would count a gap as a reproduction failure only when it exceeds two things: the noise band measured on the reproducing stack, and the tolerance the governing rulebook sets. Anything inside both is normal variance. If your numbers differ from a paper by some percentage, compare the stacks before you compare the figures. Undisclosed differences in driver, NCCL or quantisation are candidate explanations, and you cannot tell which until both stacks are written down.

Rulebook or study	Tolerance it applies
MLPerf Mobile audit	The auditor rebuilds the vendor app on a factory-reset device. Results are valid if within 5% of the submitted scores
MLPerf submission rules	Issues under 2% cumulative performance difference may be waived, and Preview results may degrade by up to 2% when published
Sequential Isolation protocol (vLLM, A100)	3% coefficient of variation across five repetitions as a run-to-run stability reference, not a gap tolerance
Tool-calling noise-floor audit	The rerun floor was 0.28 to 1.1 percentage points and the prompt-perturbation floor 10 to 19 points. A reported gap must clear the wider floor

The last row shows why one noise band is not enough. Chen et al. found the perturbation floor 11 to 58 times wider than the rerun floor on the same endpoints. A small gap that survives identical reruns can still disappear when the prompt wording changes. The noise floors also need re-measuring whenever decoding, serving configuration or the grader changes.

A deviation log for a committee is a comparison record. For each declared figure it lists the source and release, the disclosed stack and workload fields, the reproducing stack, the delta, the measured noise band, the materiality threshold agreed in advance, and a verdict. SPEC’s rule for anything outside its run rules is a good model: disclose the deviation clearly and mark the affected metric as an estimate at the point of use, not in a footnote.

Raw logs and artefacts worth keeping for a later audit

Keep whatever would let a sceptic rebuild the number without you. That means the untruncated framework logs, the config files, seeds, the system description and the environmental telemetry, all stored so that tampering is detectable. MLPerf Training requires result logs to include full framework logs and every MLLOG line. A submitter who cannot produce the untruncated, unmodified log when a reviewer asks may have the result removed. SPEC CPU 2017 lets any member request the rawfile behind a public result not marked as an estimate, and it must be supplied within 10 working days with its configuration information.

The MLPerf mandatory file list makes a useful baseline: summary and detail logs, accuracy JSON, user configuration, calibration or weight-transformation code, a replication README and a system-description file. Seeds belong on the list wherever randomness enters. MLPerf requires the seeds for non-deterministic weight transformations, and its load generator builds query traces from a fixed seed. Benchmark carpentry guidance recommends logging ambient temperature and sampled power in CSV, JSON or YAML.

Storage needs tamper evidence. A tamper-evident log is one whose modification can be detected by someone who did not write it, and a database row protected only by access control does not meet that bar. Tamper evidence has a limit as well: it does not prove the entries were true, or that nothing went unwritten. Our piece on turning GPU performance numbers into auditable evidence covers how these artefacts become evidence packages for regulated pipelines. The reproducibility checklist for fair, trustworthy benchmarks turns the disclosure fields into a form teams can run against every submission.

Who can package a benchmark so an external reviewer can rerun it?

TechnoLynx sets up throughput and latency benchmarks an outside reviewer can audit: pinned driver, CUDA, cuDNN, NCCL and container versions, archived logs and a measured noise band. Outside help pays off when the claim faces a committee, a customer audit or a contested comparison. When choosing a partner, ask for a sample disclosure pack and a sample noise-band report before anything else.

This engagement suits neither a team that only needs a one-off number for an internal slide nor one whose claim depends on a certified test laboratory’s sign-off; for the latter, an accredited lab is the right route. If an existing public result already covers your hardware and configuration, citing it with its version and rules is cheaper than commissioning anything. Engineering help also stops short of regulatory authority. We can build and document the evidence, but the acceptance decision belongs to the auditor or committee.

Why vendor and leaderboard numbers need a second, independent reference

Before you publish or act on a vendor or leaderboard figure, check it against at least one independently produced result for the same hardware or model. If the claim is contested, get a neutral re-run under identical conditions. LynxBenchAI serves as that second reference point for hardware: it publishes cross-vendor results on the public PyTorch stack and shows how each device behaves, though it does not measure the reader’s own application.

In one audit, most vendor benchmark rows left disclosure incomplete. A Digital Applied audit of 42 benchmark rows from nine AI vendors, published on pages dated up to August 2026, found no row that was both fully disclosed on the vendor’s page and independently confirmed. Of the 41 rows it could score, only 10 named an evaluation harness, and those came from three vendors. The same benchmark name covered 26% and 88.4% for one model, because the two figures came from different benchmark versions. The audit is careful to call this a finding about disclosure rather than dishonesty, and its recommended stance is to treat vendor tables as claims pending qualification.

The Chatbot Arena leaderboard carries its own selection effects. The Leaderboard Illusion study of Chatbot Arena (LMArena) found that some providers privately tested many variants and could withhold scores, which biases published results through selective disclosure. In one month, as many as 27 Meta models were tested privately before the Llama 4 release. The study also found 205 of 243 public models silently deprecated, against 47 officially listed. Any editorial use of such a ranking should disclose those known limitations alongside the number.

Audited programmes show what independent verification catches. MLPerf Inference v0.5 received over 600 submissions from 14 organisations, and its audit tests cleared 595 as valid. Review found about 40 issues in roughly 180 closed-division results. The validation suite includes a caching test that compares runs on duplicate and unique sample indices, and an alternate-random-seed test that defeats seed-specific optimisation.

Check whether the submission covers a representative workload mix or one favourable case. The code-benchmark re-evaluation found that test inputs designed for correctness were often too small to expose real performance differences. Independent results may use different precision or batch settings from the vendor’s claim, so normalise before comparing. Very new chips may have no independent coverage at all. When a deadline rules out verification, the memo should say so plainly: which figures are vendor-only, which disclosure fields are missing, and that the comparison is provisional.

Why pre-register a benchmark protocol before the first run?

Pre-registration stops cherry-picking because the choices that flatter a result (metrics, workloads, run counts, outlier rules) are fixed before anyone sees the outcome. One two-phase pre-registration protocol for predictive modelling declares the research question, data splits, metrics and baselines before training. It then declares the method, seeds and hyperparameters before testing. The authors suggest that committing timestamped versions of the forms to a code repository is a lightweight but effective implementation. Deviations do not invalidate the work, provided they are stated so readers can tell planned analyses from exploratory ones.

For a throughput or latency benchmark, we would pre-register the workload and prompt set, the metrics and percentiles, the run count, the noise-band method, the materiality threshold and the outlier rule. This supports two reporting rules from the ETH Zurich guidance: report all results rather than only the best, and report how many outliers were removed. MLPerf builds the same idea into scoring by requiring a set of independent runs, explicitly to reduce variance and the potential to cherry-pick.

Can vendor-supplied throttling logs be trusted on their own?

No. A throttling log written and curated by the party whose result it explains is one input, not evidence. Reconcile it against your own measured latency and throughput variance. Prefer a contractual right to audit, or an independent third-party attestation, over raw logs alone. The physical mechanism is real enough. DVFS adjusts each GPU’s clock to stay within power and thermal limits, so throttling does move throughput.

Reconciliation means lining up the vendor’s throttle events against your own time series of latency percentiles, GPU clocks, power and temperature, and treating material discrepancies as grounds to escalate. Benchmark rulebooks show what audit access looks like in practice. MLPerf Inference requires submitters to give the auditor two days of hardware access under NDA: one for pre-agreed tests and one for follow-up tests. When to choose which: audit rights suit ongoing verification, and attestation suits a periodic confirmation where direct access is impractical.

Each route has limits. Reconciliation only detects discrepancies at the granularity and timing of your own measurements. Hash chains and similar mechanisms make edits detectable, but they cannot prove the vendor’s entries were true in the first place. An attestation is only as good as the auditor’s independence and access scope.

Who updates the comparison rules when a new precision format or hardware class appears?

Publish the rule as a versioned document that every submission references. Keep it separate from the versioned harness specification it points to. Name one governance body with authority to issue new versions, and state the grandfathering policy in the rule itself. TechnoLynx’s AI Governance and Trust offering builds approval evidence packs that survive an external auditor, a procurement committee or a regulator. The AI Governance and Trust landing describes the scope.

Released versions must not change. Semantic Versioning states that once a version has been released its contents must not be modified, and any change ships as a new version. Where the hyperparameter verifier and the constraints table disagree, MLPerf Training treats the verifier version on the submission date as the source of truth. Its review decisions do not create precedents. They have to be written into the rules through the normal process.

MLPerf handles new requirements in new rule versions. MLPerf Training added mandatory logging of lowest numerical precision in linear, attention and communication paths “from v6.1 onwards”. It has also issued version-scoped exceptions, such as v5.0-only batch-size rules for Llama 3.1 405B and a v6.0-only allowance for a reference learning-rate bug. SPEC CPU 2017 changed a publication rule at V1.1, after which results had to be measured on the actual system claimed. Unresolved ambiguity caused one benchmark dispute. InferenceMAX had not specified whether warm-up was allowed, a submitter used one, and the parties then agreed to disallow warm-up and extend benchmark length by up to 5×.

For rankings published before a rule exists, grandfather each result under the rule version in force when it was submitted, label it with that version, and require re-evaluation only when the governance body explicitly mandates it. Versioning makes that decision possible to record, but it does not make the decision. The policy still has to be written down.

Profile the anomaly before you defend the number

When a rerun disagrees with a published figure, the defensible next step is profiling, not another batch of reruns. Profiling explains why one variant beats another, separates essential computation from overhead, and supports fair, reproducible benchmarking. The InferenceMAX stall was diagnosed that way. A py-spy trace showed the process stuck in NCCL initialisation, nvidia-smi showed 0% GPU utilisation, and perf top traced the CPU cycles to the PTX JIT compiler library.

A disclosure pack tells an auditor what ran. A profile tells them why the number came out as it did.

FAQ

Does the CUDA toolkit need to match the driver’s CUDA version?

Not exactly. NVIDIA’s backward compatibility lets a newer driver run applications built with an older toolkit, and minor-version compatibility allows an older driver within the same major family, with limits. Compatible is not the same as patched, so record both versions.

How do I find the CUDA, cuDNN and NCCL versions actually in use?

nvidia-smi -q reports the driver and user-mode CUDA version on the host. Inside the container, PyTorch can print the CUDA version it was built for and the cuDNN and NCCL versions it actually loads.

What is MLPerf Inference?

MLPerf Inference is an industry benchmark suite, driven by more than 30 organisations, that sets rules for comparing inference systems across very different architectures. It measures single-stream latency, server throughput under a latency bound, and offline throughput.