A hardware and stack shortlist is only credible if an outside team can rerun the benchmark unmodified. That takes a full artefact set: container images pinned by digest, the exact configs and seeds, a tagged and archived copy of the harness, raw logs from every run, and a disclosure of the hardware, drivers and libraries used. It cannot guarantee bit-identical output across different platforms, though. Across architectures, the most you can honestly promise is agreement within a tolerance.
LynxBenchAI records the stack behind its own published results. The benchmark declares the environment it installs, and each submission records the driver, framework and runtime versions it ran on, so anyone can check a score against its stack afterwards. A practical first step is to run the LynxBenchAI Personal Edition on one candidate machine and read its submission record next to the score.
What goes in a benchmark artefact set an outside team can rerun unmodified
The set has to hold everything a stranger needs to rebuild the environment, start the runs and check the outputs. Nothing should depend on what the author remembers. SPEC’s CPU 2017 rules spell this out. A public result needs a full disclosure of results and configuration details detailed enough to reproduce it independently: the tester’s configuration file, a flags definition, the hardware, firmware/BIOS and software, and every configuration choice that differs from the default. The SPEChpc 2021 rules ask for the same and add that any choice a tester knows affects performance must be documented. The SC25 Reproducibility Initiative puts it in practical terms: a precise description of the environment, an easy way to recreate it (such as a container image), and one straightforward trigger such as a shell script or Makefile.
A useful distinction from the Docker artifact tutorial (2023) is that the experiment scripts are the primary artefacts, while the Dockerfiles are meta-artefacts: they exist to rebuild the setting the results came from. A benchmark package needs both.
Artefact set checklist
| Component | What it fixes | Where the practice comes from |
|---|---|---|
| Container image referenced by digest | OS, libraries, framework build | Docker digests are immutable where tags are not |
| Build recipe (Dockerfile or Apptainer definition file) | How the image was made, readable as documentation | Recipes double as human- and machine-readable documentation |
| Configs and run parameters | Batch size, precision, threads, ranks, flags | SPEC requires the config file and a flags disclosure |
| Fixed random seeds | Generated inputs, data order, query traces | MLPerf fixes seeds and the generator |
| Tagged, archived harness | The procedure itself | MLPerf LoadGen must come from a tagged approved revision |
| One-command trigger | How a run starts | SC25 asks for a script or Makefile |
| Raw logs for every run | Every observation behind the summary | MLPerf lists mandatory log files |
| System description | Hardware, firmware, driver, interconnect, network path | SPEC and MLPerf both require it |
| Expected outputs and tolerance | What counts as a match | SPEChpc checks outputs against a benchmark-defined tolerance |
How do you pin containers and drivers so a benchmark gives the same results on another team’s machines?
Reference the image by its digest rather than its tag. A Docker digest is a SHA-256 hash of the image content and cannot change, while a tag can be reused or moved to a different image. Pulling by digest gets you exactly the image that was originally built. Singularity takes the same approach with a SHA-256 hash: an author can publish the hash with the image so others can confirm they are running identical code.
Inside the image, pin every package to an exact version rather than a range, so a rebuild from the recipe resolves the same dependency set. The HPOBench authors give the reason. Not one of the community benchmarks they collected came with a full list of dependencies, and some had broken against newer releases of scikit-learn and TensorForce.
The image does not contain the host kernel. A container is isolated from the host operating system except for the kernel. So the GPU kernel driver belongs to the rerunning site, and its version has to be written into the system description rather than assumed.
For HPC reruns, guidance on reproducible Dockerfiles points readers to Singularity. On a shared HPC allocation, Apptainer can pull, run and build from most Docker Hub containers without changes. Shipping an OCI image plus its Apptainer definition therefore covers both cloud machines and national-lab clusters. A 2026 power-aware benchmarking study used this pattern across vendors: Apptainer containers built on the NVIDIA NGC PyTorch image with CUDA 12.4+ for NVIDIA nodes, and on the ROCm PyTorch image with ROCm 6.3.0+ for AMD nodes.
A tag can also be reassigned to a different image later. Archiving the image file itself with the release keeps it usable later. The FAIR software paper suggests taking a snapshot from GitHub and archiving it on Zenodo, which is built for archival and reproducibility.
Bit-identical results stop at the architecture boundary
Even a fully pinned stack is not guaranteed to give bit-identical output across platforms. PyTorch’s documentation says reproducible results are not guaranteed across releases, individual commits or platforms, and may differ between CPU and GPU even with identical seeds. The same documentation warns that different scaled-dot-product attention backends accumulate in different orders, so their results can differ in the last bits for the same inputs.
Within one architecture, bit-identity is achievable. A 2026 study of hash-linked measurement records found that two cards of the same Blackwell model returned bit-identical matrix products at every size and precision tested. The same authors note that committed seeds can regenerate the inputs on any device, but replaying a check to bit-exactness still needs the same architecture. Their replays on other cards also showed that throughput stayed a property of the individual card.
In practice, the artefact should state its promise in two tiers. Same hardware, same release and deterministic settings should give bit-identical numerics. Any other hardware should give outputs within a stated tolerance and throughput attributable to that device. Setting seeds is not enough on its own. PyTorch reproduces the same random sequence only when the seed is fixed and every other source of nondeterminism has been removed in the same environment. So the deterministic settings used belong in the configs, too. MLPerf sets a comparable boundary from the rules side: floating-point operation order, random traversal of inputs and rounding are the only acceptable sources of non-determinism.
Pinned software does not remove run-to-run variance
Variance comes from the hardware as well as the software. The OSDI 2018 study by Maricq et al. found that servers sold as identical can show performance differences that statistical tests detect reliably, which makes reproducible experiments harder. The more variance the hardware has, the more repetitions you need to reach statistical significance, and estimating that number well requires substantial prior data. HPOBench makes the same point about containers: preserving the environment does not by itself give replicable results, so users still need enough seed replicates and must account for hardware differences.
SPEC builds repetition into its run rules. SPEC CPU 2017 runs each benchmark either three times, taking the median, or twice, taking the slower run, and SPEChpc 2021 reports the median of an odd number of runs or the lower median of an even number. The SPEC Research Group’s principles go further and ask for the full distribution of measured performance, including the variation around any aggregate.
This matters when reading a LynxBenchAI score. Each test in the 26Q3 results is one timed window after a discarded warm-up, not a median of repeats. If two shortlisted candidates are close on the category you care about, we would rerun both with repetitions on the machines you can access before treating the gap as real.
Versioning the harness so next year’s rerun is still comparable
The harness is part of the measurement, so it needs a version as precise as the image’s. A harness that changes between years silently breaks comparability with earlier results. HPOBench describes how fast the Python ML ecosystem moves: after a major release, a benchmark can stop being usable by the community. Its fix was to keep versioned containers in a registry, with older harness code still available in git.
Tag the exact harness revision behind each result. Maricq et al. identify the code versions for their paper with a single git tag, and MLPerf requires LoadGen to be compiled unaltered from a tagged approved revision. Then archive that tagged revision under a persistent identifier. Zenodo issues a DOI for each version and a concept DOI for the software as a whole, and recommends citing the specific version DOI so others can retrieve the exact artefact used.
LynxBenchAI scores are compared within one release only, such as the 26Q3 results. Citing a LynxBenchAI result with its release name (for example, the 26Q3 results) does the same job as a harness version DOI.
Why ship raw logs rather than only summary numbers?
A summary number is a claim. The raw log is the evidence that lets a reviewer recompute the claim instead of trusting it. MLCommons treats logs as part of the submission: a package must contain system metadata, benchmark code, setup and run scripts, and result logs for every system-implementation combination tested. The mandatory files include mlperf_log_summary.txt, mlperf_log_detail.txt and mlperf_log_accuracy.json. For the end-to-end RAG workload, which the compliance tests cannot check, MLPerf reviewers use a detailed JSON log of every LLM call and retrieval step to decide whether the run was fair.
SPEC enforces the same idea. If a SPEC CPU 2017 licensee publishes a result without marking it as an estimate, any SPEC member can request the rawfiles, which must be supplied within 10 working days. SPEC also reserves the right to act if the rawfile is withheld or shows different performance from the claim. The 2026 hash-linked records work recomputes the residuals and reductions behind its tables from the archived observations with a standard-library checker. How those records support public claims is covered in verifying and disclosing trustworthy benchmark claims.
Per-test records also show precisions that produced no result. In the 26Q3 results, the NVIDIA GeForce RTX 3080 records 0.0 for FP8 on every inference model, meaning that precision produced no result on that executor. The RTX 3080’s inference category score of 240.5 does not show which cells were empty. A reader with the per-test record can.
What do ACM artifact badges check in a benchmark package?
ACM and USENIX adopt at least three badges. Available means the artefact is publicly available in an archival system. Functional means reviewers judged it easy to use, practical and well documented. Reproduced means independent execution supported the authors’ claims. The Available badge typically requires a public snapshot that links back to the paper. A Git branch that may move later does not qualify.
Reviewers download the artefact, read its documentation and the paper, and expect those instructions to be enough to roughly reproduce the paper’s numbers. The SIGOPS review of EuroSys 2022 is explicit that the aim is to validate a paper’s major claims, not to reproduce its exact results. That fits the tolerance-bounded promise above. The same review used checklists for each badge and argued that publishers should settle three questions: when it is acceptable to run on hardware the authors provide, how much automation experiments should have, and which forms of packaging are acceptable.
Hardware access is one of those questions, since the reviewer may not own the device under test. Some evaluations give reviewers remote access to a machine with everything pre-installed. MLPerf’s audit requires the submitter to provide two days of hardware access so the auditor can run pre-agreed tests.
The vocabulary changed in ACM’s Artifact Review and Badging version 1.1 (August 2020). On the recommendation of the National Information Standards Organization (NISO), ACM swapped its use of “reproducibility” and “replication” to match the wider research community. In ACM’s earlier usage, replicability meant a different team using the same experimental setup. Whatever term a venue uses, the goal on this page is the same: a different team running your artefacts and getting your result.
How a reproducible artefact set keeps a cross-vendor shortlist fair
A shortlist is fair when each candidate ran the same procedure under a disclosed stack, so any difference in results can be traced to the hardware and its stack rather than to how carefully each one was set up. SPEChpc 2021 states that purpose: its run rules exist to make published results meaningful, comparable and reproducible. The University of Reading benchmarking principles add that recording programming environments and compiler versions is what allows results prepared on different systems to be compared fairly. Tuning effort needs the same discipline. One guideline for neutral benchmarks is that no method should be favoured through more extensive parameter tuning.
The cross-vendor study above built separate images for NVIDIA and AMD hardware. The artefact set has to pin and disclose both stacks, as the CUDA and ROCm containers above do. LynxBenchAI handles the stack question with a public-software rule. A submitter may diverge from the declared environment (an optimised PyTorch build, a newer driver, a different runtime) as long as every component is a version released to the public. A build made for the benchmark, or tuned against it and shipped to nobody else, falls outside the methodology. A score therefore describes how the mainstream stack performs as distributed. The rule is declared rather than enforced at run time: the client does not verify that a stack is publicly obtainable. Instead, the recorded versions make the rule auditable after the fact. SPEC takes the same line, requiring that components in a published result be generally available, documented and of production quality. MLPerf’s “Available” category similarly requires software versions, hardware configurations, software stacks and dockers to be specified.
The stack does show up in the numbers. In the 26Q3 results, the NVIDIA GeForce RTX 5090 on native Linux scores 985.8 in inference, while the same card under WSL scores 891.8. The 26Q3 results list the two platforms as separate entries. A shortlist that recorded only “RTX 5090” would have lost that difference. How disclosed baselines carry this into vendor comparisons is the subject of disclosed baselines for fair cross-vendor comparison.
Who can give a cross-vendor shortlist a reproducible, version-recorded baseline?
LynxBenchAI publishes benchmark results that record the driver, framework and runtime behind each score, so a shortlist starts from a disclosed stack that anyone checks against its score afterwards.
Which benchmark tool can give our shortlist a reproducible starting point?
LynxBenchAI gives a shortlist published, version-recorded results for each covered device. It scores training, inference and compute workloads across precisions, and its submission record captures the driver, framework and runtime a result ran on. The 26Q3 results cover 48 devices across GPU and CPU categories, each with its own device page. Start from the LynxBenchAI results and leaderboard, then run the Personal Edition on the machines you can access.
LynxBenchAI measures out-of-the-box stack behaviour, not tuned configurations. It runs a fixed set of models, such as MobileNetV3, DistilBERT and Flan-T5 Small, not yours. If you need to measure your own model, the packaging described above is still your work. For CPU and HPC system results reviewed against formal run rules, the SPEC suites are the established route. For audited, peer-reviewed accelerator submissions, MLPerf’s submission process is.
FAQ
Is there a reproducibility crisis in computing research?
Survey evidence points to one. A 2016 Nature survey by Baker found that 70% of 1,500 researchers had tried and failed to reproduce someone else’s work. In computer science, Collberg and Proebsting showed in 2016 that more than half of the work published at venues including ASPLOS, SOSP and VLDB could not be reproduced because the code was missing or would not compile.
