To reproduce MLPerf results, check out the code at the tag or branch that matches the published round. Then use the same code the submitter submitted, and compare the outcome with the 2% convention MLCommons auditors use for “materially different” performance in an audit. A reproduction is an executed run, so it is stronger evidence than a spec-sheet estimate. A result that has not been through MLCommons review is unverified, and anything you publish from it must clearly disclose that.

How far a reproduction tells you anything depends on what the published number measured: the suite, the division, the scenario and the submitter’s exact stack. Each is covered below, with the checks that let a buyer tell a configuration that will transfer from one that was tuned for the submission.

Reproduce MLPerf results step by step

MLCommons rules make reproduction a defined procedure rather than guesswork: results that cannot be replicated are not valid, and the benchmark code must stay available for as long as the results are actively used.

1. Pin down the exact result. The required footnote on any MLPerf citation gives the suite, version, division, benchmark, scenario and a result ID in the form major-version.minor-version.entry.benchmark.
2. Check out the matching round. The inference documentation names a tag or branch per round, for example git checkout v3.0 to reproduce v3.0 results or the r2.1 branch for v2.1. Current submissions are made from master at any commit after the seed release, and for v4.1 the tag was created from master after the results were published. Auditors ask the submitter to use the same code that was submitted.
3. Pull the submitter’s own implementation. An inference submission holds the performance scores, the benchmark code, a system description (accelerator count, CPU count, software release, memory system) and the LoadGen logs, all public on GitHub. A training submission includes all code and libraries needed to reproduce the training sessions. Vendor guides take the same route: AMD’s v5.0 inference guide clones the submission repository at its v5.0 branch, and NVIDIA’s Training v5.0 guide builds Docker containers from the mlcommons/training_results_v5.0 repository.
4. Rebuild the data and the model exactly. For Training, download the dataset with the reference script and verify it against its checksum. Apply the quantisation the submitter described publicly, which the rules require to be described well enough to reproduce. AMD’s v5.1 guide gives the sequence as download the model and dataset, prepare the Docker container, quantise, then run the benchmark scripts.
5. Build LoadGen from the tagged revision, unaltered. The fixed random seeds are set in mlperf.conf.
6. Run performance, accuracy and compliance. In AMD’s guide, the first run in each scenario measures throughput and the following runs check accuracy and compliance with submission constraints. AMD’s logs report minimum duration, minimum queries and early stopping as satisfied, and SDXL accuracy must fall inside the published bounds (for example, an FID score between 23.0108 and 23.9501).
7. For Training, run the full set. MLCommons measures each training benchmark a benchmark-specific number of times, drops the lowest and highest results and averages the rest.
8. Compare against the 2% audit threshold. Then read the gap using the next section.

Why does a reproduced MLPerf score come out lower than the vendor’s published number?

Because results differ with hardware configuration and operating conditions, and submission configs are tuned for a particular system size. AMD’s own reproduction guide warns that individual results might differ from the submitted benchmarks because of variations in hardware configuration and operating conditions. “Lower” is not guaranteed, though. In AMD’s MLPerf Inference 6.1 round, Futurum (2026) reports that seven partners, including Dell, Crusoe, MangoBoost and MiTAC, landed within 4% of AMD’s numbers on average, and some beat AMD by up to 2%.

NVIDIA says its Training v5.0 configs are tuned for a particular system size, in both optimisation flags and hyperparameters, and that running one on a system of a different size is possible but not guaranteed to perform as well. Training adds randomness on top. Convergence is nondeterministic, so the score has to come from multiple runs, and MLCommons notes that even the multi-run average leaves imaging results varying by roughly ±2.5% and other benchmarks by roughly ±5%. A single training run of yours therefore cannot be held to a 2% line.

The 2% audit tolerance, and what to do with a gap

The MLPerf Inference audit guidelines ask the auditor to confirm that reproduced results are within reasonable tolerance of the submitted ones, and by convention MLPerf treats 2% as the threshold for materially different performance. The auditee’s job is to help the auditor get results “reasonably close” to those submitted. The same 2% appears in the submission rules: an issue that needs correcting but changes performance by less than 2% cumulatively may be waived by the review committee. After publication, one ground for an objection is that fixing the non-compliance cuts performance by more than 5% cumulatively.

We read a gap in that light. Beyond 2%, compare your configuration line by line against the submitter’s before concluding anything about the hardware.

There is a reason to bother at all: executed runs beat arithmetic. In the MLPerf Inference paper, SSD-ResNet-34 needed 175 times more operations per image than SSD-MobileNet-v1, yet its measured throughput was only 50 to 60 times lower. That is why LynxBenchAI treats empirical execution of real software as the reference standard, and specification-based estimates as a fallback for when nothing can be run.

Checking a vendor’s MLPerf number for cherry-picked configurations

A vendor’s MLPerf number can be checked against the configuration dimensions MLPerf’s own rules require to be disclosed, instead of arguing about the headline. Where the number will be disputed, the stronger answers are an independent re-run or full disclosure of your own parameters.

LynxBenchAI’s published results give a procurement panel a device reference for that check. LynxBenchAI’s published results show how each device runs the public PyTorch stack a team’s own models run on, so a vendor figure far out of line with a device’s measured standing is a question worth putting to the vendor. LynxBenchAI does not run or measure your own workload.

A checklist for spotting a non-representative MLPerf submission

• Version, division, category, scenario, chip count, verified status. MLPerf’s messaging rules require any comparison to state differences in each of these.
• Available category. Available hardware must be rentable or purchasable, and the software versioned and supported for general use. An audited system must have shipped to at least one third party.
• Precision, parallelism and micro-batch size. From Training v6.1, every result log must report the lowest precision used in linear, attention and communication compute, each parallelism dimension, the micro-batch size and the config filename.
• Full software stack. Available submissions must specify the software version of all components, the hardware configurations, the software stacks, the dockers and the settings of all components.
• Run selection. A training score is meant to be the median expected result, and running repeated sets of N runs to find the lowest is prohibited.
• Plausibility. Reviewers are more likely to audit a result that is inconsistent with the hardware’s known characteristics, or one where nobody can see how the performance was achieved.
• Is it an MLPerf result at all? Futurum (2026) contrasted AMD’s 3.3x claim, which spanned workloads and configurations AMD selected, with MLPerf deltas measured on fixed configurations under closed-division rules and reproduced by seven parties.

The checklist does not replace an independent re-run. MLCommons itself says cherry-picking cannot always be proven conclusively, and its corrective actions include requesting new runs with exactly the same software and hyperparameters, or withdrawal.

When the vendor says your benchmark is not representative

The answer to a “not representative” objection is a disclosure note that states the stack, workload definition, batch size, precision and measurement method, with the raw run parameters attached as a footnote. MLPerf’s own footnote fields (suite, version, division, benchmark, scenario, date and source of retrieval, result ID) make a good skeleton for that note. A vendor whose production workload really does differ from your test can still fairly object.

When a score drops after a driver or framework update, who can establish whether the vendor’s release caused it?

In our view, a team that can rebuild both the old and the new stack, pinned to exact versions, and run them side by side on the same hardware and data. Rebuilding a vendor’s container to prove a regression is engineering work TechnoLynx does. The software layer alone can move a result a long way: Futurum (2026) reports that AMD’s MI355X delivered up to 38% more GPT-OSS-120B throughput at 8 GPUs in round 6.1 than in 6.0, on the same silicon, with software as the only variable.

The evidence to assemble is a version-tagged, side-by-side export, plus proof that fixing your own pipeline did not restore performance. MLPerf’s rules for a submission that fails its convergence (RCP) test show what deviation evidence looks like: a written explanation backed by A/B experiments, loss curves against the reference, and comparisons of intermediate activations. Without the pipeline-side check, the package shows correlation with the vendor’s version, not root cause.

Who can independently verify a vendor’s MLPerf numbers for us?

TechnoLynx verifies a vendor’s cited MLPerf-style number on a buyer’s behalf, and builds the evidence pack when a regression has to be traced to a vendor release. LynxBenchAI provides the neutral device reference throughout. The work starts by framing the question precisely: which claim, and which configuration dimensions matter for your deployment. TechnoLynx then chooses a benchmark or disclosure method that suits your real workload, runs or audits the comparison on representative hardware, software and precision, checks the results against the vendor’s cited figures and documents every discrepancy. You receive an audit-ready pack with the raw run parameters attached.

This engagement is not worth it for a buyer who only needs a published number explained: the footnote fields and the checklist above cover that. It is not MLCommons review either. Without that review, the result must be disclosed as unverified. Another route is the submitter’s own reproduction recipe: AMD publishes one for its MI325X inference submission and NVIDIA one for Training v5.0.

MLPerf Training vs Inference: two suites, two reproductions

Training and Inference results are reproduced separately: MLPerf Training measures the wall-clock time to train a model on a specified dataset to a quality target, MLPerf Inference measures latency-bounded throughput under a scenario, and each suite has its own rules document.

LynxBenchAI scores training and inference as separate categories for each device, so the first step is to check a device’s standing in the suite you care about before reading either MLPerf figure.

LynxBenchAI’s releases score training and inference as distinct categories. In the 26Q3 results, the NVIDIA H100 80GB HBM3 scores 709.3 in training and 1678.1 in inference, and the two are not interchangeable.

On the training side, timing starts when the system touches any training or validation data and stops when it reaches the quality target on the validation set. The original Training paper allowed up to 20 minutes of model-creation time to be excluded, and it required 5 runs for vision tasks and 10 for others. Under the current training rules, UNET3D needs 40 runs, with the 4 fastest and 4 slowest dropped.

On the inference side there are four scenarios: single-stream, multistream, server and offline. In the 2019 results, the server scenario’s latency constraint cut throughput relative to offline by as little as 3% on some systems and by 50% on others. The dataset, LoadGen and accuracy script are fixed by MLCommons for every submission, and a closed submission must cover every applicable scenario for a benchmark. An offline figure therefore says nothing reliable about server behaviour, and the reproduction has to match the scenario that was cited.

Even where one model family appears in both suites, separate validation runs are still needed, because convergence criteria and latency criteria are different things. Writing a submission specification is the hardware vendor’s job. MLCommons names the rules as the official source of truth, and the submission guide asks for the latest round’s branch or tag.

Rebuilding a submitter’s container: behavioural equivalence, not identical bits

Expect a rebuilt image to differ at the byte level. Test instead whether it contains the same software at the same versions, the functional equivalence researchers treat as essential. A build counts as reproducible only when any party can recreate bit-by-bit identical artefacts from the same source, environment and instructions, and Docker guarantees none of the reproducibility levels researchers have tested.

LynxBenchAI publishes results showing how each device runs the public PyTorch stack, measuring the hardware rather than a vendor’s image, so the first step is to check the device’s standing there before judging a rebuilt container.

A 2026 study of 2,000 GitHub repositories found that only 56% produced any buildable image and that just 2.7% of those were bitwise reproducible without infrastructure changes. A separate study found that, of images that all built between July and October 2023, only 72% still built less than two years later. That study distinguishes three levels: bitwise identity, functional equivalence (the same components at the same versions) and plain rebuildability. It checked functional equivalence by extracting package lists with Trivy from the historical and rebuilt images and comparing them.

For an MLPerf diff, we would compare the package and driver lists first, then the benchmark outputs within a tolerance. Parallel floating-point reductions on GPUs can change summation order from run to run, so identical sources do not guarantee identical numbers. Pinning helps: the MLPerf submission rules accept a commit hash or tag as a way to identify public-repo dependencies, and the Docker research recommends pinning base images and packages, clearing caches and setting SOURCE_DATE_EPOCH.

Writing an MLPerf-style claim that will not be misread

Publishing an MLPerf-style result without a formal MLCommons submission is permitted. Without review, though, the result is unverified, and the messaging guidelines require you to say so in every use of the benchmark name and score.

TechnoLynx builds these disclosure packs as part of its AI Governance and Trust work. The engagement first settles whether formal certification or a methodology-aligned public disclosure is the real goal. It then defines the seed, run-count and power protocol, implements the harness on the target hardware, re-runs to check variance against MLCommons tolerances, and packages the result for publication.

The formal route is a defined process: an encrypted tarball checked by a submission verifier, a required review of at least one other submitter’s entry, and an objection process run through GitHub issues. To use the MLPerf name, a result must follow both the letter and the spirit of the rules. If you skip that process, three labelling rules apply:

• A comparison of an unverified result with a verified one must carry a fixed footnote saying that unverified results have not been through an MLPerf review and may use inconsistent methodologies or implementations.
• You may not imply that your use is verified or official.
• Results may only be compared with results from compatible versions of the same benchmark and scenario, and non-submitters may not publish until two weeks after the official publication date.

Content that breaks these rules must be taken down within 3 business days of MLCommons reporting it.

Reproducibility controls to mirror without submitting

Control	What the rules fix
Fixed LoadGen seeds	mlperf.conf sets the query-sample, sample-index and schedule seeds for every benchmark.
Seed-gaming check	An alternate-random-seed test replaces the official seed, because optimisations based on the seed are prohibited.
Minimum duration	600,000 ms for single-stream, multistream, server and offline scenarios.
Training run counts	A benchmark-specific number of runs, with the highest and lowest dropped and the rest averaged.
Training power	Every participating node is measured at least once per second across the timed portion, with at least 60 samples per run.
Meter accuracy	A meter qualifies at a given accuracy at 1 Hz if it is SPEC or revenue grade, or if it matches such a meter over five one-minute windows under different loads.

In our judgement, power is the hardest control to mirror, because a meter qualifies only if it is SPEC or revenue grade, or matches such a meter over five one-minute windows.

Is the vendor’s submission config a best case that won’t match your deployment?

Check whether the disclosed configuration is one you would actually run. An Available inference result must list the versions of all software components, the hardware configuration, the stacks, the containers and the settings. From Training v6.1, the log also records precision, parallelism and micro-batch size. A best-case setup is usually visible if you read the disclosure instead of the headline.

LynxBenchAI publishes results for each device on the public PyTorch stack, separately for each platform, so the first step is to compare the vendor’s disclosed configuration with the device’s standing on the platform you will deploy on.

LynxBenchAI’s results show the same device scoring differently on different platforms. In the 26Q3 results the NVIDIA GeForce RTX 5090 scores 597.5 GT on native Linux and 545.7 GT under WSL, and 985.8 against 891.8 in inference.

Scale is the first mismatch to check. NVIDIA’s Llama 3.1 405B reproduction sources a config named config_GB200_128x4x112xtp4pp8cp2_cg_dplast.sh, with tensor parallelism of 4 and pipeline parallelism of 8. A result tuned at that size says little about an eight-GPU server. Precision is the next thing to check. Futurum (2026) gave weight to AMD’s disclosure that its GPT-OSS-120B runs used OpenAI’s released MXFP4 checkpoint with an FP8 KV cache, the standard high-performance setup rather than aggressive quantisation.

A public performance claim, such as a figure from an earnings call, can be checked against the same disclosure fields. Look for a disclosed entry whose workload, precision, batch size and stack match the claim, then ask whether that stack is what ships in volume. If no entry matches, say plainly that the claim is not supported by any disclosed benchmark. That is not the same as proving it false. If the system description does not state driver, firmware and power-cap settings, treat that as a gap to close before relying on the number. Under the Training rules, logs from a previous round may be resubmitted only if system.json matches exactly.

Rules version, division and run duration: settle them per suite

Training and Inference each have their own rules document, each defining its own divisions, so the version and division have to be confirmed for each result separately against the rules for that round.

LynxBenchAI scores training and inference as separate categories for each device, so the first step is to check a device’s standing in the suite a result belongs to before confirming its version and division.

Does a Closed-division or Open-division MLPerf result change how far you can reproduce it?

Yes. The inference Closed division requires preprocessing, postprocessing and a model equivalent to the reference. It allows calibration for quantisation and forbids retraining. The training Closed division requires the same preprocessing, model, training method and quality target as the reference. That ties a Closed result to the public reference implementation. In the Open division the model may differ and retraining is allowed. Open results are generally not directly comparable with each other or with Closed ones, and each must document how it departs from Closed. Any comparison involving an Open result must say how it would fail to qualify as Closed.

Versions matter too. The submission guide says to use the latest round’s branch or tag, and for training hyperparameters the verifier version on the submission date is the source of truth.

Does MLPerf Inference require a sustained run, or can a short burst count?

A sustained run is required. The mlperf.conf sets a 600,000 ms minimum, ten minutes, for the single-stream, multistream, server and offline scenarios. AMD’s v5.0 logs show the same minimum in both its offline and server runs. LoadGen stops issuing queries only once the minimum time has elapsed and any minimum query count has been reached. The 2019 paper set a 60-second minimum so that measurements would capture power-management and DVFS equilibrium, and mlperf.conf now sets ten minutes. The 2019 paper gives the minimum duration as the way to capture power-management equilibrium, so check the current round’s text directly if a thermal question depends on it. The LynxBenchAI methodology works differently: each test is one timed window after a discarded warm-up, not a median of repeats.

If a submitter disputes another’s result, the review committee decides the objection, not the raw number. Up to two submissions are audited each round. A post-publication objection can end with the objection rejected, the result moved to Open, or the result removed for intentional cheating. MLCommons keeps a change log of results modified or invalidated after publication. Divergence alone proves nothing, because legitimate differences in systems and environments produce it too.

FAQ

Is MLPerf open source?

The benchmark implementation code must be open-sourced under a licence that lets commercial entities use it freely for benchmarking. The reference models are open source as well.

Is training harder to benchmark than inference?

The MLPerf Training paper names three challenges specific to training: optimisations that raise throughput can lengthen time to solution, time to solution varies widely because training is stochastic, and system diversity makes it hard to benchmark fairly with the same code and hyperparameters.

What does MLCommons do?

MLPerf, an MLCommons trademark, sets the rules and best practices that keep results comparable across very different system architectures. Its general submission rules govern the submission, review and publication process for every MLPerf suite.