A cross-vendor GPU benchmark result needs a disclosure complete enough for someone else to reproduce it. That means the device and its firmware, the driver, the framework, compiler and container versions, the precision and quantisation method, the batch size (or the rule that chose it), the workload definition, the power cap and thermal state, and the measurement window. SPEC’s accelerator run rules put the principle in one line: a report of observed performance is meaningful only if the conditions of observation are stated. Disclosure is the minimum, though, and it is not enough by itself. The methodology also has to be fixed and versioned before anyone sees a result. Someone other than the submitter has to be able to rerun it, and there has to be a defined way to settle a disputed figure.
What do we need to publish so someone else can reproduce our benchmark numbers?
Publish a disclosure pack next to every results table. It should hold the configuration fields below, the scripts and seeds that ran the workload, the raw logs and the normalisation rules, so a losing vendor or a sceptical ministry reviewer can rebuild each figure instead of arguing about it.
LynxBenchAI gives this kind of comparison a public reference point. Its published results show how each device runs the public PyTorch stack that most teams’ own models run on, so they are broadly indicative for custom PyTorch models too, but they do not run or measure your own workload. For a report built on your own workloads, TechnoLynx can frame which metrics (throughput, latency, memory, energy, cost per unit of work) and which workload mix the decision depends on. It can then fix the normalisation rules, run the candidate accelerators under documented conditions, cross-check the figures against MLPerf-style disclosure norms and package the evidence for release. The first step is agreeing the metric set before any run.
The disclosure checklist for a cross-vendor GPU benchmark
| Field | What to record | Why it moves the number |
|---|---|---|
| Hardware identity | Model, accelerator count, memory size and configuration, host CPU, interconnect | SPEC requires the memory configuration choices a buyer could make to be documented, not just the size |
| Firmware, BIOS and system tuning | BIOS settings, tuning parameters, performance-relevant services shut down | SPEC ACCEL requires tuning parameters, including BIOS settings, to be named in the disclosure |
| Driver | Name and exact version | In MLPerf Mobile, MediaTek’s optimised Neuron Delegate instead of the generic NNAPI driver changed Dimensity 1100 performance by more than 10% |
| Framework, libraries, compiler, container | Framework build, CUDA or ROCm, kernel libraries, compiler and flags, container image | MLPerf requires software versions, stacks and Docker images for every Available submission; compiler flags alone made one Caffe build almost 2× faster than another |
| Precision and quantisation | Datatype per test, and the quantisation method described well enough to reproduce | MLPerf requires the quantisation method to be publicly described at a level where it could be reproduced |
| Batch size or batch rule | The value, or the rule that selected it per device | Scaling the problem size with the hardware without saying so is one of Bailey’s classic ways to mislead an audience |
| Workload definition | Model and exact variant, dataset, input sizes or sequence lengths, pre- and post-processing | ResNet-50 has at least five variants, which is why MLPerf pins v1.5 and its initialisation and augmentation |
| Run conditions | Power cap, ambient temperature, cooldown between tests | MLPerf Mobile’s artifact run-time state is battery power, 20 to 25°C room temperature and 5-minute cooldowns between tests |
| Measurement window | Warm-up handling, minimum duration, what was inside the timer | MLPerf Inference requires at least 60 seconds so power management and DVFS reach equilibrium |
| Runs and aggregation | Number of runs and how they were combined | SPEC ACCEL reports the median of three runs |
MLPerf and SPEC already require disclosures close to this list. An MLPerf Inference submission contains the performance scores, the benchmark code, a system-description file covering accelerator count, CPU count, software release and memory system, and the LoadGen logs for each run. MLPerf’s rules also state that results which cannot be replicated are not valid results.
Should the harness meet that bar when the report is not a formal MLPerf submission? We would adopt MLPerf’s disclosure and replicability requirements and leave out the full submission machinery, unless the stakes justify the extra documentation work. MLCommons itself argues that evaluation rigour should be proportional to the consequences of failure.
The SafetyRepro authors conclude that disclosing a configuration is not, on its own, sufficient to reproduce a score across mainstream evaluation packages today. So the pack should include the harness itself and a replay command, not just its settings. Full disclosure also will not settle whether the chosen workload mix is representative: stakeholders can still contest that choice, and the pack only makes the disagreement visible.
Presenting several metrics so nobody quotes only the biggest one
MLPerf Training reports time to train for each benchmark and declines to give a suite-wide summary score, because any weighting implies a choice that no single population of users shares. Show throughput, latency, memory, energy and cost per unit of work side by side, per test and per precision. A reader then sees where each device leads and where it trails, and no single figure can be lifted out as the result.
When a vendor’s number and the independent number disagree
Publish the methodology gap before any corrected figure. Say which fields differ, then present the independent figure as a measurement under its own stated conditions, not as the vendor’s number “fixed”. SPEC’s fair-use rules set requirements for anyone quoting SPEC results publicly. Quoted results must name a publicly available source, the responsible licensee and the date the data was retrieved. Non-compliant numbers must be clearly disclosed as such, and any basis of comparison has to be stated. Ask the vendor for its raw methodology first. SPEC ACCEL sets a precedent: for any public SPEC ACCEL result not marked as an estimate, any SPEC member may request the rawfile, due within 10 working days. Where a vendor declines, we would describe the discrepancy on the page rather than present it as resolved. The wider process of verifying and disclosing trustworthy benchmark claims covers what to do with a claim once the gap is on the page.
What configuration fields must be disclosed for a fair cross-vendor comparison?
Vendor-submitted numbers are comparable only when the driver, firmware, batch size, precision and workload are disclosed and either match or have been normalised.
LynxBenchAI gives the shortlist a common reference in its published results, measured on the public PyTorch stack, and the first step is checking each vendor figure against the matching entry.
For numbers on your own representative workloads, TechnoLynx can adapt a cross-vendor methodology with explicit disclosure and normalisation rules, run it across the candidate accelerators and SDKs, and triangulate the results against independent figures. It then hands over a reconciliation protocol your editorial or procurement team can apply to future vendor claims. The first step is agreeing the objective and the workloads that matter for the shortlist.
The December 2023 MI300X versus H100 exchange shows why the fields matter. AMD’s blog states that NVIDIA’s rebuttal used TensorRT-LLM on H100 rather than vLLM, and compared FP8 on H100 with FP16 on MI300X. NVIDIA’s own post lists TensorRT-LLM v0.5.0 for its batch-1 figures and v0.6.1 for its latency-threshold measurements. Its build command enables FP8 with a maximum batch size of 14. AMD’s configuration lists a ROCm 6.0 pre-release, a PyTorch 2.2.0 pre-release, vLLM and FP16. Both sides disclosed enough for a reader to see that the competing numbers measured different stacks at different precisions, which is exactly what disclosure is for.
Checking a vendor figure field by field
| Field | Comparable when | If it diverges, print something like |
|---|---|---|
| Precision | Same datatype and quantisation method on both sides | “Vendor figure measured at FP8; comparison figure at FP16. Not directly comparable.” |
| Batch size or batch rule | Same value, or the same selection rule | “Vendor figure at batch 14; comparison at batch 1. Throughput and latency are not equivalent.” |
| Workload definition | Same model variant, dataset, input or sequence lengths, pre- and post-processing | “Vendor used 2,048-token inputs; comparison used a different sequence mix.” |
| Software stack | Publicly released versions of framework, runtime and driver, all disclosed | “Vendor figure uses a runtime not used in the comparison; the gap may reflect software, not hardware.” |
| Measurement method | Same window, warm-up rule, scenario and aggregation | “Vendor reports peak; comparison reports a sustained window.” |
| Hardware and power | Same SKU, memory, interconnect and power cap | “Power cap not disclosed by the vendor.” |
The wording in the right-hand column is our suggestion, not a standard: what matters is that the divergent field is named in plain language next to the number. Bailey’s list of ways to mislead with performance results is still a useful red-flag check. It includes quoting 32-bit results against 64-bit results on other systems, presenting an inner kernel’s speed as the whole application’s, and projecting results to a full system without saying so.
In mobile ML, the software code path a runtime takes drastically changes measured performance. Performance between generic delegates and vendor SDKs can differ by 7× on some chipsets, because of poor or buggy op implementations. A TensorRT figure, a Qualcomm AI Engine figure and an Edge TPU runtime figure therefore describe three code paths. Treat them as unverified until the SDK, its version and the precision are known.
Two vendors used different driver releases: does that invalidate the comparison?
Not automatically. Record both versions and check that both are publicly released. Compare with a caveat when the difference is within reproduction tolerance, and rerun on pinned versions when it is not. SPEC ACCEL offers a reference threshold: testers are encouraged to republish when they learn of a change that reduces an overall metric by more than 5%.
Should we commission an independent or witnessed re-run, and how is a disputed result resolved?
Commission one when the outcome will be contested, and agree the re-run and its tolerance before results exist. MLPerf Mobile’s audit builds the vendor’s app, installs it on a factory-reset device and accepts the result only if the auditor’s numbers fall within 5% of the submitted scores. TPC requires its enterprise-class benchmarks to be audited by an independent auditor before publication. Benchlist settles disputes by having the challenger run the published replay command on their own hardware with the same dataset and methodology hashes. Disputes that land within the reported σ, or that fail to rerun the exact command, are flagged as frivolous and forfeit the bond.
InferenceMAX had not specified whether warmup was allowed. After NVIDIA used warmup and AMD asked if it was allowed, all parties agreed to disallow it and extend the DeepSeek benchmark by up to 5×. If neither vendor will share raw logs, the minimum to insist on is the configuration fields above plus a reproduction you control, either witnessed or independent. A joint witnessed run needs both vendors to cooperate. Under a publication deadline, the fallback is to publish the figure with the unresolved fields named beside it.
For community tables or repeated submissions, a shared template and fixed run script make every result run the same way. MLCommons’ mlperf-sysinfo tool probes CPU, memory, accelerators and framework version, and warns about empty fields such as firmware, software versions, flags and patches before capture. On top of that, we would make driver version, power limit, quantisation format and tensor-core engagement mandatory fields for every submission.
Map a vendor’s cited methodology against LynxBenchAI’s methodology field by field: precision, model and workload, software stack and measurement method. If any key field diverges, the vendor number is printed as non-comparable, with the differences named. LynxBenchAI holds model and precision constant across devices and adapts only the batch size, raising it until throughput plateaus inside a defined noise band. A vendor figure taken at one fixed batch size is therefore a different measurement from a LynxBenchAI score.
Normalising a vendor’s claimed uplift against an entry in the same LynxBenchAI release requires matching workload, precision and platform. The platform alone moves the score: in the 26Q3 results the GeForce RTX 5090 on native Linux scores 985.8 for inference, against 891.8 for the same card under WSL.
Reproducibility artefacts and an official release-identity record are two separate deliverables. The artefact bundle (configs, seeds, scripts, versions, workload spec) lets anyone rerun the test. A frozen, signed-off record of SKU revision, driver, firmware, VBIOS and topology is what lets a number be called the shipping configuration rather than an engineering-sample result.
Sustained, not peak: the measurement window MLPerf requires
MLPerf Inference requires every benchmark to run for at least 60 seconds so that it measures the equilibrium behaviour of power management and DVFS, not a transient burst. On top of the time floor, each scenario sets a minimum workload: 1,024 queries for single-stream, and at least 24,576 samples for offline. The server scenario is run five times, and the reported result is the minimum of the five. For power, MLPerf Power loops a workload that finishes early until it reaches 60 seconds, then averages power over the valid samples. MLPerf Endpoints goes further and requires each point on its throughput curve to run for at least 600 seconds.
LynxBenchAI gives a public reference for how each device runs the public PyTorch stack under one stated measurement method, and the first step is checking a vendor’s figure against the matching entry in its published results.
Time-to-throttle is a different matter. The rules fix a minimum run time, so we treat time-to-throttle and a separately reported transient peak as useful supplementary context rather than part of the compliant number. MLPerf Inference runs two rounds a year, in February and August, so the window and averaging rules should be checked against the current round’s rules document rather than remembered from a past one.
In the 26Q3 LynxBenchAI results, each test is one timed window taken after a discarded warm-up, not a median of repeats.
When a dispute turns on sustained behaviour, write a right to an instrumented re-run into the agreement and name the telemetry to be logged, so vendor-supplied logs are not the only evidence. NVIDIA’s DCGM documentation defines numbered field identifiers for driver version, CUDA driver version and GPU topology, so an audit clause can cite them by name.
Can a losing bidder overturn the committee’s technical score?
A losing bidder can only reopen the score on solid grounds if the method was weak. If the normalisation methodology (batch rule, precision, workload controls) was fixed, versioned and disclosed to every bidder before bidding, and a party other than the committee can reproduce it, there is little left to appeal.
TechnoLynx can define and document that methodology with the technical committee, design the benchmark so it is independently reproducible, run it across the bidders’ hardware under the fixed protocol, and prepare the disclosure package and audit trail an appeal will be judged on. This is part of its AI Governance and Trust work. The engagement starts by defining and documenting the normalisation methodology. LynxBenchAI’s published results serve as a public reference for how each candidate device runs the public PyTorch stack.
Defensible to every bidder
Disclosing batch size, precision and workload settings to all bidders in advance is what makes the score defensible. The known cost is that vendors can tune specifically for the published tests. Bounding the optimisation effort keeps results comparable. LynxBenchAI prepares the benchmarked models once, before any target device is known, ships the same prepared artefacts to every vendor’s hardware, and applies the stricter constraint to all vendors when their kernels disagree about what is permissible. A workload that fails or times out is recorded as a zero rather than dropped. In the 26Q3 results, a precision that produces no result on a device is shown as 0.0, as with every FP8 inference test on the GeForce RTX 3080. The neutrality principle from computational benchmarking applies directly: parameter tuning effort should not favour one method over another. Fixing axis values, combination constraints and metric definitions before results exist, and timestamping them, blocks edits to the rules after the fact.
Would the methodology hold up to a third-party auditor?
MLCommons describes reproducibility from two directions. A peer reviewer should be able to rerun the evaluation from the specification. An auditor should be able to trace who ran it, when, on what system, with what inputs and scoring, what exceptions occurred and how results were approved. MLPerf’s audit guidance tells submitters to expect to provide hardware access so the auditor can replicate runs, and the auditor may request source access to binary components. Instead of seeking a certificate, test the methodology against both questions, and add fixed seeds, disclosed firmware versions and a statement of run-to-run variance.
Power figures need the same scrutiny: nvidia-smi’s built-in sensor samples only about 25% of runtime on A100 and H100. SPEC’s power methodology asks for logging at least once per second, AC uncertainty below 1% and calibration within the past year.
Running the LynxBenchAI Personal Edition on a stock stack a third party can match
A reproducible stock run has three parts. Install the benchmark from its public package on a publicly released driver and framework. Keep the version record that comes with the result. Treat any non-public component as making the run something other than a stock baseline. LynxBenchAI benchmarks AI performance on your own hardware with stock PyTorch, across accelerators, backends and precisions. It measures how your device runs the public PyTorch stack. It does not measure your application, so it fits someone placing their own hardware against published results, not someone validating a production service.
LynxBenchAI declares the environment it installs. A submitter may diverge from it (an optimised PyTorch build, a newer driver, a different runtime) only with versions released to the public. A build produced for LynxBenchAI, or tuned against it, falls outside the methodology. The rule is declared, not technically enforced: the client does not verify that a stack is publicly obtainable. The submission record captures the driver, framework and runtime versions, which makes the rule auditable after the fact rather than at run time. MLPerf draws the same line with its Available software stack, which may contain only components that are well supported for general use.
To confirm you are not running a vendor-tuned build by accident, compare the recorded versions against the vendor’s public releases. Any component you cannot match to a public release should mark the result as tuned. This check has limits: version records go stale, since inference software ships incremental releases, some just days apart. The operating system is part of the system under test as well. The 26Q3 results list the GeForce RTX 3080 on native Linux at 160.5 GT and the same card under WSL at 158.8 GT as separate entries.
Which published methodology should a cross-vendor comparison anchor to?
Anchor to MLPerf’s closed-division rules when the workload maps cleanly onto an existing suite. When it does not, build a mapped methodology on MLPerf and SPEC disclosure norms and publish the mapping. Either way, fix the workload mix, metrics, aggregation rule and disclosure criteria in advance, and commission independent reproduction when a vendor or region is expected to contest the outcome.
LynxBenchAI gives every party the same public reference on the hardware itself, measured on the stock PyTorch stack, and the first step is checking each candidate against its published results.
TechnoLynx can fix those choices with you up front, select or design the methodology, implement a reproducible harness across the target accelerators and precisions, validate the results statistically, and publish the versioned methodology for later re-runs. The first step is framing the comparison problem before any run.
MLPerf’s closed division requires the same models, datasets and quality targets, equivalent pre- and post-processing, and a model equivalent to the reference. It allows calibration for quantisation and prohibits retraining. Open-division submissions must document how they deviate from the closed division. A bespoke mapping deserves the same treatment, with the deviations listed rather than implied.
Which metrics to fix before anyone sees a result
Fix the primary metric per workload, the latency bound, the precision set, the quality target, the run count and the aggregation rule, and record them in a dated methodology version. MLPerf Inference’s quality rule is a good model: almost all implementations must reach within 1% of the FP32 reference model’s accuracy. The representative workload mix is the hardest choice to defend. Derive it from production request profiles where they exist, and publish the rationale. Measure whole systems, not chips: one study found a Pascal system outperforming a Volta system because of a faster CPU-GPU interconnect.
Publishing the aggregation rule
If the rubric requires one aggregate, cite a published rule. SPEC ACCEL’s base metric is the geometric mean of the per-benchmark medians of ratios against a reference system. LynxBenchAI reports training, inference and compute categories plus a GT aggregate, and GT is ordinal, so a GT difference is never read as a percentage.
When a region whose preferred vendor lost pushes back
We would rest the normalisation choices on the disclosure pack, an independent rerun and a measured run-to-run spread. MLPerf Training sets run counts so that, for vision tasks, 90% of entries from the same system fall within 5%. It drops the fastest and slowest runs and reports the mean of the rest. A margin of error comes from repeated runs, and the 26Q3 LynxBenchAI results do not carry one per test, because each test is a single timed window. A procurement comparison that needs a stated margin should repeat its runs.
Kernel microbenchmarks and NCCL collectives
The 26Q3 compute category covers dense and sparse matrix multiplication on a single device and includes no collective tests. For a cluster comparison that procurement and operations both accept, we would sweep ring and tree algorithms across message sizes from latency-bound to bandwidth-bound, and pin the NCCL version. Version matters here: the InferenceMAX team traced a slow resource leak to NCCL 2.26 when CUDA graphs were enabled across many container restarts.
A benchmark report structure procurement will accept
- The decision the report serves and the methodology version and date it ran under.
- The disclosure table for each system, in the checklist format above.
- Per-test results for each precision, before any aggregate.
- The aggregation rule, then the aggregate.
- Run counts and run-to-run spread.
- Deviations and estimates marked next to each affected number. SPEC requires “est.” or “estimated” beside every estimated figure, not a footnote at the bottom of the page.
- Limitations, discussed openly, since a benchmark without them risks misleading readers.
- Replay instructions and the artefact bundle.
Versioning and pinning so a six-month-old result is still comparable
Pin every dependency by exact version and hash, publish a lockfile, version the methodology itself, and state the window within which scores compare. Loose pinning breaks as package ecosystems move on. In a re-execution study of MatBench, only 2 of 28 submissions could be installed and pass import checks from their original dependency metadata. The same authors recommend tools such as conda-lock to produce a fully resolved lockfile, so that environment resolution survives changes in upstream repositories. The SafetyRepro release pins model repositories by commit hash and records a SHA-256 for each downloaded weight file. Pinning has a cost. SemiAnalysis notes that inference software releases can be days apart, so a benchmark taken at one point in time quickly stops representing the latest achievable performance. The answer is to rerun under a new methodology version, not to edit an old one. LynxBenchAI scores compare within a release, such as 26Q3, and not across releases.
Who can run a cross-vendor benchmark that survives a vendor appeal?
TechnoLynx designs and runs cross-vendor accelerator comparisons with an audit trail usable in vendor appeals. That covers framing the metrics and workload mix for your decision, fixing a disclosed and versioned normalisation methodology before any bids or runs, executing it across the candidate hardware under documented conditions, and delivering the evidence pack and audit trail. LynxBenchAI supplies the public reference layer. Its published results measure each device on the public PyTorch stack, and LynxBenchAI lets you benchmark your own hardware on the same stock PyTorch.
This work does not suit every case. If your workload maps cleanly onto an MLPerf suite and published MLPerf results already cover your candidates, reading those results with the caveats above may be enough. If you need a formally certified result, the route runs through the MLCommons, SPEC or TPC submission and audit processes themselves. And if the goal is one headline number by next week, no defensible methodology will deliver that.
FAQ
What is a benchmark report?
A benchmark report is a measured result together with a description of the system that produced it. A SPEC ACCEL report, for example, covers the system, the host, the accelerators and how the benchmarks were built and run.
What are MLPerf benchmarks?
MLPerf is an industry benchmark family whose rules are designed to keep results comparable across systems with very different architectures, developed with more than 30 organisations. MLPerf Inference defines four scenarios: single-stream, multistream, server and offline.
