GPU benchmark audit evidence holds up only when a reviewer who has no reason to trust you can rebuild the result from the package alone. That package needs the exact hardware and software stack, the seeds, data and configuration, the raw logs, a baseline with a tolerance justified before the run, and a record showing nothing changed afterwards. A headline number or a vendor datasheet figure gives the reviewer none of that. Published benchmark rules already require much of it. SPEC requires a full disclosure detailed enough for a result to be independently reproduced, and the MLPerf Inference rules state that results which cannot be replicated are not valid results.

You need a versioned, reproducible validation method scoped to the specific review it has to pass.

The records GPU benchmark audit evidence has to contain

SPEC’s position is that a reported observation is meaningful only if the conditions of observation are stated, so a published result must describe every condition that affects performance. The checklist below collects what the published rules and validation methods name.

Record	What to capture	Where the requirement comes from
Accelerator	Model name, count per node, connection to host, ECC enabled, memory size, driver name and version	SPEChpc disclosure rules
Full software stack	Versions of every component, the hardware configuration, software stacks, Docker images and settings	MLPerf Inference rules for “Available” submissions
Run metadata	Accuracy metric values, CPU/GPU model and memory, ONNX Runtime, framework and driver versions, dataset name and preprocessing, batch size, resolution, date	AMD Vitis AI accuracy methodology
Seeds	Every seed used in non-deterministic weight transformations	MLCommons submission rules
Dataset integrity	A checksum script, with the dataset unchanged at the start of each run	MLPerf Inference rules
Code and scripts	Benchmark code, setup and execution scripts, system metadata and result logs for each system-implementation pair	MLCommons submission rules
Logs and run config	mlperf_log_summary.txt, mlperf_log_detail.txt and mlperf_log_accuracy.json plus user.conf	MLCommons mandatory files
Tuning	BIOS settings, environment variables, kernel and file-system options, other performance-relevant options	SPEChpc tuning disclosure
Disabled services	Any performance-relevant daemons that were shut down	SPEChpc notes requirement

The last two rows are the ones that go missing most easily, because they live in system images rather than in code. SPEC addresses this directly: a tester who installs from a pre-configured image is still responsible for finding and documenting every performance-relevant customisation in it.

MLPerf adds requirements for throughput and latency claims on load scenario, latency percentile and run duration. Those are covered in depth in our guide to building audit-defensible throughput and latency claims.

Is there a published rubric for benchmark evidence, such as MLPerf’s submission rules?

Yes. The MLCommons submission rules and SPEC’s run and reporting rules are two published rubrics for structuring benchmark evidence. MLCommons rules cover the submission, review and publication process across all MLPerf suites. SPEChpc’s rules exist to make published results meaningful, comparable and reproducible.

Three features of these rubrics are worth copying into an internal or regulated evidence package.

• A fixed boundary between what the tester controls and what the referee controls. In MLPerf Inference, MLPerf supplies the dataset, the LoadGen traffic generator and the accuracy script, and submitters implement only the system under test. The LoadGen has to be compiled from a tagged, approved revision without alteration.
• Automated compliance checks before human review. MLPerf HPC runs mlperf-logging scripts against submissions and log files, and only after that do submitters review each other’s work. When MLPerf Inference v0.5 was reviewed, about 40 issues were found in roughly 180 closed-division results, many of them automatically by checker scripts, and only 166 of those results were released.
• Estimates kept separate from results. SPEC requires any public SPEChpc result that does not follow its run rules to be clearly described as an estimate.

The authors of the MLPerf Inference paper concluded that auditing and auditability were necessary to ensure result integrity and reproducibility. Whatever your sign-off body is, that conclusion is the reason to adopt a rubric rather than invent one.

How do you prove the GPU pipeline gives the same answers as the one it replaced?

You prove equivalence by comparing against a disclosed baseline under a tolerance you can justify, and by testing the properties a tolerance cannot see. Before setting any tolerance, accept that a legitimate GPU result need not match the CPU result. NVIDIA’s floating-point documentation lists the reasons: the GPU has fused multiply-add, parallel algorithms reorder operations, the CPU may compute at higher internal precision, and many maths functions are not required to be correctly rounded. NVIDIA concludes that differences are not automatically evidence that the GPU result is wrong. PyTorch makes the same point at the framework level: completely reproducible results are not guaranteed across releases, commits or platforms, or between CPU and GPU, even with identical seeds.

Moving between GPU vendors also produces discrepancies. In LLNL’s differential testing of more than 600,000 generated programs, discrepancies between NVIDIA and AMD GPUs appeared in 0.98% of FP64 runs and 9.00% of FP32 runs. The causes they traced were maths library calls, FP64 versus FP32 precision, and code converted with HIPIFY.

Where a defensible tolerance comes from

A tolerance a reviewer accepts is one tied to a reason, not copied from another project. Three published approaches illustrate the options.

Quality relative to the reference model. MLPerf Inference requires almost all implementations to reach within 1% of the FP32 reference model’s accuracy. ResNet-50 v1.5 scores 76.46% Top-1, so an equivalent implementation must reach at least 75.70%. The MLPerf authors also relaxed the rule for mobile networks whose accuracy loss was unacceptable without retraining. MobileNet models were trained for quantisation-friendly weights so the window could be narrowed to 2%, SSD-MobileNet’s requirement was set to 22.0 mAP, and retraining was disallowed to keep submissions comparable. A relaxed tolerance is defensible when the reason for the relaxation is published alongside it. AMD’s Vitis AI methodology states its 1% tolerance as 1 percentage point absolute, so an 85.0% baseline accepts 84.0% or higher. Write down whether your tolerance is absolute or relative, because “1%” means different things in the two cases.

Application-level numerical bounds. In the GPU port of the CReSS weather code, the integrated application counted as valid when two pressure-based metrics both stayed below 10⁻⁴. The port reported values of 1.0×10⁻⁵ and 5.6×10⁻⁵.

Tolerances calibrated from the kernel’s own correct runs. One hash-linked benchmarking scheme sets the acceptance tolerance as a fixed margin of 3 times the largest residual observed over correct repeats at a given precision and size. Its authors derive the tolerance from rounding-error analysis and calibrate it to the device’s measured residual floor, so ordinary roundoff passes and a genuine error does not. A study of tensor-kernel tests took the p95 of maximum absolute error across passing cases of the correct kernel and multiplied it by a 1.5 safety factor. On its paired buggy subset, that calibration raised bug-detection recall from 73.2% to 82.4%, at a cost of 20 new false positives in 1,882 controls. The same study found median hand-picked tolerances between 100 and 200 times looser than the calibrated ones.

What a tolerance check cannot tell you

Passing a tolerance check is not sufficient on its own. In the RESOLVE study, two AI-generated kernels for ELU and hardtanh did no computation at all, yet both passed the tolerance tests shipped with KernelBench. RESOLVE also tests for nondeterminism by perturbing execution timing to expose races, and it compares repeated executions bit for bit, so no tolerance is involved at that stage. In a reviewer’s terms, the evidence package should report the tolerance result together with a determinism result and a check that the kernel actually touched memory.

Two-baseline designs also isolate where an accuracy gap came from. Vitis AI separates an FP32 Original Model Baseline, a Quantized Model Baseline on CPU and the accelerator run. If the quantised CPU baseline meets its target but the accelerator run fails against it, the cause is accelerator-specific.

Kernel-level profiling is a further validation layer, as in the CReSS port. Our article on profiling-based validation for GPU workloads goes deeper on that layer than this checklist does.

Which edge cases must the validation suite cover before sign-off?

The suite has to cover the inputs and control paths that a single representative run never reaches. That is the reliability work that catches regressions and edge-case failures before a release gate, and it is a separate discipline from measuring accuracy. Five gaps a sign-off suite has to close follow.

1. Control paths outside the captured snapshot. In the CReSS port, an AI-generated benchmark left out a branch that ran only at an intermediate timestep. The snapshot test missed it, and the error appeared only after integration. The authors note that their kernel artefacts validate the captured states for one scenario and do not certify every control path.
2. Special values. LLNL’s case studies traced discrepancies between NVIDIA and AMD GPUs to maths functions such as fmod and ceil and to variations in handling NaN and Infinity. In one case ceil returned 0 on nvcc and 1 on hipcc, which led to a division by zero on the NVIDIA device.
3. Alternative valid inputs. SPEC treats a runtime optimisation as unsafe if a different but valid input makes the program fail where it would not without the optimisation.
4. Caching and seed-specific tuning. MLPerf detects caching by running duplicate sample indices and comparing them against unique ones. It also replaces the official random seed with alternates and measures the result.
5. Hard classes in calibration data. When a quantised model falls more than 1% below its original baseline, Vitis AI advises verifying that the calibration set covers all classes and edge cases, including occlusions, poor lighting and scale variations.

The CReSS team kept validated GPU kernels switchable by conditional compilation, so subsets could be enabled one at a time to locate a discrepancy. The same switch is worth building into a port meant for sign-off, because it turns “the integrated run failed” into a search the reviewer can follow.

Is a signed benchmark report with raw artefacts enough, or should reviewers get a reproduction environment?

An offline audit of a hash-linked record establishes that the record is intact. It does not prove the measurement, so high-stakes or surprising results also need reproduction access. The hash-linked records paper draws the distinction exactly: its offline audit shows the residuals are the stated functions of the observations and that nothing was altered, but on its own it does not re-run the measurement.

MLPerf’s audit triggers make a practical rule for when to go further. A submission is more likely to be audited if its performance doesn’t fit the known characteristics of the hardware, if the committee can’t see how the performance was achieved, or if the hardware and software are not publicly available. In that situation the submitter should expect to explain the mechanisms involved, give the auditor hardware access to replicate the runs, and possibly provide source access to binary components. The auditor then checks that reproduced results are within reasonable tolerance, by convention MLPerf’s 2% threshold for “materially different” performance.

The lighter option still has teeth. If a SPEChpc result is published without being marked as an estimate, any SPEC member can ask for the rawfiles, and they must be provided within 10 working days. For internal sign-offs, ship the artefact package by default and agree in advance the conditions that trigger reproduction access.

How do you show the model still behaves as validated, release after release?

Keep the reference artefacts frozen, and re-run the cheap checks against them on every release. Reserve full revalidation for changes the cheap checks cannot cover. PyTorch does not guarantee reproducible results across releases or individual commits, so a framework update can change results. The CReSS workflow shows the pattern: when profiling flagged an anomalous kernel, only that kernel was revised, and it was revalidated against the same dumped reference data before going back into the application.

Reference data has a cost. One minimal dump configuration in the CReSS study produced more than 400 GB, so the authors call for a lifecycle strategy that picks representative dump points and keeps only reusable reference artefacts. Separate the claims you hope to re-derive from the ones you cannot. Inputs regenerated from committed seeds let a successor device recover 20 of 20 residual floors and 8 of 8 numerical classes from a retired GPU’s record. That GPU’s 8 throughput figures were properties of its particular silicon and could not be recovered.

What logging and checkpointing make a multi-GPU run traceable?

A multi-GPU run is traceable when every rank’s random state, data order and checkpoint can be restored and compared. One reproducibility guide says to get training deterministic on a single GPU first and only then extend to several GPUs. With Horovod, that also means setting HOROVOD_FUSION_THRESHOLD=0. The same guide says to seed every pseudorandom generator and to make the data loader deliver batches reproducibly, without sharing generator state between threads. In PyTorch, torch.use_deterministic_algorithms() selects deterministic implementations and raises an error on operations known to have none. Setting cudnn.benchmark to False makes cuDNN choose its algorithm deterministically, possibly at some cost in performance.

These settings matter for evidence. In one experiment, a single nondeterministic index_add operation left all 1,000 trained models with unique weights, even though they converged to similar loss. A reviewer looking at such a run cannot reproduce it at all, even on the original machine.

Checkpoint design decides what “restore” means. With CRIUgpu, a checkpoint taken on a 4-GPU A100 system must be restored on another 4-GPU A100 system, and an 8-GPU or mixed system will fail. More generally, compatible topology means the same GPU number, type, memory size and interconnect. PyTorch Distributed Checkpoint takes the opposite approach: it reshards at load time, so a checkpoint saved in one cluster topology can be loaded into another. It writes at least one file per rank. Record which kind you used, because a same-topology restore makes a stronger reproducibility claim than a resharded one.

Redundancy across devices adds an independent check. With two cards of the same model computing identical inputs, the output difference was exactly zero for deterministic kernels. A single injected bit flip pushed that difference from 0 to 1.59×10⁻¹.

Who can build a GPU benchmark evidence package our reviewers will accept?

TechnoLynx builds approval-grade evidence packs and GPU performance engineering work. That covers task-specific evaluations, governance evidence packs aligned to named rubrics and lineage for production decisions, which a buyer, auditor or model-risk committee can read and sign against. We treat that evidence as an engineering output, not a policy document. The release-gate side of the work (regression suites, drift detection, validation packages) sits within our Production AI Reliability practice.

The limits are firm. We do not give legal advice, certification or regulatory sign-off, and we cannot promise audit-readiness. Teams that want a public, cross-vendor comparison can submit through MLCommons, which runs its own peer review and objection process.

Scoring GPU tender bids so the method survives a losing bidder’s objection

A GPU tender’s scores survive challenge when the benchmark method, the scoring rubric and the weightings are fixed and published before any bid is scored, and every bidder’s hardware is then measured under the same protocol. Vendor-supplied numbers are a weak basis for an award decision, because they are not measured under one disclosed method across all bidders.

TechnoLynx defines the benchmark workload set and controls (batch size, precision, data) to match the tender’s real use cases. TechnoLynx then runs the same normalisation methodology independently on every bidder’s hardware, maps the normalised metrics to award points in a disclosed rubric, and packages the raw results, configurations and logs into an auditable record for objection defence. Before writing the tender, LynxBenchAI’s published results give a broadly indicative view of how devices run the public PyTorch stack. They do not run or measure the tender’s own workload.

On the procurement side, the requirement is clear. World Bank guidance requires procurement documents to include the complete evaluation criteria, their weightings and the specific way they will be applied. UK public-sector practice sets the scorecard before the tender is issued and treats a change to criteria or weightings after publication as a serious compliance risk. A score without written justification is close to impossible to defend in a standstill period or legal challenge, so each score needs a short written rationale. Evaluators should also score each bid against the published criteria, not against the other bids.

The challenge grounds tell you what to document. The grounds listed under the Procurement Act 2023 include changing award criteria or weightings after submission, mathematical errors in scoring, and irrational scores. In Geodesign Barriers Ltd v The Environment Agency (2015), the High Court found the authority’s disclosed evaluation documents unsatisfactory, and the case is cited for the need to keep thorough records. For a GPU tender, thorough records mean the raw benchmark logs and configurations behind each score, alongside the rationale for it.

Three caveats belong in the tender file. The protocol has to be fixed before bids arrive, or it will look tailored after the fact. Whether the workload represents the authority’s real use is a separate question from the scoring mechanics, and a disclosed method can itself draw an objection that it favours one architecture. World Bank guidance also warns that demanding criteria in a constrained market risk producing no compliant bids. How to build the measurement harness itself is covered in building a trustworthy, disclosed benchmark harness for GPU/ASIC procurement.

What to log before citing someone else’s benchmark in a public report

An agency citing a third-party benchmark should keep an internal audit log of the verification steps it actually took: what was checked, against what evidence, by whom and when. Where full provenance was not available, it should attach a disclosure note to the published figure. The two answer different questions. The log shows funders and reviewers that the agency did its own checking, and the note tells readers where that checking stopped.

Where the stakes justify it, TechnoLynx can run the independent third-party benchmarking an agency commissions. TechnoLynx then cross-checks the results against the agency’s published figures before citation, and supplies the verification report in a form the agency can archive and reference in future audits. For lower-stakes figures, the alternative is the agency’s own documented verification with a disclosure note.

The log needs a defined structure. One proposed audit-trail schema gives every event an event_id, timestamp, system, actor, event_type, model, dataset and deployment identifiers, a details payload, and the fields prev_hash and curr_hash. Its hash chain links each event to the one before it. In the proof of concept, curr_hash is a SHA-256 digest over the event payload plus the previous hash, starting from a fixed “GENESIS” value. Auditors can replay the chain to confirm the trail has not been altered or truncated since it was written. Content-addressed links have another useful property: anyone holding the records can recompute the head, later entries only append, and earlier heads stay valid as the record grows. Human decisions belong in the trail as structured approval and waiver events, not in email threads.

The 2020 verifiable-claims report noted that the absence of standards for measuring compute use makes claims about it harder to verify. The closest published convention is SPEC’s: where its rules cannot be followed, the deviations must be clearly disclosed and the metrics clearly marked as estimated. A note modelled on that convention states the benchmark conditions, the scope checked and the known gaps, and labels the figure as unverified by the agency where that is the case. Applying such notes inconsistently across a report invites more scrutiny, not less.

The 2020 report on verifiable AI claims describes third-party audits as able to confirm or refute developers’ earlier claims. It also argues that standardised compute reporting would let an auditor surface activity that was not reported. An audit trail shows the sequence of events but not what caused what, so the agency still has to interpret the record. Regulated maintenance decisions, such as Part 11 trails for model-triggered equipment replacement, need their own treatment beyond this logging pattern.

Defining flash-flood warning lead time so it can be checked

A flash-flood lead-time figure can be checked only when it names its reference event: first alert, sustained alert or a severity-threshold crossing. A handful of replayed historical events is limited statistical evidence, not a settled accuracy figure.

Numerical caveats for a GPU-ported reconstruction pipeline

NVIDIA notes that parentheses or thread counts alone can change a final result. The 2024 study of floating-point non-associativity found that run-to-run variability can approach the tolerances used in high-accuracy molecular simulation tests.