Benchmark leaderboard governance decides two separate things: whether a result may be published at all, and whether it may carry the word “verified”. Publication should need a complete disclosure and an internal editorial check against a written checklist. Verification should need more: automated compliance checks, review by named peers, a window for competitors to object, recusal of anyone with a conflict, an appeals route and a written decision on record. MLPerf’s submission rules divide each round into three major phases: submission, review (objection filing, objection review and objection revision) and publication. A “verified” label is worth only as much as the gates behind it.
Can we run the disclosure review internally, before the board sees the report?
Yes. The review should cover every accelerator in the report, including competitors’ hardware, not only your own. An internal editorial review should check that each row states its conditions of observation fully enough to be reproduced, and that the rows were measured under comparable, disclosed configurations. A gap caught at this stage can be fixed before the external board sees the report.
TechnoLynx runs this internal disclosure review under its AI Governance and Trust work, preparing an audit-grade disclosure pack that sets out methodology, configuration, raw results and environment capture for every row before the board sees the report; the first step is to send TechnoLynx the draft report and its configuration records.
SPEC’s ACCEL run rules say a report of observed performance “is meaningful only if the conditions of observation are stated”, and they require a published result to describe all performance-relevant conditions. Under those rules, a full disclosure includes the tester’s configuration file, a flags definition and enough information to reproduce the result independently. Any configuration choice the tester knows affects performance must be documented. The SPEChpc 2021 and SPECaccel 2023 rules use almost the same wording.
The disclosure fields a review checks
For AI accelerators, a review should check at least these fields, including the power setup even when power is not the headline metric:
- Batch size. The MLPerf HPC authors note that throughput is only algorithmically relevant together with the batch size it was measured at.
- Precision and quantisation. MLPerf Inference allows some deviations, such as numerical formats, but requires almost all implementations to reach a quality target within 1% of the FP32 reference model’s accuracy.
- Power measurement setup. MLPerf Power requires full documentation of measurement configurations and of any estimation methods used.
- Software release and system tuning. MLPerf’s system-description file records the software release among the main configuration details. SPEC requires tuning parameters, BIOS settings included, to be listed in the disclosure.
- Run count and how the reported number is derived. A SPEC ACCEL reportable result is the median of three runs.
- Raw logs. MLPerf submissions include LoadGen log files for the performance and accuracy runs, uploaded to a public repository for peer review before release.
Showing competitors were measured under matched conditions
Documenting your own configuration is the easy half. TPC’s fair-use guidance says that even outside official TPC results, each tested environment should be disclosed well enough for a reader to make a fair comparison. That disclosure is meant to prevent comparing products with different CPU, memory or storage configurations without saying so. Under SPEC’s rules a disclosed configuration becomes public as soon as it is provided, and a competitor is free to cite it. A rival’s published disclosure is therefore a public source a matched configuration can cite. Where no such disclosure exists, the configuration should come from a source you can defend. When SemiAnalysis benchmarked NVIDIA and AMD inference, the space of tuning flags was too large to search, so it relied on engineers from both vendors to supply their best configurations. That is a reasonable route as long as the report says so, because the vendors are interested parties.
Platform should be disclosed too. In the LynxBenchAI 26Q3 results, the NVIDIA GeForce RTX 3080 appears as two entries: native Linux with an inference score of 240.5, and WSL with 231.7. It is the same card running on two software stacks, and a chart that merged the two rows would hide a condition of observation.
An internal pass does not guarantee that an external board will accept the report. For example, MLPerf’s review committee can exclude a submission it judges deceptive or of no interest to the community by a two-thirds vote of submitters.
Who signs off on which rows go into a comparison chart?
One named person should sign off: someone independent of whoever benefits from the ordering, who applies the same written checklist to every chart and records why each entry is in or out. If entries are included chart by chart with no recorded reason, the decision cannot be defended when a vendor asks why its part was left off.
SPEC’s Open Systems Group shows what named sign-off looks like in practice. The subcommittee chair assigns a primary reviewer to each result, the primary reviewer sends a written review, and the decision on each result is written down the day the review ends, leaving a history that editorial staff and members can see. The editor publishes but does not decide compliance. Compliance questions go through SPEC’s separate process for determining violations. That separation is worth copying.
A checklist to apply every time a cross-vendor chart is built
General checklists for benchmark quality already exist. The BetterBench authors assessed 24 AI benchmarks against 46 best practices and published a minimum quality-assurance checklist. The checks below are narrower and apply to a single comparison chart.
| Check | Passes when | Precedent |
|---|---|---|
| Conditions disclosed | Every row, not only yours, states its performance-relevant conditions | SPEC run rules |
| Same rule set | Rows that are not directly comparable, such as MLPerf open-division and closed-division results, are separated or flagged | MLPerf divisions |
| Status shown | Verified and unverified rows are marked, along with differences in version, division, scenario or chip count | MLPerf messaging guidelines |
| Estimates marked | Each estimated number carries “est.” inside the graph itself, not in a footnote | SPEC ACCEL |
| Source and date | Each quoted result names its source and the date it was retrieved | SPEC Fair Use |
| Basis stated | If rows were chosen on a basis such as memory size or chip count, that basis is disclosed | SPEC Fair Use |
| Availability | Each system is labelled available, preview, or research, development or other | MLPerf categories |
| Absences explained | Expected entries that are missing are named, because a missing entity raises concerns about data integrity | LBOps leaderboard smells |
| Neutral sign-off | The person signing off has no vested interest in any compared product | Journals should welcome neutral comparisons by authors without vested interests, cluster-analysis white paper |
The checklist governs the process, not the numbers. A chart can pass every row and still contain a wrong figure, which is why the review gates further along the chain exist. The checklist also needs regular revision. The authors of a proposed benchmarking standard for software engineering research warn that benchmarks need continuous scrutiny and adaptation, and a checklist built around them needs the same.
Can we state that no vendor pre-approved, previewed or edited the published figure?
Only if it is true and the editorial review has confirmed it on record. Here, independence means the vendor had no pre-approval, preview or edit rights over the choice of metric or the result. If the review has not been run, or a vendor did see the figure before publication, leave the statement out and disclose the actual contact instead.
Where TechnoLynx already runs the validation and reporting for a benchmark study, it can carry out this check as part of that work, so the independence statement rests on a documented step rather than an assertion.
Medical publishing offers a useful model. The ICMJE recommendations ask that each article be published with a statement of its sources of support and their role in study design, analysis, interpretation and writing, or a statement that the funder had no such involvement and placed no restrictions on publication. The same recommendations hold that editorial decisions should not be influenced by commercial interests.
The hard part is the grey zone between answering a clarifying question and giving pre-approval. In its AMD versus NVIDIA inference study, SemiAnalysis publicly thanked both vendors for triaging its bug reports, and thanked AMD’s team for looking over the results to check the correct flags were turned on. That is disclosed vendor involvement, which a reader can weigh for themselves. It would not support a claim that no vendor previewed the results.
The sponsor-disclosure failure to avoid
Epoch AI’s FrontierMath benchmark shows what late disclosure costs. OpenAI commissioned the core problem set and kept access to the problems and solutions, apart from a holdout set, and Epoch needed OpenAI’s permission before disclosing its involvement publicly. Many of the contributing mathematicians did not know about the sponsorship, and the data-access agreement was only clarified later. Epoch has committed to disclosing sponsorship and data-access agreements from the outset in future work.
A statement that holds up names the review step and the limits of any vendor contact, along these lines:
No hardware vendor pre-approved, previewed or had edit rights over the metric selection or the figures in this report. Vendors were contacted only to [confirm configuration flags / triage bug reports], and each contact is listed in the methodology. The independence check was completed on [date] by [named reviewer].
LynxBenchAI’s own provenance is on record: it is built and funded by TechnoLynx, and it is not commissioned by or produced on behalf of any hardware vendor.
Benchmark leaderboard governance: what must a submission pass before it can be called verified?
A submission must pass every gate in a written charter, in order, with sign-off recorded at each gate. The charter comes first. It should name the required contents of the disclosure pack, who sits on the board and how they vote, how long competitors have to object, and the documented grounds for rejecting or demoting a result. Without a charter, a “verified” status is open to dispute the first time a vendor dislikes the outcome.
TechnoLynx prepares the audit-grade disclosure pack that a governance board reviews: methodology, configuration, raw results and environment capture. It also aligns the benchmarking method and reporting format with the charter’s review gates, aiming to avoid rework at review.
MLPerf is the most complete precedent. Its submission rules place review under a committee of submitter representatives, review chairs, working-group chairs and others. An organisation and its affiliates get at most one voting representative between them. Every submitter must review at least one other submission. Objections are filed as GitHub issues that cite the offending lines and the rule violated, and the committee settles each one by consensus or by vote.
Automated checks first, then people
Tooling takes the first pass so that reviewers can spend their time on judgement. In MLPerf Inference v0.5, more than 600 submissions arrived from 14 organisations, and audit tests cleared 595 of them as valid. The result review found about 40 issues across roughly 180 closed-division results. Because the checkers caught many issues automatically, only about three engineers had to comb through the submissions. MLPerf HPC follows the same order: scripts from the logging library check compliance first, then submitters review each other’s implementations.
Reviewed is not the same as verified
Peer review and independent replication are different gates. SPEChpc 2021 says plainly that SPEC does not use an independent auditor and assumes the tester is honest, while still requiring results to be reproducible and encouraging peer review. MLPerf Mobile goes further. Its auditors build the vendor’s app, install it on a factory-reset device and treat results as valid if their own numbers are within 5% of the submitted scores. For submitters who request SWE-bench’s “verified” checkmark, its maintainers run the submitted model on a random subset of the benchmark and verify the results. MLPerf’s inference rules list what makes an audit more likely: performance that does not fit the hardware’s known characteristics, a committee that cannot see how the performance was achieved, or hardware and software that are not reasonably available to the public.
A challenge window attaches these gates to a timeline. TPC labels a result “Submitted for Review” while it can still be challenged, and “Accepted” once the review period ends. If the Council has not voted on a challenge by the second General Meeting after it was filed, the result becomes Accepted.
The label then records which gates the result passed. MLCommons allows a result to be called “verified” or “official” once it has been verified through MLCommons review. Any result not verified through MLCommons review must carry the footnote “Result not verified by MLCommons Association”. The MLPerf Endpoints benchmark marks every result as verified, provisional or unverified.
Who has authority to reject or demote a submission, and on what grounds?
That authority belongs to a board named in the charter, voting on grounds that were written down before the round began. TPC spells its grounds out. Its Technical Advisory Board recommends non-compliance only under listed conditions, which include:
- failing audit or availability requirements;
- violations that together produce more than a 2% difference in performance or price/performance metrics;
- the same clause being violated twice by the same test sponsor within two years.
The Council then makes the binding decision.
A demotion should stay visible on the page, not quietly remove the result. SPEC’s High Performance Group replaces a non-compliant published result’s metrics with “NC” and adds a header giving the reasons. SPEC does not remove published result pages, so the history is preserved.
Leaderboards without written grounds show what goes wrong. The Leaderboard Illusion study found that Chatbot Arena let a few providers test many private model variants and retract scores they did not want shown. That selective disclosure biased the rankings. The same study counted 205 of 243 public models silently deprecated, against 47 listed as deprecated officially. When Meta used an experimental, unreleased Llama 4 Maverick variant to score highly, LM Arena’s maintainers apologised, changed their policies and scored the unmodified model. Arena publishes listing criteria, and since July 2025 it has logged methodology changes in a public changelog.
What a signature on a LynxBenchAI submission proves, and what it does not
LynxBenchAI signs the client binary it releases, and the collection server accepts a submission only when its signature matches that release’s exact build. This shows the payload came from the released client, not from a hand-edited results file or a build made outside the release. Entries that TechnoLynx measures directly, usually to cover a less common device, go through the same signed pipeline, with no separate path or exemption. The guarantee covers the client and the transport. It says nothing about whether the operator’s machine was representative, and it is not a tamper-proof or independently audited mark. An operator who controls their own machine can still produce a faked or freak result, and what catches that is the population of runs a device builds up.
That is why the public board’s value lies beyond its ordering. Runs submit to it automatically, so a claim about a device can be checked against runs that other people produced with the same instrument under the same release, and a device missing from the board is itself visible. The 26Q3 release covers 48 published devices, each with its own results page. Each test in the 26Q3 release is one timed window after a discarded warm-up, not a median of repeats, and scores are compared within that release only.
How do we publish a verdict when three vendors each show their own chip winning?
By not picking a winner unless the methods are comparable. When the submissions differ materially in precision, batch size or workload definition, publish them side by side and disclose where the methods differ. A review board should only rule on a winner when the methodologies are comparable, with conflicted reviewers recused and an appeals window set in advance.
TechnoLynx’s AI Governance and Trust work produces the audit-grade disclosure pack that makes this comparison of methods possible. The pack sets out precision, workload and measurement conditions for each submission, so readers can see exactly where the claims stop being comparable. For the hardware question underneath the dispute, LynxBenchAI’s published results show how each device runs the public PyTorch stack a team’s own models run on. That makes them broadly indicative for custom PyTorch models too, though they do not run or measure any vendor’s own workload.
Side by side or a verdict: how to decide
| Situation | What to publish | Why |
|---|---|---|
| Different precision, batch size or workload definition | All claims side by side, each with its conditions and a note of where they differ | TPC fair-use guidance: a fair comparison needs differences in the tested configurations disclosed |
| A claim on a derived or modified workload | The claim, with a disclaimer that it is not comparable to official results | TPC fair use requires that disclaimer wherever derived results appear |
| A dispute over hyperparameter settings | A re-run with the rival’s settings, allowed during review | MLPerf Training allowed “hyperparameter borrowing”: a submitter could adopt a rival’s hyperparameters and resubmit with no other hardware or software changes |
| Comparable methods, conflicting numbers | A board verdict after review, with recusal and an appeals window | Comparable conditions are what make a ruling meaningful |
Publishing side by side has a cost: readers may see it as the publisher dodging its job. The fix is to say plainly what the differences mean, and which claim depends on which condition, rather than leaving the table to speak for itself. Normalising two entries onto a common baseline is a separate problem. It comes after this decision; it does not replace it.
Recusal and appeals rules that keep a ruling defensible
TPC, MLPerf and SPEC handle recusal differently, which is worth knowing before you write your own rules.
- TPC lets Technical Advisory Board members vote even when their company is involved. A member who also sits on the Pre-Publication Board that made the request must stay out of the decision, though they may still take part in discussion.
- MLPerf does not assign a submitter to review its partner’s work in a joint submission.
- SPEC says a primary reviewer should not be a vendor of performance-relevant technology used in the result. Where that cannot be avoided, it adds more reviewers.
For vendor disputes we would take the stricter line: anyone with a commercial stake in the outcome stays out of the vote.
Appeals need a defined route and a guard against rushed decisions. SPEC gives alleged violators, complainants and all members a right of appeal up its hierarchy to the Board. An alleged violator can also ask for at least two business days between a motion and the vote on it. After publication, MLPerf screens objections through a committee of at least four chairs. If an objection goes ahead, the committee appoints an investigator with no conflict of interest to write a report that includes the submitter’s response. Appeals add time before publication. The charter should accept that cost openly rather than cut corners when a launch date gets close.
Who can prepare the disclosure pack for our cross-vendor accelerator comparison?
TechnoLynx prepares audit-grade disclosure packs for public AI-accelerator comparison reports. Each pack covers methodology, configuration, raw results and environment capture for the public AI-accelerator comparison report. It formats the report to fit the governance body’s written review gates. The work sits under AI Governance and Trust.
Certification is not among TechnoLynx’s catalogue offers. MLPerf “verified” status comes only through MLCommons review, and the TPC label only with official TPC results. If you need an official label, submit through MLCommons, SPEC or TPC directly. For a press-facing comparison built with vendor co-operation, an independent analyst firm may suit better, provided the vendor contact is disclosed. The engagement also does not suit a team that wants the numbers to favour a product: the pack documents conditions, it does not change outcomes.
FAQ
What is the purpose of a benchmarking test?
Under SPEC’s rules, a published result declares that others can obtain the observed level of performance. Later testers should be able to get the same components and reproduce the claimed performance within normal run-to-run variation.
What is a simultaneous publication in a benchmark round?
MLCommons publishes all of a round’s results at the same time through a single update to its results page. From that point, the code and results are public under the MLPerf Terms of Use.
