You can trust a published GPU benchmark claim when it meets three conditions. It discloses the full configuration behind the number (hardware variant, firmware, driver and software stack, power cap, precision, batch size, sequence length, benchmark version, subset and harness). Someone other than the vendor could rerun it from that disclosure. And the configuration matches the comparison you are actually making. SPEC’s run rules say the same thing in one sentence: any result used in public needs a full disclosure of results and configuration details sufficient to reproduce it independently. A figure that fails the test can still be reported, but only as a vendor-reported, unverified claim, with the gaps named next to it.

Which configuration details must a published GPU result disclose?

A result has to disclose all performance-relevant conditions, because a performance observation is meaningful only if the conditions it was observed under are stated. The fields below are the minimum we would check before treating a GPU figure as evidence rather than an advertisement.

Field	Why it moves the number
Exact hardware variant and system	In one test, A100 80GB instances from three cloud providers turned out to be PCIe, SXM4 and virtualised GRID parts
Firmware/BIOS and all non-default settings	SPEC’s full disclosure explicitly includes hardware, firmware/BIOS, software and every configuration choice that differs from default
Driver, CUDA/ROCm and library versions	Provider setups for the same GPU name differed in driver and CUDA versions; MLPerf system descriptions list the OS and libraries with their versions
Power cap and clock settings	Power limits override manual clock locking, so a locked clock throttles if it pushes draw past the cap
Precision	The 2023 NVIDIA and AMD dispute turned partly on FP8 on one side and FP16 on the other
Batch size	Batch size one gives the fastest response but far fewer inferences per second than batching under a fixed response-time budget
Input and output sequence length	Both vendors’ Llama 2 70B figures in that dispute were tied to 2,048 input and 128 output tokens
Benchmark version, task subset, harness, effort	These four fields decide whether a reader can even identify which test produced a number
Compilation flags	Flags alone can change performance for the same kernel on the same hardware
Source, test date and retrieval date	SPEC’s fair-use rules require a public source, an identifiable licensee and the date the data was retrieved

The last row matters more than it looks. It turns a number into a dated claim someone can go back and check. The four-field rubric (version, subset, harness, effort) is the one stated in an audit by Digital Applied, which collected 42 benchmark rows that nine AI vendors published on their own pages up to August 2026. In that audit, 29 of the 42 rows were missing at least one applicable field, and only 10 of the 41 scorable rows named an evaluation harness. No row was both fully disclosed and independently confirmed. The audit’s authors are careful to say this measures disclosure, not dishonesty; an under-disclosed number may be accurate, but you cannot know that from the page.

We go deeper into building such a record in full configuration disclosure for AI benchmarks, including which fields a reviewer needs to reproduce a run.

Why vendor figures for the same GPUs disagree

AMD’s launch comparison used vLLM at FP16 on both the MI300X and the DGX H100, with Llama 2 70B at 2,048 input and 128 output tokens. NVIDIA replied with measurements on TensorRT-LLM, gave the command lines used to reproduce them, and argued that a batch-size-one comparison is unrealistic for cloud serving. AMD’s response pointed out that NVIDIA had switched to TensorRT-LLM, compared FP16 on MI300X with FP8 on H100, and converted relative latency into absolute throughput. AMD then claimed a 1.3x latency advantage even with TensorRT-LLM on the H100.

The two sides’ figures differed in software stack and precision, and NVIDIA also disputed AMD’s batch-size-one setup. The Register’s analysis at the time made the same point: the comparisons hinged on the preferred software stack, FP8 versus FP16, and batch size. For a reader, the lesson is mechanical. Before comparing two vendor numbers, line up their footnotes field by field. If precision, stack or batch policy differ, you are not looking at a comparison at all.

The same discipline applies when you publish your own cross-vendor numbers; building trustworthy, reproducible cross-vendor benchmark disclosures covers the publisher’s side of that problem.

One run is an observation, not a verdict

Identical GPU SKUs do not deliver identical performance. A study of five clusters, covering over 90% of the GPUs on each, measured 8% average performance variation (maximum 22%) between GPUs of the same architecture and vendor SKU within each cluster, with outliers up to 1.5x. The variability also depended on the application: 22% for ResNet-50 against 1% for PageRank.

Benchmark rules account for run-to-run variation. MLPerf Training requires 5 runs for vision tasks and 10 for the others, drops the fastest and slowest, and reports the mean of the rest. MLPerf Mobile’s auditors rebuild the vendor app on a factory-reset device and accept the submission only if their numbers come within 5% of the submitted scores. SPEC expects later testers to reproduce a claimed result “within a small range to allow for run-to-run variation”.

So a single vendor run, however carefully configured, tells you what one unit did once. A gap smaller than the unit-to-unit spread measured for that workload is not evidence that one GPU is faster.

How do I read a leaderboard entry to see what was actually measured, and on which release?

Read a leaderboard entry for its release and its conditions first, and its rank last. Rank is the property most exposed to selection effects. Two failure modes are documented. A benchmark name with no version tag can hide very different tests: on one vendor’s own Terminal-Bench v3.0 figure the same model scored 26% of tasks, while Artificial Analysis’ independent v2.1 board shows 88.4%. And a leaderboard that lets providers test private variants and publish only the best one biases scores upwards. Researchers auditing Chatbot Arena showed this, and two identical checkpoints they submitted scored 1052 and 1069.

LynxBenchAI makes an entry checkable on those terms. Every result belongs to a named release, and scores are comparable only within that release. LynxBenchAI signs the client binary it releases. The collection server accepts a submission only when it carries a signature matching that release’s exact build, so a submitted payload came from the released client and not from a hand-edited results file. The signature covers the client and the transport and nothing more. It does not prove the operator’s machine was representative, and an operator who controls their own machine can still produce a freak or faked result. What catches that is the population of runs a device builds up on the public leaderboard. A claim about a device can be checked against runs other people produced with the same instrument under the same release.

The 26Q3 results show why the conditions on an entry matter:

• The NVIDIA GeForce RTX 3080 has one entry on native Linux and another under WSL, scoring 160.5 and 158.8 GT.
• The A100-SXM4-80GB scores 440.3 GT and the A100 80GB PCIe scores 403.4. These are two variants behind one marketing name.
• On the H100 80GB HBM3, DistilBERT inference scores 2611.7 at FP16 and 1362.7 at FP8, so a figure with no precision attached is not one number.
• The A100 and RTX 3080 entries show 0.0 for FP8 inference, which means that precision produced no result on that executor. It is not a slow result.

GT is an ordinal aggregate, so a GT difference should not be read as a percentage. Each 26Q3 test is also one timed window after a discarded warm-up, not a median of repeats, which is why we read a device’s run population before any single entry.

Did the press-kit figure come from the stack the vendor ships in volume?

Check whether the configuration behind a press-kit figure is the one you can buy, install and support. If the vendor will not say, treat the figure as a best case. A vendor runs more configurations than it presents. If forty were tried and the best one reported, the reported number is biased upwards by selection alone. Precision Federal’s checklist therefore asks how many configurations were evaluated, and whether the baseline got the same tuning budget as the product, since a comparison against an undertuned alternative is the commonest way a real number becomes a misleading claim.

Benchmark bodies handle the “ships in volume” question through availability rules. MLPerf’s available category requires hardware that can be rented or bought, and software that is versioned and supported for general use, specifically to discourage benchmark-specific engineering. SPEC allows pre-production results only if all components will be generally available within three months of first publication. SemiAnalysis’s MI300X study listed the builds it tested, from public stable releases to a hand-crafted custom image and an unmerged development build, and said readers should judge the results on the public stable releases.

To tie a press-kit figure to a leaderboard entry, three questions settle most cases:

1. Same model, dataset version and subset? Nested subsets can make the full set the easiest one, so two scores that each disclose their subset can still be different tests.
2. Same precision and software stack? If not, the comparison repeats the mismatch of the 2023 dispute.
3. Is the leader present? When a benchmark’s public board leader is missing from a vendor’s table, the bold result means “best among the models we chose to show”.

When is a published result apples-to-apples with my workload?

A published result matches your workload only when the model, precision, batch and sequence shape, software stack and executor all line up with how you will run it. Sharing a workload category (“LLM inference”, “vision training”) is not enough. SPEC says this about its own disclosures: a flag description lets a technical reader form a preliminary judgment, but a complete judgment often needs testing on the user’s own application. Variability being application-specific points the same way: SGEMM and ResNet-50 showed different variability on the same cluster.

Published LynxBenchAI results show how each device runs the public PyTorch stack your own models run on, so they are broadly indicative for custom PyTorch models. They do not measure your model or application. If your hardware is not covered, LynxBenchAI benchmarks AI performance on your own hardware with stock PyTorch. When the decision depends on your own model’s behaviour, nothing replaces running that model.

Should we bring in an independent party rather than trust vendor-submitted numbers?

For a high-stakes procurement, yes. Run your actual model end to end on each candidate stack, let each vendor apply its own best tuning, and have the runs executed or audited by someone with no stake in the outcome. TechnoLynx can run that comparison for you, starting with framing your model’s inference requirements and a representative traffic profile before any hardware is touched.

LynxBenchAI gives that comparison a device-level reference by publishing release-named results for how each device runs the public PyTorch stack, and the first step is to look up each candidate device’s results under the current release.

Letting each vendor tune its own stack is fairer than forcing one common configuration. MLPerf Mobile supports vendor SDKs, noting that delegates do not showcase the hardware’s full capabilities and that delegate and vendor-SDK performance differ by up to 7x on some chipsets. Self-tuning still needs oversight, though. In SemiAnalysis’s study, the MI300X numbers came after months of AMD engineering support while the H100 and H200 ran out of the box, and the authors disclosed that asymmetry instead of hiding it. Fairness means each side gets its best stack and the report records how much effort each stack received.

Independence matters because self-reporting raises concerns about incentives for accuracy. Third-party auditing has promoted consistency and accountability in industries such as finance and air travel. Measure throughput, latency, memory, energy and cost per unit of work on each stack under representative load, not only peak throughput.

Three caveats limit how far the result travels. A scaled-down proxy of your model may miss bottlenecks that only appear at production scale. Time-boxing the vendors unevenly reintroduces the bias you were trying to remove. If the stakes are low, a smaller internal proxy run may be all the decision needs.

Who can run an independent benchmark of our own model on both candidate stacks?

TechnoLynx runs this comparison independently for teams that need an answer before committing. The result is a fair comparison of your own model on each candidate stack, each with its vendor’s best tuning, reporting throughput, latency, memory, energy and cost, with every configuration asymmetry flagged. For the device-level reference alongside that work, LynxBenchAI publishes release-named results for how each device runs the public PyTorch stack, and the first step is to look up each candidate device’s results under the current release.

This engagement is not worth it when the decision is reversible or the spend is small. It is also unnecessary when a vendor’s audited MLPerf submission already matches your model, precision and scenario closely, or when your own team has access to both systems and no incentive tied to either vendor.

How should I word and footnote a vendor figure I cannot verify?

Attribute the figure to the vendor, state plainly that it has not been independently verified, name the conditions that are missing or do not match your comparison, and give the source and retrieval date. LynxBenchAI applies a similar standard to its own results: each is tied to a named release and can be checked against other runs.

Benchmark bodies already supply the model. MLPerf requires any reference to a result to carry a footnote with the suite, version and division, the benchmark and scenario, the date and source of retrieval and the result ID, and an unverified result must state “Result not verified by MLCommons Association”. SPEC calls it a fair-use violation to imply a non-compliant number is compliant. It also requires “est.” next to each estimated number rather than a footnote buried at the bottom of the page. TPC prescribes a disclaimer for derived results stating that they are not comparable to published results. Researchers do this too: the MLPerf Power paper marks one of its figures as unverified and says the data only shows general trends.

Reporting an under-disclosed claim without endorsing it

Put the vendor in the sentence, and use reporting verbs (“NVIDIA reports”, “AMD states”) rather than verbs that adopt the claim (“achieves”, “delivers”, “is faster”). A footnote pattern we would use:

Vendor-reported, not independently verified. Source: [vendor], [document title], published [date], retrieved [date]. Disclosed: [e.g. GPU model, precision, framework and version]. Not disclosed: [e.g. batch size, driver version, power cap, harness]. Not matched to this comparison: [e.g. different precision or sequence length]. Reported for reference; not a measured result of this publication.

We name the missing fields so the caveat stays consistent from article to article and anyone can audit it. Avoid “verified”, “confirmed” or “official” unless a named body actually reviewed the result, because footnote wording that implies a check which never happened is worse than no footnote. MLPerf’s rules make the same point: you may not imply your use is verified or official.

Sign-off language for a normalised comparison

When figures from runs that were not identical are normalised onto one chart, say so in the main text, name what differed, and avoid “equivalent” or “matches”. MLPerf requires any comparison to identify differences in version, division, verified status, scenario or chip count in the main text, table or figure itself. Mixing verified and unverified results requires a footnote warning that unverified results may use inconsistent methods. SPEC accepts normalised historical comparisons only if their approximate nature is apparent from the context.

Normalisation can hide method differences that change the result. Phrases like “matches or exceeds” deserve particular suspicion. On a 500-task benchmark near 70%, each score carries roughly a 4-point interval, so a model 3 points behind can still be described as matching. If the critical fields are missing (precision, stack, batch policy), send the figure to editorial review before it is printed rather than shipping it with a softer footnote. That review step needs a written threshold for what counts as adequate, or it will be applied inconsistently and slow everything down.

When the claim is that a GPU engine matches the CPU engine it replaces

A claim that a GPU engine reproduces the CPU engine it replaces needs a pre-registered tolerance and a formal equivalence test, not a side-by-side glance. TechnoLynx’s AI Governance and Trust work covers this sign-off: framing the tolerance with risk and quant teams, designing the test and path plan, running the dual comparison and packaging a reproducible report for audit.

FAQ

What does a benchmark score mean?

SPEC describes a published result as a performance observation, a declaration of expected performance and a claim about the maturity of the methods used. A score only means something alongside the conditions it was observed under. Under the LynxBenchAI methodology, scores are higher-is-better and comparable only within the release that produced them.