You can buy GPU or ASIC hardware on a benchmark number when three things are true. Every number the harness accepts carries its versioned configuration: hardware revision, driver and firmware, framework and compiler versions, precision, batch size, power cap and harness settings. The workload comes from your own production traces, not from a synthetic test a vendor can tune for. And any gap between devices is put down to silicon or software only after profiling. In coding-agent evaluation, Digital Applied (2026) reports that identical model weights run through different scaffolding harnesses commonly score 10 to 20 percentage points apart. That evidence comes from SWE-Bench coding benchmarks rather than accelerator throughput tests. The harness variables it names include iteration budget, context-window management, tool-call format and error handling.
What every vendor submission must disclose before its number enters the comparison
Every submitted number should arrive with a fixed-field disclosure form, and a form with a required field missing should be rejected, not filled in by guesswork. TechnoLynx defines that disclosure schema around your workloads and builds the checklist and validation tooling that flags incomplete submissions. Where it helps, it aligns the fields with LynxBenchAI run metadata, so your internal runs and vendor-submitted numbers use one format. TechnoLynx’s first step is to frame the verification problem against your own benchmark goals.
LynxBenchAI gives the disclosure form a shared reference format, since its run metadata records how each device runs the public PyTorch stack; the first step is to map your checklist fields onto that run metadata.
For “Available” submissions, MLPerf Inference requires the software version of every component, plus the hardware configuration, software stack, Docker images and the settings of every component. MLPerf Training asks for a system description covering node count, processor and accelerator types, storage, interconnect, operating system and library versions. SPEC CPU 2017’s full disclosure for public results includes firmware/BIOS details and every configuration choice that differs from the default.
Disclosure checklist for a GPU/ASIC benchmark submission
| Field | What to require | Why it is required |
|---|---|---|
| Hardware | Accelerator model and revision, count, host CPU, memory, interconnect | MLPerf Training requires accelerator counts and types plus network interconnect in the system description |
| Software stack | Driver, firmware, CUDA/ROCm or vendor SDK, framework, compiler, container image, each with a version | Missing software versions hamper reproducibility (van der Kouwe et al.) |
| Precision and quantisation | Numeric format and the quantisation method, described well enough to repeat | MLPerf requires the quantisation method to be described publicly at a reproducible level |
| Batch size and scenario | Batch size, concurrency, and whether the run is offline, server or single-stream | MLPerf measures each scenario differently, for example Poisson-arrival queries in server mode against a single bulk query offline |
| Power | Power cap or mode, if not the default | SPEC requires non-default power choices to be documented |
| Harness configuration | Harness version, run command, flags, warm-up, iteration and timeout rules | An undisclosed harness makes a score non-reproducible (Digital Applied, 2026) |
| Randomness and data | Fixed random seeds, with the dataset unchanged at the start of each run | MLPerf requires fixed random seeds and an unchanged dataset |
| Logs | Raw result logs for every performance and accuracy run | MLPerf submissions include load-generator logs for peer review |
| Provenance | Who ran it, and the date the result was produced or retrieved | SPEC requires sources, an identifiable licensee and a retrieval date for quoted results |
In a study of up to 1,045 foundation-model leaderboards, Zhao et al. (2024) found that only 35.12% linked to a specific GitHub repository, though 76.65% had explicit submission channels. Some leaderboards, MathVista among them, require contributors to submit output files along with scores so operators can verify them independently. The authors propose a machine-readable “Leaderboard Bill of Materials” that records model and benchmark versions for every entry. A procurement harness needs that same inventory.
A submission that fails the form can still be shown, but never as an equal. SPEC’s fair-use rules treat implying that a non-compliant number is compliant as a violation, and recommend tagging such numbers “(nc)”. Know the limits of the form, too. A rigid template will not anticipate a new backend or precision format. Disclosure also proves only that the stated conditions are comparable, not that the run was clean.
Publish the dispute process before the first ranking, not after the first complaint. Name a review body with written authority to re-check run metadata, re-bucket a submission by configuration (precision, cluster size) or disqualify it, and use those published criteria. MLCommons publishes such a process. Objections are filed as GitHub issues that cite the offending lines and the rule broken. A submission judged deceptive or not of interest can be excluded by a two-thirds vote of the submitters, and the stated aim of review is fairness, not litigating competitors out. IOSCO’s principles call for published procedures for error reports, and respondents to its consultation agreed that the oversight committee should not be dominated by administrators or submitters.
Should each vendor be pinned to a common stack, or allowed its best-tuned software?
Report both, labelled. Run one baseline on a common, disclosed stack, and a separate vendor-tuned run whose deviations are listed. TechnoLynx frames that comparison with you: the metric, the target workload, and which layers stay fixed versus vendor-best. It then runs every candidate under the agreed baseline, checks the results against MLPerf or vendor-published figures where relevant, and delivers a report with a full reproducibility manifest. LynxBenchAI’s published results serve as an outside reference for how each device runs the public PyTorch stack.
LynxBenchAI anchors the common-stack baseline, because its published results show how each device runs the public PyTorch stack; the first step is to compare your baseline runs with those results before any vendor-tuned run.
SPEC CPU 2017 already formalises the split. “Base” metrics must build every benchmark with a common set of optimisations and the same flags within a suite. “Peak” lets the tester choose submit commands, copy counts or thread counts per benchmark. MLPerf’s closed division allows any framework, including PyTorch or ONNX, provided the latency and accuracy bounds are met and the reference weights are used. It permits calibration for quantisation but no retraining. Milabench takes the opposite approach to MLPerf’s flexibility. It prefers standardised implementations over vendor-contributed pipelines and argues that MLPerf’s flexibility can trap optimisations inside vendor code and hurt reproducibility.
Our own position bears on the choice. Vendor-specific APIs are not magically faster than standard ones, and the same code performs essentially the same. What a vendor API offers is access to lower-level features, and using that access takes real engineering commitment that is not always worth paying for. So a best-tuned number tells you what the device can reach if your team makes the same investment. The common-stack number shows the device on standard code, without that extra investment.
Which layers to pin, by comparison type
| Comparison | Hold fixed | May vary, if disclosed |
|---|---|---|
| Two vendors’ current accelerators | Workload, data, precision target, metric, harness, accuracy bound | Driver, compiler, kernel libraries and collective library (NCCL vs RCCL), each a generally available release |
| Old vs new generation, same vendor | Everything you can, including CUDA/cuDNN and framework versions, to isolate the silicon | A newer stack only if the old chip cannot run it, with the gap disclosed |
| GPU vs ASIC | Task, data, quality target, reporting rules | The whole software path |
| Tender justification from published results | The workload match itself | Nothing: if the match is weak, reproduce the run |
Milabench’s authors could not run every vendor on the same PyTorch version, and they note that the PyTorch version can have a major impact on performance. When that happens, say so in the report rather than hiding it. For the vendor-tuned run, require what MLCommons calls an Available software stack: components that are well supported for general use and available at submission time, not a private branch.
Repeat runs are part of the protocol, not an optional extra. For its time-to-train results, MLPerf Training requires five runs for vision benchmarks and ten for the rest, drops the fastest and slowest, and reports the mean of the remainder. Published MLPerf results can support a procurement case, since standardised benchmarks exist precisely because peak specifications do not predict delivered performance. But their evidential value depends on how closely the published workload matches yours. Where the match is weak, an independent reproduction makes the justification much stronger.
Tokens per second, per dollar or per watt?
Choose the unit by what your deployment rewards, and publish it as the stated basis of comparison. SPEC allows a basis of comparison built from cost or other non-benchmark data, provided that basis is clearly disclosed. A 2026 survey of LLM accelerators puts it this way: latency-critical serving rewards low per-token latency, while throughput-critical serving rewards aggregate tokens per second per watt. The same survey says efficiency must be reported in delivered rather than nameplate terms, because peak TOPS per watt does not predict the energy of a real serving workload.
The summary statistic is another place comparisons go wrong. Fleming and Wallace (1986) showed that the arithmetic mean of normalised numbers is meaningless and that the geometric mean is the correct average. They also recommended reporting the minimum and maximum alongside the mean, since any mean misleads under large variance. The mistake still happens: van der Kouwe et al. found 41% of the applicable systems-security papers they surveyed averaging overheads with the arithmetic mean. Putting mixed-precision and cross-vendor results on one scale is covered in making GPU benchmarks comparable with a normalisation framework.
Checking a vendor’s claimed figures before the committee sees them
Treat a vendor’s datasheet figure as a claim until a party without a stake reruns it under your traffic pattern and reports worst-case tail latency, not just the average. MLPerf’s audit rules for its own submissions show what scrutiny looks like. A submission is more likely to be audited when its performance does not match the hardware’s known characteristics or when reviewers cannot see how it was achieved. The submitter must then give an auditor two days of hardware access to replicate runs. Digital Applied (2026) poses three questions to ask before trusting a vendor score: who built the evaluation, who scored the run, and whether an independent team has reproduced it. Even an independent lab’s result is only as trustworthy as the lab’s published method.
LynxBenchAI gives the committee an outside reference point for a vendor’s claimed figures, since its published results show how each device runs the public PyTorch stack; the first step is to compare the datasheet figure with those results before commissioning a rerun.
Is the performance gap the silicon or the software stack?
Treat a throughput gap as stack-dependent until it survives a profiled, controlled re-run across several driver and compiler versions. Only then call it a silicon limit. TechnoLynx designs that harness on your workloads, holding the compiler fixed while the workload varies and then the reverse. It runs Nsight Systems profiling on each configuration, checks that the attribution reproduces across driver revisions, and packages the raw logs and configs into a disclosure pack. That work sits within TechnoLynx’s GPU Performance Engineering service.
In the independent benchmarking cited by a 2026 accelerator survey, the AMD MI300X reached about 620 BF16 TFLOPS against a marketed 1,307, slower in practice than the H100. The survey attributes this to the relative immaturity of ROCm. Milabench’s authors likewise trace the difference between FLOP counts and real-world results across their tested accelerators to software stack maturity, noting that CUDA’s kernels have been tuned alongside AI workloads for over a decade.
Profiling is how you tell the two apart. This is the practical side of our view that context comes from profiling, not from reading blog posts. Nsight Systems GPU Metrics samples SM utilisation, tensor core activity, warp occupancy and PCIe, NVLink and memory bandwidth, which answers whether the GPU is idle, full or blocked on IO. NVTX ranges project CPU-side regions onto the GPU timeline. PyTorch functions and tensor shapes can be traced without changing source code. Each report records the OS and driver versions it was captured on, which helps the disclosure pack. Profiling has its own costs. Nsight documents significant runtime overhead for features such as GPU memory tracking and tracing all CUDA APIs. Our page on finding and fixing GPU bottlenecks through profiling covers the profiling workflow in more depth.
Word the result honestly. “As currently supported on ROCm 6.1” is a defensible claim, while “intrinsic hardware capability” usually is not. A software-bound conclusion also goes stale as stacks improve, so the method should be rerun at each chip or driver revision. For public or tender-facing reports, put raw logs and configs in an appendix next to the summary tables, and agree a redaction policy for vendor-sensitive tuning details.
Should we build the benchmark harness ourselves or have an independent lab run it?
A self-built harness is enough for an internal sanity check, as long as its rules and configurations are written down. For a procurement-grade or public comparison, have an independent party run it or audit it, because being seen to be free of vendor favouritism is part of what you are buying. Fairness does not mean forcing identical code onto dissimilar hardware. It means pre-registering one fixed set of models, datasets, precision targets and metrics, applying it identically to every entry, and publishing every tuning choice.
TPC, IOSCO and MLPerf each set out who checks results or methods. TPC’s policies require each benchmark subcommittee to maintain an auditor exam of at least 50 questions for every benchmark standard. IOSCO’s principles call for an independent internal or external auditor to review whether the administrator keeps to its stated methodology. MLPerf’s training paper describes peer review for rule compliance before any result is published. Even a well-run neutral harness generalises only as far as its workload resembles yours. MLPerf’s own rules allow approximations and quantisation, so such differences have to be disclosed and controlled.
Who can build and run a disclosed benchmark harness for our GPU and ASIC shortlist?
TechnoLynx builds and runs disclosed, versioned benchmark harnesses for GPU and ASIC shortlists. That covers the disclosure schema, a workload spec drawn from your telemetry, common-stack and vendor-tuned runs, Nsight-based attribution, and a report with a reproducibility manifest. On build versus buy: an in-house team with the time and the skills can run MLPerf-style rules itself. Bringing in a specialist pays off when the comparison has to survive a losing bidder or an audit committee, or when nobody internally can profile ROCm and CUDA paths to the same depth. When you choose a partner, check that it has no stake in the result, publishes its method, and hands over raw logs. LynxBenchAI is the measuring tool for the hardware itself: its published results show how each device runs the public PyTorch stack, but it does not run your workload.
This engagement is not a fit if you need a result certified under MLPerf or TPC rules. That goes through MLPerf’s peer review or TPC’s audited process. It is also overkill for a one-off spec-sheet comparison. For that, the published results and a careful read of the vendors’ disclosures may be enough. The MLPerf reference implementations are open source and licensed for free benchmarking use.
How do we define a representative benchmark workload from our own production logs?
Write the benchmark workload down from your own telemetry before any vendor runs anything. That means the request mix, request and response size distributions, arrival times, the peak-to-average ratio, and the specific kernels or behaviours that dominate your cost. LynxBenchAI’s published results then narrow down which devices deserve the full run.
LynxBenchAI narrows the shortlist before the full run, because its published results show how each device runs the public PyTorch stack; the first step is to check your candidate devices against those results.
Systems evaluated under unrealistic synthetic workloads degrade in real deployment. The BurstGPT authors, working from 121 days of Azure OpenAI traces, found that unrealistic synthetic workloads do not reflect how auto-regressive LLM serving behaves. They also found that concurrency patterns differ by service and model type. At the same requests-per-second rate, their bursty real trace put more pressure on KV cache management than a more uniform workload did. Database research shows the same thing. Redbench reports that standard TPC-H and TPC-DS benchmarks miss traits of real cloud workloads, such as query repetition. In the Redset trace, fewer than 0.1% of queries account for roughly 25% of CPU time, and caching speedups of 1.1 to 2.98 times appeared only under trace-shaped workloads. Those are warehouse results, not GPU ones, but the lesson transfers: an averaged synthetic load hides exactly the skew that decides sizing.
A few rules keep the spec honest. Model both average sustained load and the worst-case spike, because a single number understates risk wherever missing an SLA is costly. For LLM decode, specify a bandwidth-sensitive metric rather than peak FLOPS, since attention during decode stays memory-bandwidth-bound regardless of batch size. Say how trace repetition is treated. MLPerf prohibits caching across queries, and its load generator detects caching by comparing runs with unique and duplicate sample indices. To stop vendors tuning to the one published trace, keep hidden or rotated variants. MLPerf does the equivalent with an alternate-random-seed test that defeats seed-specific optimisation. If your logs do not yet capture request sizes and arrival times, add that instrumentation first. A workload spec guessed in a meeting is just another synthetic benchmark.
FAQ
What is MLPerf Inference?
MLPerf Inference is an industry and academic benchmark suite for measuring the inference performance of ML hardware, software and services under standard metrics. It defines four scenarios (single-stream, multistream, server and offline) and shows that performance can vary drastically between them.
What are tokens per watt?
Tokens per watt is throughput divided by power draw, a measure of energy efficiency for LLM serving. It matters most for throughput-critical serving, and it should come from measured rather than nameplate power.
