A GPU benchmark for procurement is defensible only when it is treated as one specific comparison, not a headline score. That means the methodology is disclosed, the reproduction tolerance was agreed before anyone ran the test, the raw results sit in an evidence package that traces back to your requirements, and the cost and headroom assumptions behind the figure are written down. Take any one of those away and you are left with a vendor claim, however carefully it was measured. The rest of this article covers what each of those pieces looks like when a contract, a review board or a losing bidder is going to test it.
Why one GPU benchmark number misleads a procurement decision
Software moves GPU numbers, not only silicon. The Milabench team at Mila (2024) reports that new PyTorch versions often bring gains ranging from 5% to more than 100% on specific benchmarks, and they attribute the gap between their synthetic and real-world results more to software stack support than to hardware limits. The same team could not run every vendor on the same PyTorch version because of timeline constraints, and they warn that the version can have a major impact on performance. So a figure that arrives without its software versions is incomplete, and two such figures cannot be compared.
Run-to-run noise is the second problem. In Bouthillier et al. (MLSys 2021), the variance from arbitrary choices and sampling noise was of the same order of magnitude as the published improvements between pipelines. Hoefler and Belli (SC15) note that performance variation on supercomputers is severe enough that several large procurements wrote upper bounds on variation into the vendor’s deliverables.
The third problem is the operating point. MLPerf Inference found that adding a latency constraint (the server scenario, compared with the offline one) cost some systems as little as 3% of throughput and others as much as 50%. SemiAnalysis makes the same point about LLM serving: if a GPU wins 4x at an interactivity level too slow for any real chatbot, the win means very little in practice. A single number therefore answers one question at one batch size, one precision, one latency bound and one software version, and a procurement decision has to say which of those questions it is asking.
What has to sit behind a GPU cost figure before operators can be compared
An operator’s per-dollar or per-GPU-hour figure can only be compared with another operator’s when it itemises power, amortised hardware capex, networking and overhead, says whether the GPU utilisation behind it was assumed or measured, and is broken down by workload type, model size and batch size instead of being blended into one number. TechnoLynx runs that comparison for operator due diligence, setting the comparability requirements, normalising batch size, precision and workload mix, running representative workloads on each operator’s stack, and reporting where measured cost per unit of work differs from what the operator disclosed. If an operator will not provide the breakdown, treat its headline figure as non-comparable.
Utilisation is one assumption that moves these figures directly. The cost framework in Beyond Benchmarks: The Economics of AI Inference (2025) builds hourly GPU cost from depreciation, power and maintenance, sets utilisation to 100% as a standardised baseline, and notes that effective hourly cost is the base cost divided by utilisation. On that definition, a figure quoted at full utilisation understates the cost per useful hour by a factor of 1/0.7, or about 1.43x, for a fleet that actually runs at 70%. The same paper’s A800 estimate leaves out the host server, rack, networking and labour. Another operator’s figure might include all of those, which is exactly why the components have to be listed.
The Safeguard TCO analysis (2026) makes the matching point from the buyer’s side: the fair owned-cost number is fixed cost divided by utilisation measured over the whole life of the hardware, including ramp-up and idle weeks. It also warns that normalising rental prices by vendor peak throughput gives you a peak figure, not measured performance. SemiAnalysis estimates that colocation rent and electricity usually make up less than 20% of TCO. So a 20% difference in tokens per megawatt moves total cost by less than 4%, while hardware margin dominates. We go further into turning these inputs into one decision metric in making TCO decisions comparable with blended cost metrics. The metric to aim for is cost per successful request or useful output, not cost per GPU-hour.
Scoring a vendor whose model you cannot see inside
When a vendor’s internals are hidden, the only fair test is to score its outputs against an independent reference that neither side controls, using one verification framework applied identically to every vendor. For day-ahead wind-power forecasts we would take ECMWF’s ERA5 reanalysis as the reference for retrospective scoring, or an operational analysis when the comparison needs to sit closer to real time. ERA5 is itself a model-based reanalysis with its own resolution and latency limits, so it is a fair yardstick rather than ground truth.
For GPU hardware, we would take a run on a public software stack as the equivalent independent reference. LynxBenchAI publishes how each device runs the public PyTorch stack that teams’ own models run on, so its results are broadly indicative for custom PyTorch models too, although they do not measure your own workload or application.
What evidence package will a technical review board accept?
A review board needs a formal package with three parts: the benchmark methodology, the raw results, and traceability from each result to the requirement it supports, so the SKU choice can be followed back to what the programme actually asked for. TechnoLynx builds that package by running the shortlisted SKU configurations under representative workloads, profiling each one end to end with a latency breakdown, and assembling the methodology, raw results and requirement traceability into the artefact the board reviews. Published LynxBenchAI results can narrow the shortlist before that run.
LynxBenchAI publishes how each device runs the public PyTorch stack, which narrows the SKU shortlist before the evidence package is built, and the first step is to check the shortlisted devices against its published results.
There are worked precedents for the format. An IETF draft specification defines an evidence package as a ZIP archive that must contain a manifest.json plus media and testcases directories. In that format, media files are named by their SHA256 checksum, with the checksum and media type recorded in the manifest. For the JUPITER exascale procurement, each benchmark lived in a single Git repository as its single source of truth, holding the description, JUBE scripts, results and source code. Those JUBE scripts formed part of the procurement documentation and defined execution parameters exactly. Each repository was archived as a tar file for the specification package, and large input data came as a separate download with a verifying hash.
A raw number tells a reviewer how fast a system is, while traceability tells them why that speed matters for the programme.
Is an in-house reproducible benchmark enough for the contract record?
We would treat it as enough, provided it is reproducible and fully documented and the programme’s contract or regulatory language does not require independent verification. The JUPITER team set reproducibility as a hard procurement requirement because both the site and the system provider had to be able to run the suite and get the same results. Bidders used that suite to prepare and commit their performance numbers, and the procuring entity chose the vendor on those values plus other criteria. Bring in a third party running the same methodology when the contract requires it, or when vendor neutrality is likely to be contested. Reproducibility also has a shelf life. Hoefler and Belli point out that naming a well-known system is not enough, because regular software upgrades on that system are likely to change its performance.
What should each bidder submit so GPU scores hold up under audit or a bid challenge?
Every row below has a public precedent. Together they make a submission checklist:
| Submit | Why it is needed |
|---|---|
| Software versions of every component, hardware configuration, containers and settings | MLPerf requires this for “Available” submissions so results can be reproduced |
| Precision and a reproducible description of the quantisation method | MLPerf requires quantisation methods to be described publicly at a level that allows reproduction |
| Run logs, user configuration and accuracy logs | MLCommons lists these as mandatory files for every submission |
| Accuracy against the full-precision reference | MLPerf requires almost all implementations to reach within 1% of the FP32 reference accuracy |
| Spread across runs, with confidence intervals | Hoefler and Belli’s rule is to report confidence intervals for nondeterministic measurements |
| Source code, compile scripts, output and verification files | The NERSC-10 RFP requires the subcontractor to supply all of these |
| Whether each result is baseline, ported or optimised | NERSC-10 accepts results in those three categories, defined in its run rules |
| How any estimate (as opposed to a measurement) was produced | NERSC-10 asks offerors to describe how they determined their estimates |
| Clear labelling of any result that did not follow the run rules | SPEC treats implying a non-compliant number is compliant as a fair-use violation |
What a submission shows also depends on the harness every bidder runs. We cover that side in building a trustworthy, disclosed benchmark harness for GPU/ASIC procurement.
How much capacity headroom above peak load should a GPU benchmark show before go-live?
The go/no-go figure for the board is a measured margin: (provisioned capacity minus worst historical peak) divided by worst historical peak. That margin is measured against the worst peak, not the average, and it only counts if failover has been tested. A cluster sized exactly to the past peak has a margin of zero, which leaves nothing for a worse-than-historical event. We would present that as a no-go or a conditional go, never as a pass.
LynxBenchAI publishes device results on the public PyTorch stack, which helps narrow the hardware before headroom testing; it measures the hardware, not your workload, so the first step is to check candidate devices there.
The reason is mechanical. EdgeNext’s capacity-planning guidance notes that if every GPU is needed to meet normal peak demand, one failure immediately pushes the remaining fleet past its latency target. That guidance also recommends testing steady-state and burst load separately and running reduced-capacity tests to confirm N-1 behaviour. Google’s SRE book goes further and provisions for N+2, so peak traffic can be served (possibly degraded) with the two largest instances unavailable. It also says to establish the resource-to-capacity ratio by load testing, not by tradition. A margin with no tested failover or degradation plan behind it is not a complete verdict.
One capacity study found that scaling average traffic to estimate peak is only rough. In a SPEC ICPE 2015 database-replication capacity study, scaling average traffic by 1.5x to approximate the peak gave a maximum replication-lag estimate about 1.7x that of the full numerical model. The authors also argue that forecast errors which underestimate traffic can break the SLA, which makes headroom necessary rather than optional.
On sizing once for three to five years without revalidation: in the same study, a headroom factor of 1.44 under an assumed 30% annual growth ran out in about 13 months. Demand patterns also shift, so the historical peak may not bound the future worst case. Revalidation is triggered by change, not only by growth. The EdgeNext guidance says to repeat benchmarks after model, quantisation, runtime or driver changes, and NERSC-10 requires benchmarks to be re-run after significant software upgrades or hardware repairs. None of our sources gives an industry-accepted margin for clinical inference servers, so the number has to come from the risk appetite the board signs off, stated explicitly.
Agreeing the INT8 versus FP32 parity tolerance with the OEM first
We would make parity sign-off rest on a written tolerance spec the OEM owns, covering IoU, box-coordinate and confidence-score deltas, signed before the INT8-versus-FP32 comparison runs. We would treat the thresholds and their approver at the OEM as facts about the OEM’s programme, and get them in writing by sending a draft through that approver’s chain. Where the INT8 model exists to cut serving cost, that optimisation work falls within TechnoLynx’s Inference Cost-Cut Pack, but the tolerance it is judged against stays the OEM’s decision.
We would use a precedent spec from another supplier or programme as a first draft, not a binding answer. It may reflect a different risk appetite, sensor stack or regulatory context, so it needs this OEM’s explicit re-approval. The closest public precedent is aggregate: MLPerf’s requirement that implementations reach within 1% of FP32 reference accuracy, which its authors found easy to meet for 8-bit integer arithmetic without retraining. MLPerf Mobile sets all its targets above 93% of FP32 accuracy, including for INT8 models.
A study of tensor-kernel tests found median tightening factors of 100x to 200x when hand-picked tolerances were recalibrated from the 95th percentile of observed error times a 1.5 safety factor. The parallel we would draw is to base the draft parity bounds on the measured run-to-run spread of the FP32 model itself, so the bounds reflect real variation instead of a guess. Without a pre-agreed written tolerance, even a technically excellent comparison can be rejected at sign-off. Checking that a converted model still behaves like its original is a question of production AI reliability as much as of speed.
How do you set an acceptance tolerance between a vendor’s claimed GPU throughput and your own reproduction?
Set it as an asymmetric pass/fail rule, not a single percentage. Reproduce the vendor’s submission on a disclosed reference stack with the claimed batch size, precision and workload. Accept the score only if the reproduction falls inside a tolerance band registered before the run, and fail it outright if the reproduction uncovers any undisclosed change in configuration. TechnoLynx sets this up as a reproduction harness, building the reference stack to match the vendor’s disclosed hardware, precision, batch size and workload, capturing telemetry of what actually executed, computing the delta and documenting it in a report a third party can audit.
| Reproduction outcome | Verdict |
|---|---|
| Configuration matches the disclosure, result within the band | Reproducible: accept the claim |
| Configuration matches the disclosure, result outside the band | Non-reproducible: flag it, do not average it away |
| Undisclosed lower precision, smaller batch or different workload found | Fail, whatever the score delta, because the comparison is no longer like-for-like |
There is a concrete model for the band. MLPerf Mobile auditors build the vendor-specific app, install it on a factory-reset device and reproduce the latency, throughput and accuracy, and results count as valid if the auditors’ numbers fall within 5% of the submitted scores. The wider MLPerf rules say plainly that results which cannot be replicated are not valid results. Review is not a formality either: the MLPerf Inference authors found about 40 issues in roughly 180 closed-division results. MLCommons also lets the review committee waive fixes that change cumulative performance by less than 2%, and opens a post-publication investigation when a fix to a non-compliant submission cuts performance by more than 5%.
Do not copy that 5% into every contract. Run-to-run variance has been found on the order of the differences being compared, so the band has to come from measured variance on that configuration. Hoefler and Belli’s reporting rule gives a usable template: collect measurements until the 99% confidence interval is within 5% of the reported mean, and say so. A band set too wide accepts inflated claims, and a band set too narrow flags legitimate stack variance as failure.
The reference-stack run is the deciding run, and the vendor’s own figure is the claim it checks, not equal evidence alongside it. Requiring a pass under both stacks sounds stricter, but we would not use it as the test. The FlexBench authors (2025) report from hands-on MLPerf work that heavily over-optimised results from a few chip makers are rarely reproducible out of the box on other models, software versions or hardware configurations. Milabench frames the underlying trade-off: MLPerf lets vendors reimplement pipelines, which encourages optimisation but costs reproducibility, while Milabench uses fixed, standardised implementations so its results reproduce and generalise. We would also profile the reproduction run rather than trust a final throughput number alone, because the profile shows what actually executed. For running this across several vendors, see running an independent cross-vendor GPU/ASIC inference throughput benchmark.
How can you tell whether a vendor’s throughput figure relied on lower precision or batch size than it disclosed?
Instrument the reproduction to log the precision and batch size that actually ran, and compare that log with the disclosure. Comparing final throughput numbers alone is not enough. The gain from substituting precision is large. In the DEEP-GAP study (2026), INT8 reached up to 14x on a T4 and up to 18x on an L4, while FP16 gave around 1.8x to 2.2x on the T4. Batch size matters too: the T4 needed batches of 256 or more to reach peak throughput, while the L4 peaked at 16 to 32. INT8 also cut memory footprint by about 2x to 3x, which allows larger batches in the same memory. A silent change of either kind can therefore move a score far beyond any reasonable band.
MLPerf’s validation suite shows the other checks worth borrowing. Responses are logged at random in performance mode and compared against the accuracy-mode log. A duplicate-sample test detects caching by checking whether repeated queries run significantly faster than unique ones. An alternate-random-seed test catches optimisations tuned to the official seed. The limit is access. Without the vendor’s configuration metadata or runtime telemetry, only the score delta is visible, and a partial substitution can look like normal variance. Fixing batch size, precision and workload before comparing is covered in a normalisation framework for fair GPU benchmark claims.
A vendor publishing sustained figures against rivals’ submitted numbers has MLPerf conventions to reuse: a minimum run of 60 seconds so power-management and DVFS equilibrium are captured, and power averaged over a run of at least 60 seconds, as MLPerf Power does. Presented that way, the sustained-state figure is routine disclosure under a known protocol.
Re-verifying submitted numbers after the contract is signed
A submitted number should stay auditable after award, which means the right to re-run the tests has to be in the contract, not assumed. TechnoLynx can act as the independent party that re-runs a vendor’s submitted benchmark on your hardware pool against the agreed methodology, as a third-party GPU cluster benchmarking engagement.
LynxBenchAI publishes device results on the public PyTorch stack, giving a public reference point alongside a vendor’s submitted numbers; the first step is to compare those submitted figures with its published results.
Building re-verification into the contract
NERSC-10’s RFP is one public example. Its performance targets are negotiated once the final configuration is defined, converted into requirements at a technical decision point, and must keep meeting acceptance criteria for the life of the system. The university may run any of the tests at any time to check compliance. Each benchmark must be run at least five times, give correct answers and meet or exceed the commitment. For re-verification, MLPerf’s audit gives one pattern: two days of hardware access, with a pre-agreed test list on day one and follow-up tests on day two. For very long-running workloads, partial runs are accepted.
Recourse when re-verified numbers fall short is a contract question for your counsel. The engineering contribution is to make the shortfall unambiguous, by writing the commitments as acceptance criteria with the tolerance and run count stated in advance. MLCommons lists the outcomes of an objection: it is found not valid, the result is moved to the open division for non-compliance, or it is removed for intentional cheating.
Before a customer’s benchmark result appears in public material
Before a customer’s result is quoted publicly, follow the discipline SPEC applies to public use of its results. Name the source, the responsible party and the retrieval date. Clearly disclose the basis of any comparison. Mark anything that did not follow the run rules as non-compliant. That includes publishing the exact hardware and software stack alongside the number, as you would for any public benchmark claim. If the customer’s numbers come in below the published figures, we would re-test on the documented stack first and then check framework versions, since the PyTorch version can have a major impact on performance. Environment-specific failures do happen. SemiAnalysis traced a 30-minute Blackwell stall to NCCL in a vLLM image lacking prebuilt Blackwell code, which others missed because their JIT cache was populated. If the gap still cannot be explained, withdraw the claim.
Who can independently re-run a vendor’s GPU benchmark before we sign?
TechnoLynx re-runs vendor-submitted GPU benchmarks on your hardware against the methodology and tolerance you agreed with the bidders, and reports whether each claim reproduces. On build versus buy: an in-house team can do this if it can pin the environment, instrument the runs and defend the method to a review board, and a specialist makes sense when neutrality is likely to be challenged or that capacity does not exist. When choosing a partner, look for one that publishes its method, works on your hardware rather than its own lab, and has no stake in which vendor wins. For the hardware shortlist before any run, LynxBenchAI publishes device results on the public PyTorch stack; it measures the hardware, not your workload. For deeper coverage of claim auditing, see building reproducible, audit-defensible performance claims.
Other routes exist. MLPerf results carry independent audits. Some vendors publish reproduction steps: AMD documents a four-step reproduction of its MLPerf Inference v5.1 submission. Open suites such as JUPITER’s are released as open source, and Milabench ships Docker images for reproducible runs.
FAQ
Is 90% GPU usage too high?
For a serving fleet, it can be. The Safeguard TCO analysis notes that running at 95% busy usually means queueing delay, so most teams keep headroom, which lowers achievable utilisation.
What are the criteria used for evaluating bids?
That depends on the procurement. JUPITER used a TCO-based value-for-money approach in which value came from the number of reference workloads run over the system’s life, and it selected the vendor on committed benchmark values together with other evaluation criteria.
