To turn benchmark data into a procurement score or a merge decision, you need a method that is fixed and published before anyone sees a result. For MLPerf results, that means scoring each benchmark on its own primary metric, scaled against a reference system or the best offer, after checking that results share benchmark and scenario and recording any difference in version, division or verified status. Criteria that no benchmark measures, such as vendor roadmap stability, get scored from documented public evidence rather than vendor assurances. On the engineering side, the same discipline becomes a benchmark harness run as a required CI check within a defended duration budget.
These three jobs (procurement scoring, scenario libraries and CI gating) look unrelated, but they share one failure mode. A score depends on the pipeline that produced it, and procurement guidance warns against altering scoring criteria once evaluation has begun.
Turning MLPerf results into rubric points without looking arbitrary
To convert MLPerf results into technical-merit points, write down the conversion before scoring starts. Map each MLPerf category to one criterion, normalise the submissions so they are comparable, scale each result on its primary metric, and apply the same formula to every vendor.
LynxBenchAI publishes training, inference and compute scores for each covered device under one published method, giving a panel citable evidence for its technical-merit criteria; the first step is to check whether the devices in your tender appear in the current release.
What MLPerf measures, category by category
MLPerf is a set of separate benchmarks, each with its own metric, and it deliberately has no single summary score. The MLPerf Training paper gives two reasons. Any summary score implies a weighting, and no weighting fits every user. A summary score also becomes less meaningful when a submitter skips some benchmarks. This is the key fact for a procurement panel: the weighting is your decision, and MLCommons will not make it for you.
MLPerf Inference defines four scenarios (single-stream, multistream, server and offline), and performance can vary sharply between them. Training uses time to train to a defined quality target, which captures both system speed and accuracy. The criterion column below is our suggested mapping, not one MLCommons publishes.
| MLPerf result | Primary metric | Technical-merit criterion it can evidence | Control before scoring |
|---|---|---|---|
| Inference, offline | Throughput in samples per second | Batch processing capacity | Version, division, chip count, verified status |
| Inference, server | Queries per second achievable while meeting the QoS bound | Online serving capacity under a latency bound | QoS latency bound met, version, division, verified status |
| Inference, single-stream | 90th-percentile latency in the MLPerf Inference paper; current edge rules report the 99th | Per-request latency (edge, robotics) | Required scenario in the edge suite |
| Inference, multistream | Number of streams supported within QoS | Multi-camera or multi-sensor concurrency | Edge submitters may infer multistream from single-stream |
| Training | Minutes to reach the quality threshold | Training throughput at fixed model quality | Fastest and slowest runs dropped, mean of the rest reported |
If the criterion is the capacity of a hosted service, MLPerf Endpoints measures the system you actually buy. It tests the model, hardware, serving software and networking together under production-like load, and reports a Pareto frontier across concurrency levels.
Normalising submissions before they become points
MLPerf submissions each describe their own system under test, so a scorecard has to check that two numbers are actually comparable before ranking them. MLCommons treats results as compatible when they come from the same benchmark and scenario in compatible versions. Any comparison must state differences in version, division, category, verified or unverified status, scenario or chip count. A result that never went through MLCommons review is unverified and has to be disclosed as such.
Precision deserves its own line in the method. Closed-division inference submissions may use calibrated quantisation but no retraining. Open-division results may change the model altogether, so you cannot score them against closed results as if they were the same thing. Our approach would be to accept only closed, verified results for a weighted criterion, and to record any open-division evidence as unscored context. The question of which baselines to disclose across vendors is covered in more depth in disclosed baselines for fair cross-vendor comparison. Reading NVIDIA MLPerf submissions as procurement evidence works through one vendor’s results in detail.
Has any MLPerf scoring formula survived a public-tender audit?
Expect to write your own MLPerf-to-score formula and defend it on how it was built. The precedents that do exist are about procedure. German NHR computing centres score each benchmark on runtime or throughput, scaled against a known reference system or the maximum across offers. They warn against giving the worst offer zero points, because one bad bid would then shift the scores of every other bid. These centres combine benchmarks as a weighted sum, and they report that no algorithm derives the weights.
Directive 2014/24/EU requires the relative weighting of award criteria to be stated in the procurement documents. UK government bid evaluation guidance requires the method to be disclosed and followed. It also says the formula should be scenario-tested with dummy bids before it is finalised. Construction tendering guidance uses the same 0-to-10 weighted pattern with a written audit trail.
A 2026 audit of LLM benchmarks found that a single pipeline choice could move a model’s score by more than 80 percentage points and reorder the rankings. That audit covers model evaluation, not hardware, but the lesson transfers: a benchmark score depends on the pipeline that produced it.
Published LynxBenchAI results show why the weighting decision matters. In the 26Q3 results, the AMD Instinct MI300X scores 1023.3 in compute against 487.1 for the NVIDIA H100 80GB HBM3. The H100 scores higher in inference, 1678.1 against 1304.0. The NVIDIA H200 scores 812.0 in training against 727.2 for the MI300X. So the order depends on which category carries the most weight, which is why the weights have to be published first. Per the LynxBenchAI methodology, each 26Q3 test is a single timed window after a discarded warm-up, and its scores can be compared only within 26Q3.
Intel Gaudi 3 is not in the 26Q3 release. For hardware a release does not cover, LynxBenchAI benchmarks AI performance on your own hardware with stock PyTorch.
Scoring vendor roadmap stability when no benchmark measures it
No benchmark measures roadmap and software-ecosystem stability, so score it from public, verifiable indicators. Group them into evidence categories (published roadmap statements, cancelled or delayed products, software release cadence, framework integration) and define what earns each score band before any bid is read. Vendor material is admissible only as a labelled vendor claim.
Intel’s Gaudi line shows how the categories work in practice:
- Cancelled products: in January 2025 Intel said Falcon Shores, planned as the successor to Gaudi 3, would become an internal test chip. Intel’s interim co-CEO also conceded that “it’s not enough to just deliver the silicon”.
- Successor gap: one third-party tracker records that, as of its last check, no Gaudi 3 successor had reached general availability.
- Published roadmap statements: in October 2025 Intel announced a data centre GPU code-named Crescent Island, with customer sampling expected in the second half of 2026. That date is a vendor claim.
- Software release cadence and integration: Intel’s own deck lists Gaudi software releases 1.11 to 1.20 between July 2023 and February 2025. Intel’s Gaudi 3 documentation describes native PyTorch integration plus DeepSpeed, Hugging Face and vLLM support. Both are vendor sources.
On these sources, the record shows one cancelled successor and one announced product with a sampling date, which we would score as documented but unproven, a judgement the panel should record with its reasons. None of our sources covers NVIDIA’s roadmap, so score NVIDIA from the same categories using equivalent public records rather than from reputation.
Public buyers already have patterns for this. World Bank guidance allows rated criteria that award points for how well a bidder proposes to mitigate identified risks, with every weighting set out in the procurement documents. HPC procurers report that criteria with no usable metric have to be weighted by hand. UK guidance asks for at least two independent evaluators and a moderated consensus score, with the reasons recorded. The written evidence trail is your defence, and that is a good reason to give the criterion a modest weight.
Building the scenario library from simulation and public benchmark datasets
A gating benchmark needs a scenario library with ground truth. You do not have to record all of it yourself: reuse public benchmark datasets where the sensors and environment match, generate synthetic scenarios to fill coverage gaps, and keep manual labelling for validation and for whatever neither source covers. TechnoLynx builds this kind of library through its Computer Vision R&D work. The first step is a coverage review that checks your scenario classes against the public datasets that already exist.
Synthetic generation gives exact ground truth, because the simulator knows every pose and object boundary it renders. Simulators such as NVIDIA Isaac Sim or Gazebo can export pose and bounding-box labels alongside the sensor streams. The synthetic CSE dataset shows what this looks like for multiple service robots. It covers synthetic hospital, warehouse and office environments, each in a static and a dynamic version, with stereo RGB, depth, IMU and ground-truth poses for every robot. Its authors found that dynamic objects significantly degraded the multi-robot Swarm-SLAM. Scenarios that are hard to capture in real service environments, such as unexpected dynamic objects, are where synthetic data earns its place.
Public real-world datasets cover SLAM failure scenarios recorded under real degraded conditions. SubT-MRS (CVPR 2024) spans more than 30 scenes with smoke, dust, darkness and snow, recorded on aerial, legged and wheeled robots. Its ground-truth maps were made with a laser scanner accurate to within ±2 mm. A public dataset may not match your exact rig or obstacle classes, so treat it as a base to adapt rather than a drop-in.
For large archives of unlabelled footage, sample the archive instead of re-watching it. Generate pseudo or weak labels semi-automatically, and spot-check a manually verified subset. One weakly semi-supervised detection method samples pseudo ground-truth boxes from weakly labelled images. The residual labelling error has to be measured on that verified subset and reported, never assumed to be negligible.
Synthetic data has a known blind spot. The reality gap exists because simulations are simplified to keep them computationally feasible. Domain randomisation, which exposes algorithms to many randomised simulated environments, is one way to narrow it. We recommend keeping real or public recordings for every scenario class that the gate is allowed to fail.
Wiring the benchmark harness into ROS 2 CI so a failing scenario class blocks the merge
To make a failing scenario class block a merge, run the harness as a required status check. It should execute a fixed scenario subset and exit non-zero when any class score drops below its threshold, and the per-commit subset has to fit a runtime budget the team can sustain. TechnoLynx delivers this as the Production AI Monitoring Harness. The engagement starts by agreeing your acceptable CI time per merge and which scenario classes are allowed to block.
The blocking mechanism itself is built into both major platforms. GitHub branch protection can require named status checks to pass before a pull request merges. GitLab’s “Pipelines must succeed” setting needs a successful pipeline to actually exist. A merge request with no pipeline, or with a skipped one, cannot merge. For ROS 2, the ros-tooling action-ros-ci action assembles the workspace, then runs colcon build and colcon test, so the harness can sit inside the test step. The github-action-benchmark action compares results with previous ones and, if set to, fails the workflow when a result is worse than the alert threshold. The per-class thresholds themselves (detection accuracy, trajectory error) are a separate design decision, and the harness should read them from versioned configuration.
What runtime budget per CI run keeps the gate from becoming a bottleneck?
Set a duration budget and defend it. One published example caps the required set at six minutes, so adding a test means moving or deleting another. A typical professional ROS 2 pipeline runs build, static checks, unit tests and integration tests on every pull request. Heavier end-to-end checks can be triggered on a schedule or on release branches. If the full scenario library fits inside the budget, run all of it on every merge. If it does not, gate on a representative subset and run the full library nightly to catch what the subset misses.
Setting up the gate in order:
- Agree the per-merge time budget and the list of blocking scenario classes.
- Pick a subset that covers every blocking class at least once, then time it on the CI runner.
- Pin the harness and inference configuration in executable code in the repository, not only in written descriptions.
- Make the harness job a required check in branch protection or merge settings.
- Schedule the full library on a slower cadence or on release branches.
The github-action-benchmark maintainers report swings of roughly ±10-20% on the repository’s own example benchmarks, and more when they touch network or file I/O. To keep flaky simulation scenarios from failing the gate falsely, allow one counted retry and tag infrastructure failures separately from test failures. Every scenario added to the required set consumes the duration budget. Each class the gate runs has to justify its cost against the team’s throughput.
Who can build the scenario library and CI gate for our robot benchmark?
TechnoLynx builds both halves of this: the labelled scenario library and the merge-blocking benchmark gate in your ROS 2 CI. For the library, the Computer Vision R&D work covers framing scenario coverage against public datasets, choosing the mix of public, synthetic and sampled archive data, building the simulator ground-truth export, and validating it against manually checked subsets. For the gate, the Production AI Monitoring Harness covers sizing the per-commit subset, wiring the harness into GitHub Actions or GitLab CI as a required check, confirming that failing classes really block merges within budget, and maintaining the nightly full-library cadence.
Simulation-based testing still needs human judgement to assess safety. For hardware choice rather than pipeline engineering, LynxBenchAI is the measuring tool.
One property runs through all three disciplines
A rubric weight, a ground-truth label and a CI threshold all do the same job: each one decides in advance what a number will mean. The weights and evidence bands for a tender can be locked before publication. Version a scenario library and its thresholds like code, logging every change with the reason for it.
