MLPerf Inference Benchmark: Reading NVIDIA Results for Procurement

How to read an MLPerf Inference benchmark result as procurement evidence: scenario, division, precision and batch settings, and what the number omits.

MLPerf Inference Benchmark: Reading NVIDIA Results for Procurement
Written by TechnoLynx Published on 11 Jul 2026

An MLPerf Inference result is reported for one benchmark, scenario, division and system under test. It only counts as procurement evidence once you have checked those four things against your own deployment, before you compare anything. MLPerf is developed by MLCommons, a consortium drawn from academia, research labs and industry, and its evaluations run under prescribed conditions. NVIDIA publishes per-GPU performance records based on its MLPerf submissions. What decides whether the number says anything about your workload is its scenario, division, precision and chip count.

How Does NVIDIA MLPerf Work, and How Should You Read It in Practice?

MLPerf Inference is a semantic-level benchmark. It specifies the task and the rules, and leaves the implementation to the submitter, who may optimise the reference model, run it through their preferred toolchain and execute it on their chosen hardware. An “NVIDIA MLPerf result” is therefore NVIDIA’s own implementation, run on hardware of its choice. In the v4.1 round, for example, NVIDIA applied FP4 quantisation with its TensorRT Model Optimizer library and stayed inside the closed division’s accuracy requirements.

Every submission is measured with LoadGen, the mandatory C++ harness (with Python bindings) that generates queries, tracks latency, validates accuracy and computes the final metrics. Rounds run twice a year: the .0 round in February and the .1 round in August. In the first round (v0.5, October 2019), 14 organisations sent in more than 600 submissions, and 595 were cleared as valid after review.

In practice, read a vendor’s MLPerf headline as the ceiling of that vendor’s own stack on a standard model, not as a forecast for yours. Start with the footnote. NVIDIA’s own footnotes say that per-GPU performance is not a primary MLPerf Inference metric and was calculated by dividing reported throughput by the number of accelerators. The same footnote also says that some of the Hopper comparison results were not verified by the MLCommons Association. Both details change how much weight the chart can bear.

Which MLPerf Inference Scenario Maps to My Deployment?

MLPerf Inference started with four scenarios: single-stream, multistream, server and offline. Together they emulate the workload behaviour of mobile devices, autonomous vehicles, robotics and cloud-based setups. The current submission rules also name Interactive as a scenario. Datacenter systems run Offline, Server and Interactive, and edge systems run SingleStream, MultiStream and Offline.

Scenario What it models Primary metric Closest deployment pattern
SingleStream One query at a time 90th-percentile latency of the query stream An edge device or tool handling one request at a time
MultiStream Several concurrent input streams under a QoS bound Number of streams supported while meeting QoS Multi-camera vision, autonomous-vehicle sensor fusion
Server Poisson query arrivals under a tail-latency bound Queries per second achievable while meeting QoS, reported as scheduled samples per second An online inference API with a latency SLO
Interactive Server traffic with tighter time-to-first-token and per-user token-rate limits Throughput at the stricter thresholds Chat and assistant products
Offline All samples available at once, batch size unconstrained Throughput in samples per second Overnight batch scoring, bulk document processing

The thresholds are concrete. For Llama 3.1 8B in v5.1, the server scenario requires a 2-second time-to-first-token (TTFT) and 10 tokens per second per user, and the interactive scenario requires 0.5 seconds and 33 tokens per second per user. For DeepSeek-R1, the server scenario sets a 2-second TTFT threshold and a 12.5 tokens/second/user target at the 99th percentile.

Choose the scenario whose latency bound is at least as strict as your own SLO. If your SLO is tighter than any scenario’s bound, the published result does not describe your operating point. The choice matters because, in the MLPerf Inference paper, every system delivered less throughput in the server scenario than in offline. The latency constraint caused a loss of as little as 3% on some systems and as much as 50% on others. An accelerator can lead in one scenario and trail in another, because batching helps or hurts depending on the use case.

Fix the Workload Specification Before Comparing Any Accelerator

An apples-to-apples comparison starts with a written workload specification. LynxBenchAI publishes results showing how each device runs the public PyTorch stack, so a first step is checking your shortlisted devices there. It fixes batch size, sequence length or image resolution, numeric precision and batching policy, and it treats training and inference as separate workloads. It also names the metrics you will decide on: sustained tokens per second at a fixed context length, p99 latency, cost per request, or accuracy within a latency budget. Peak TFLOPS from a spec sheet is not one of them. TechnoLynx writes that specification with your team as the first step. TechnoLynx then selects or adapts the metrics to your deployment pattern, builds the benchmark harness across your target hardware and serving backends, runs normalised comparisons checked against vendor claims, and hands the harness over for reuse in later procurements. For the device-level picture, LynxBenchAI’s published results show how each device runs the public PyTorch stack your own models run on, so they are broadly indicative for custom PyTorch models. They do not measure your own application.

The Next Platform’s 2024 analysis of MLPerf Inference v4.1 cites a peak FP16 tensor-core rating of 1,307.4 teraflops for the MI300X, next to a measured 2,530.7 tokens per second for a single MI300X on Llama 2 70B in server mode. The same analysis notes that it cannot say how much of the peak is actually generating tokens, because MLPerf does not report GPU or memory utilisation. Peak compute and delivered tokens are different quantities, and a procurement that ranks on the first is ranking on something nobody deploys.

NVIDIA’s NIM benchmarking documentation warns that tool implementations vary and that results should only be compared when definitions align. Tools differ, for instance, on whether TTFT is included in inter-token latency. Sampling settings move results too. A 2026 study of production LLM inference benchmarks measured 6,767 tokens/s at temperature 0 against 5,573 tokens/s at temperature 0.7 on identical hardware, a 21% throughput variance from that one setting. The same study found a 50% difference in processed input tokens between two tools running the same ShareGPT dataset. When you compare llama.cpp with vLLM, or a continuous-batching backend with one that does not batch continuously, you need a backend-agnostic schema that pins batch size, quantisation level and the latency definition. A composite metric such as normalised time per output token folds queue, prefill and decode time into one per-token figure.

Adopt MLPerf’s vocabulary wherever your scenario fits it. Build a bespoke harness when it does not: an air-gapped site that cannot use internet-dependent tooling, a manufacturing line where accuracy within the cycle-time budget is the real metric, or LLM output-quality questions such as character-voice drift. Those last ones are subjective and need a human panel to back any automated score. Cost per request belongs in the same specification. Normalise it across providers’ pricing and instance shapes, and re-price it when the quotes go stale. Load drives cost more than most budgets assume. On one H100 running Mixtral 8x7B in FP16, a 2026 concurrency-aware study measured $15.25 per million tokens at 1 request per second against $0.87 at 25 requests per second, a 17.5× spread on the same hardware and model. For the cost side, see where standard benchmarks stop short of cost-per-request. For training capacity, see MLPerf Training for sizing infrastructure.

How Much Do Precision and Batch Size Move a Published MLPerf Result, and How Do You Normalise Across Submissions?

Precision and batch size need the closest reading, because the closed division lets submitters choose both within an accuracy gate. The closed division requires 99% or 99.9% of the FP32 baseline accuracy, depending on the model, and permits quantisation, compiler optimisation and hardware-specific tuning as long as that requirement is met. Some models have a high-accuracy variant that must reach 99.9% of reference rather than the default 99%. Within that gate, vendors pick very different formats. The Next Platform’s v4.1 analysis notes AMD running Llama 2 70B at FP8 on the MI300X while NVIDIA ran it at FP4 on Blackwell, which accounts for part of the gap between them. In v6.1, AMD submitted Llama 2 70B in MXFP4 on the MI355X.

MLPerf lets submitters choose the best batch size for each scenario rather than prescribing one. AMD’s v6.1 DLRM-v3 submission, for instance, ran at batch 64 on the MI355X and batch 32 on the MI350X. A ranking holds only for the batch, sequence and precision it was measured at. Change them and the order can change.

To normalise across submissions, work through these steps in order:

  1. Keep only results from the same benchmark and scenario in compatible versions. MLCommons decides which versions are compatible.
  2. Keep only closed-division results when the question is hardware against hardware (see the next section).
  3. Match the accuracy variant, 99% or 99.9% of reference.
  4. Record each submission’s weight and input data types, which submitters must declare in their model metadata.
  5. Divide total throughput by the reported chip count for a per-accelerator figure, and label it as derived. MLCommons does not allow secondary or derived metrics to be presented as official MLPerf metrics.
  6. Note the power and software configuration of each system, then test the surviving results against your own latency SLO.

What Is the Difference Between MLPerf’s Closed and Open Divisions, and Why Does It Matter for Procurement?

For a hardware comparison, compare closed-division results only with other closed-division results. An open-division result measures a different model, so ranking it against a closed result mixes up model changes and hardware differences. The closed division uses the same models, datasets and quality targets for every submitter. It requires preprocessing, postprocessing and a model equivalent to the reference, and it permits calibration for quantisation but no retraining. Retraining and pruning are prohibited precisely to keep results comparable. The open division lets submitters change the model and show different performance and quality targets, including retraining or model substitution. Open submitters may also enter only some scenarios, where closed submitters must enter every applicable scenario.

The labels carry this distinction. A closed result is called “MLPerf”, and an open result must be called “MLPerf Open”. Datacenter inference results are published in separate tables for Closed, Network and Open. NVIDIA’s own v4.1 submission shows why the line matters. A depth- and width-pruned, fine-tuned Llama 2 70B on a single H200 reached 11,189 tokens/s offline in the open division, almost 3× the closed-division throughput, with accuracy slightly below the 99% threshold. That is a real result about what model surgery can buy, and it says almost nothing about the silicon.

If a vendor has submitted only to the open division, treat the result as evidence of its best achievable performance with a modified model, and ask for a closed-division run or run your own controlled test. When open and closed results are compared, MLCommons requires the ways the open result would not qualify as closed to be identified. Even inside the closed division, systems differ in accelerator count, so normalise per accelerator before comparing.

A vendor bake-off on your own workload is fairest when it copies the MLPerf structure: publish the workload, metrics and rules before anyone submits, let each vendor tune its own configuration within those rules, require disclosure of every tuning choice, and review the results before publication. TechnoLynx builds that kind of harness and review step for a buyer’s workload, from the submission interface and configuration capture to independent re-runs of each vendor’s configuration.

What Does an MLPerf Result Not Tell You About Your Model in Production?

An MLPerf Inference result does not tell you how your model, with your input lengths, your serving stack and your traffic, will behave on that hardware. MLCommons itself distinguishes the two products. MLPerf Inference measures a system running a model in fixed scenarios at a small set of predefined operating points. MLPerf Endpoints measures a served endpoint under production-like load across the full operating range. Endpoints includes the serving stack, batching, scheduling and networking because they are part of what is bought, and it reports a Pareto frontier across concurrency levels rather than single points.

Several other things are missing from a standard result. GPU and memory utilisation are not part of the benchmarks. The ban on retraining keeps results comparable, but the paper’s authors acknowledge it may fail to reflect realistic deployment in some cases. When your measured number diverges from the published one, the gap needs profiling to attribute, whether to memory bandwidth, scheduling, preprocessing or a kernel that never got the vendor’s tuning. Guessing from best-practice rules is not enough. That is why a representative benchmark of your own workload on each shortlisted SKU should settle the decision, with the MLPerf result as corroboration.

How Do I Turn an MLPerf Benchmark Number into Defensible Procurement Evidence?

An MLPerf number becomes defensible evidence when the methodology around it is pre-registered before any test runs, discloses every configuration and measurement detail, says plainly whether it complies strictly with MLPerf Inference rules or adds a supplementary protocol, and labels each number as vendor-submitted or independently run. TechnoLynx designs that pre-registered protocol for your workloads, aligned with MLPerf Inference. TechnoLynx fixes the disclosed hardware, backend and precision before execution, then runs and reports the results with provenance labels that a procurement reviewer can follow. LynxBenchAI’s published results show how each device runs the public PyTorch stack, not your own workload. Approval-grade evidence of this kind is an engineering output a non-engineer can read and sign against, not a marketing artefact.

LynxBenchAI gives the device-level baseline for that protocol: its published results show how each device runs the public PyTorch stack, and a first step is to check your shortlisted devices there.

MLCommons’ own process is the precedent worth borrowing. A submission must contain system metadata, the benchmark code, setup and execution scripts and result logs for every system-implementation combination. Closed submitters must run compliance tests, and the onus is on them to explain any flagged issue to the review committee. The validation suite checks for caching, by comparing runs with unique and duplicated sample indices, and for seed-specific optimisation, by replacing the official random seed. Audits are more likely when performance does not match known hardware characteristics, or when the reviewers cannot see how a result was achieved. The MLPerf Inference paper’s authors concluded that auditing and auditability are necessary for result integrity and reproducibility. The stated purpose of review is fairness, not litigating details to disqualify competitors.

Vendor sign-off has a ceiling. It confirms that a vendor agrees its own numbers are reported accurately. It does not confirm the comparison was fair across vendors, since the closed division lets each vendor choose batch size and precision within the rules. Pre-registration protects you against claims of post-hoc bias, not against disputes over whether the workload was relevant. Contract terms that restrict publishing benchmarks are a legal question separate from methodology.

An Evidence Record for Each MLPerf Result You Rely On

Field Why it is on the record
Submitting organisation, benchmark name and version, system under test MLCommons requires these in any disclosure of results
Suite, division, scenario, retrieval date and source, result ID Required in the footnote of any reference to a result
Chip count Any comparison must identify differences in chip count
Category: Available, Preview or RDI Categories reflect whether the system can be bought or rented
Software versions, hardware configuration, containers and settings Required for Available submissions, for reproducibility
Weight and input data types Declared in submission metadata
Verified or unverified status Unverified results must be disclosed as unverified
Derived metrics (per-chip, per-dollar), labelled as such Derived metrics must not be presented as official MLPerf metrics
The SLO and scenario mapping this result supports This is the line that makes the record usable at review

The record matters more than it seems. A 2026 Digital Applied audit of 42 benchmark rows published by nine vendors found none that was both fully disclosed on the vendor’s own page and independently confirmed. Those rows were model-capability benchmarks rather than hardware results, but the audit’s recommendation carries over: treat vendor tables as claims pending qualification.

If the comparison will be published, the write-up also needs the run conditions that move numbers, such as power limits and software versions. NVIDIA’s own v4.1 notes report that raising the H200’s thermal design power from 700 W to 1,000 W gave up to 12% more performance on Llama 2 70B.

Can We Reproduce a Vendor’s MLPerf Submission on Our Own Air-Gapped Rig?

Yes, provided you stage every artefact in advance and replicate the vendor’s hardware and software stack. Even then, expect a small spread rather than an identical number. TechnoLynx sets this up as a repeatable procedure. TechnoLynx agrees with you which scenario matches your real-time constraint and what reproduction tolerance is acceptable, extracts the matching submission’s configuration from the public MLCommons artefacts, stages code, models and weights on the air-gapped rig, configures LoadGen to the declared settings, then runs, compares and documents every deviation caused by the environment. Replicability is not optional in MLPerf: results that cannot be replicated are not valid results.

LynxBenchAI provides a reference point for that reproduction: its published results show how each device runs the public PyTorch stack, so a first step is checking the shortlisted device there before staging the rig.

The procedure runs in this order:

  1. Match the scenario to the constraint. SingleStream fits a strict per-query latency limit, Server fits latency-bounded concurrent traffic, and Offline fits maximum throughput with no latency bound.
  2. Pull the submission. The mandatory files include mlperf_log_summary.txt, mlperf_log_detail.txt, user.conf, any calibration or weight-transformation code, and READMEs for replicating performance. The source code must be sufficient to reproduce the result.
  3. Pre-stage everything. Vendor submission containers can package the runtime, quantisation kernels, inference server and harness configuration, as AMD’s v6.1 Docker images do.
  4. Use LoadGen unmodified, since no local modifications to its C++ library are allowed for submission. Load TestSettings from the reference config files, because LoadGen cannot enforce scenario requirements itself.
  5. Enable ECC. Datacenter submissions must use ECC in DRAM and HBM for all performance and accuracy runs.
  6. Match the host stack. AMD’s v6.1 guide notes that an identical DLRM-v3 container fails on host stacks older than the stated kernel, driver and ROCm versions.
  7. Run accuracy and performance modes, then compare. AMD tells reproducers to expect run variability of generally within 3%. It lists ECC disabled, a development Docker image, mismatched dataset preprocessing and thermal throttling as causes of larger deviations.

Host-stack differences from the vendor’s environment can block an exact match. In that case, document the residual gap as the finding.

What Goes into a Board-Ready Evidence Package Built on MLPerf Results?

A board-ready package pairs independently governed benchmark evidence at the quantisation and batching you will actually run with a TCO and latency analysis whose assumptions are written down, so it still holds if costs or latency diverge after purchase. TechnoLynx takes the cost side through the Inference Cost-Cut Pack, which ties the hardware evidence to cost per request at your latency SLO rather than to list prices.

Published work shows how the pieces fit. The Next Platform added GPU and full-system costs to MLPerf Inference v4.1 Llama 2 70B results to compare MI300X, H100 and H200 on value for money. It estimated roughly 2,700 tokens/s in server mode for a single H100, about 7% ahead of the MI300X, and a 56% gain for the H200 against a 33% higher GPU price. Those figures are the publication’s own estimates, and they rest on a single model, Llama 2 70B. In a board pack they belong as corroboration, labelled as derived.

The cost model should take the SLO first and derive cost from it. The concurrency-aware study above treats cost as a function of hardware, model, quantisation, offered request rate and latency SLO, and recommends declaring the SLO before deriving cost. It also shows that utilisation is an output of those variables, not an input a calculator should ask you to guess. State utilisation, power cost and amortisation period explicitly. If you cannot find a close published MLPerf result for your hardware at your precision and batching, the package has to rest on internally run, documented benchmarks with the methodology attached. How this evidence fits approver scope and governance frameworks is covered in our NIST AI RMF crosswalk for approval-grade evidence.

Who Can Turn MLPerf Results into Hardware Evidence for Our Own Workload?

TechnoLynx turns MLPerf results and your own workload into hardware-selection evidence that a procurement or model-risk review can sign against. On build in-house or bring in a specialist: an in-house team with spare GPU time and benchmarking experience can run a one-off comparison. A specialist pays off when the result must survive vendor challenge or be re-run on later procurements, because the harness, disclosure record and reproduction procedure are the hard part. On choosing a partner: ask whether they will reproduce a vendor submission, label provenance and hand the harness over. TechnoLynx’s AI infrastructure and production SaaS engineering work covers this, with LynxBenchAI as the measuring tool for the hardware itself.

The engagement does not suit a buyer who needs a single quick device ranking. For that, published closed-division results read with the record above, or LynxBenchAI’s published results, may be enough. Nor does it give policy advice, certification or regulatory sign-off. It produces the artefacts a review needs. A team that only wants rented hours on a benchmark rig is better served by a cloud provider directly.

When Your Workload Has No MLPerf Benchmark

Some workloads have no MLPerf benchmark, yet their results still need to be accepted as comparable. The honest route is an MLPerf-style methodology: fixed workload, disclosed configuration, accuracy gate and LoadGen-like traffic. Describe it in those terms and never as an MLPerf result. MLCommons requires claimed results to comply with the benchmark’s rules, and unverified results to be marked as unverified.

LynxBenchAI covers the device side of such a workload: its published results show how each device runs the public PyTorch stack, so a first step is to check your shortlisted devices there before designing the suite.

Pathology WSI Inference and MONAI Segmentation

The current datacenter task list includes medical image segmentation in the Offline scenario, but no whole-slide image classification task. The segmentation model, 3D U-Net, has a high-accuracy variant at 99.9% of reference. That makes segmentation the closest anchor, while WSI classification needs its own representative suite. That suite should define tile sizes, tile counts per slide and batch composition from real lab throughput, and publish them before any vendor runs.

A Shared Suite Across Scientific-AI and LLM Kernels

A multi-party capability suite that mixes HPC kernels such as HPL with transformer training and inference sits outside MLPerf Inference entirely. The weighting between them is a policy decision, not a measurement. Publish the weights and the mandatory-versus-optional kernel list before submissions open, so that no member can argue the score was tuned after the results came in.

FAQ

What Is MLCommons MLPerf?

MLPerf is a set of benchmarks developed by MLCommons, a consortium drawn from academia, research labs and industry, to evaluate training and inference performance under prescribed conditions. MLPerf Inference results are measured with the mandatory LoadGen harness.

Is a Bigger Batch Size Always Faster?

Bigger batches help offline throughput, but a latency bound limits them. In the MLPerf Inference paper, every system delivered less throughput in the server scenario than offline, because the latency constraint makes large batches harder to form.