MLCommons is the member organisation behind MLPerf. Its MLPerf Inference suite measures one thing very carefully: how much throughput a specific hardware-and-software stack delivers for a fixed reference model, at a set accuracy floor, while staying inside a set latency bound. That makes it an architecture-neutral, audited reference for inference performance. It does not report your cost per request at your product’s p95 latency. A top MLPerf placement is a strong starting point for a serving-config decision, and the margin calculation still has to be redone on your own serving path.
The rest of this piece covers what the suites measure, how their latency limits compare with your SLO, which figures carry over to a cost model, and how to use MLCommons’ measurement discipline as the baseline for a config choice you can defend.
Who Sets MLPerf Benchmarks, and Why Results Are Comparable Across Vendors
A working group of experts defines each MLCommons benchmark suite. The group picks the AI model to run and the dataset to run it on, sets the rules on which changes to the model are allowed, and then measures how fast a given piece of hardware runs that model. MLCommons states five goals for MLPerf: fair comparison of competing systems without blocking innovation, faster AI progress through useful measurement, enforced reproducibility, service to both commercial and research users, and benchmarking cheap enough that everyone can take part. On scale, MLPerf has over 125 member organisations and more than 90,000 reproducible results, and IEEE and ISO/IEC SC42 recognise it.
In practice, MLPerf is an instrument for fair comparison. It pins down the model, the dataset, the quality target and the load pattern. The pinning is what makes results comparable across vendors. It is also why a result says little about your own model, data or traffic.
The Suites and How They Fit Together
| Suite | What it measures | Source |
|---|---|---|
| MLPerf Training | Time to solution for training; built to handle the stochastic, high-variance nature of training | Mattson et al. |
| MLPerf Inference: Datacenter | How fast systems process inputs and produce results with a trained model | MLCommons |
| MLPerf Inference: Edge | The same question, on edge systems | MLCommons |
| MLPerf Inference: Mobile | The same question, on consumer mobile devices with different AI chips and software stacks | MLCommons |
| MLPerf Endpoints | A served endpoint under production-like load across a range of concurrency levels, consumed through a live API | MLCommons |
MLPerf Inference v6.0 added LoadGen++, an optional harness that runs LLMs on a serving-style software stack like those used in typical deployments. In the v6.1 round, MLCommons reported that over half of submitters used its new API-centric harness, which it describes as the foundation of the next-generation MLPerf Endpoints suite. The Training suite benchmarks ML training, a different question from inference. MLPerf Training for infrastructure sizing covers it separately.
What’s Worth Understanding About MLCommons First
The first thing to grasp is the “tripod” of model, dataset and rules on allowed model changes. Each MLPerf Inference benchmark comes with a canonical reference model, and in the closed division any implementation that is mathematically equivalent to that model counts as valid, with some deviations (such as numerical formats) allowed. The original paper set the quality bar for almost all implementations at within 1% of the FP32 reference accuracy. For Llama 3.1 405B in v5.0, the closed-division threshold is 99% of the FP16 reference.
So an MLPerf number answers a narrow question: how fast a particular stack serves this reference model at this accuracy floor. It says nothing about whether that model’s answers are good enough for your data or your workflow. We hold that as the single most useful thing to remember when an MLPerf figure turns up in a procurement deck. Our MLPerf explainer for AI procurement takes this from the buyer’s side.
What Does MLPerf Inference Actually Measure, and What Do the Server and Offline Scenarios Mean?
MLPerf Inference measures latency-bounded throughput: how much work a system completes while a stated share of queries finish inside a latency limit. Its authors chose this metric over raw latency or raw throughput because it reflects the constraints of real deployments. The suite defines four scenarios, each with its own query pattern and metric:
| Scenario | How queries arrive | Reported metric | Example use cases from the paper |
|---|---|---|---|
| Single-stream | One query at a time; the next is sent when the previous completes | 90th-percentile latency | Typing autocomplete, real-time AR |
| Multistream | Queries arrive at a fixed interval and late ones are dropped | Number of streams sustained within the latency bound | Multi-camera driver assistance |
| Server | Poisson-distributed arrivals | Queries per second within the latency bound | Translation website |
| Offline | Everything available up front as one batch | Throughput | Photo processing |
In the Server scenario, random query arrivals force the system to optimise for tail latency at the cost of suboptimal batching. In the Server scenario, LoadGen binary-searches for the highest queries-per-second value at which the run still passes. The figure reported is scheduled samples per second, not completed samples per second. Offline maps to bulk jobs with no latency constraint, which the paper places at the far end of a QoS range that runs from no constraint at all to milliseconds for real-time applications.
Why Server Throughput Is Lower than Offline, and by How Much
Server and Offline differ in two ways. Server imposes a latency limit, and its arrival rate is not uniform. Both make large batches harder to form, so throughput drops. In the original MLPerf Inference paper, every system delivered less throughput in Server than in Offline. For some systems the loss was as small as 3%, and for others it reached 50%.
That spread is the important part. An accelerator can stand out in one scenario and underperform in another, because batching is use-case dependent. The authors drew two conclusions. A comparison with unconstrained latency “has little bearing” on a latency-constrained scenario. And the effect of latency constraints extrapolates poorly from one model to another, even among classification models. For cost work, the implication is that an Offline headline cannot stand in for a serving decision, and a Server result for one model cannot stand in for a different model.
How Does the MLPerf Latency Constraint Relate to Your Product’s p95 Latency SLO?
MLPerf’s latency constraints are set per model and scenario. Your SLO is a target your product sets, stating how and under what conditions it is measured. For LLMs, MLPerf splits latency into time to first token (TTFT), which covers the prefill phase, and time per output token (TPOT), which covers decoding.
The published limits differ by benchmark and by category:
- Llama 2 70B, interactive: 99th-percentile TTFT of 450 ms and TPOT of 40 ms. MLCommons set these after reviewing industry research, user surveys and platform data in late 2024.
- Llama 2 70B, conversational: TTFT/TPOT of 2,000 ms/200 ms.
- Llama 3.1 405B, server: 99th-percentile TTFT of 6 seconds and TPOT of 175 ms, reflecting long-context inputs averaging 9,400 tokens.
- MLPerf Endpoints: reports TTFT at the 90th percentile.
These limits fail to map onto your SLO for three separate reasons. First, the percentile differs. A p99 bound constrains the tail more tightly than a p95 bound at the same threshold, but your threshold may be tighter or looser than MLPerf’s. Second, the definitions differ. MLPerf times latency from the moment LoadGen is scheduled to pass a query to the system until it receives the reply. Third, the workload differs. Prefill-decode interference means prioritising one of TTFT or TPOT risks missing the other’s limit.
The Google SRE book’s guidance applies here: an SLO should state how it is measured and the conditions under which it holds, and averages hide the tail that high percentiles expose. Read an MLPerf limit as a statement of the conditions MLCommons chose, not as a proxy for yours. How to pick and fix your own percentile and metric belongs to a sibling piece on what to measure so the number decides cost.
Where Does MLPerf Stop Short of a Cost-per-Request, Margin-Based Comparison?
MLPerf Inference fixes conditions that your deployment sets differently. It runs the reference model rather than yours. It reports a small set of predefined operating points rather than your load. It prohibits some optimisations, such as cross-query caching. And price sits outside its purview. Each of these choices is right for a fair benchmark, and each leaves a gap in a cost model.
The model and division. The closed division requires a model equivalent to the reference, permits calibration for quantisation and prohibits retraining. The open division allows different models and retraining, but its results are generally comparable neither with each other nor with closed results.
Caching. The rules prohibit caching queries or responses, coalescing identical queries, and caching anything tied to query indices. A KV cache is allowed only within a single query, as in the reference model. The paper explains that the ban exists to represent realistic deployments and is enforced by comparing runs with unique and duplicate sample indices. If your production stack reuses prefixes or responses across requests, that lever never shows up in an MLPerf figure, in either direction.
Fixed operating points. MLCommons itself says MLPerf Inference measures a system at a small set of predefined operating points for single-shot inference. Your cost depends on the load you actually offer. A study ran vLLM on H100 GPUs across three models and found that, once hardware, model, quantisation and latency SLO are fixed, the offered request rate dominates realised cost per token. The same study found that cost estimates which treat GPU utilisation as a fixed input understated real cost by 2.5 to 24 times at 1 to 10 requests per second, and by up to 36.3 times near idle. Getting utilisation into the cost model is covered in what to measure so the number decides cost.
Price. MLPerf Power measures full-system AC power at the wall, and only for the accompanying benchmark run. The price you pay, which depends on purchasing volume and discounts, is beyond MLCommons’ purview.
What MLPerf Endpoints Adds, and What It Still Leaves Out
MLPerf Endpoints closes part of the gap. It sends traffic to a live API endpoint over standard interfaces such as HTTP or gRPC, so the serving stack, batching, scheduling and networking are all inside the measurement. A result is a Pareto curve of at least 7 and at most 32 operating points rather than one point. Each point runs for at least 600 seconds, uses the same model and stack, and has its accuracy validated. The curve reports concurrency, interactivity in tokens per second per user, system throughput, and TTFT at P90. It is drawn as step functions rather than interpolated lines, so it shows only operating points that were verified.
Two things are still missing. First, cost normalisation is limited. System throughput per watt of provisioned power is the only normalisation MLCommons can guarantee, and MLCommons calls it a proxy for TCO. Dividing by GPUs provisioned or by dollars per GPU-hour is a derived, unverified metric. Second, the suite is young. v0.7 is a foundation release with results from five submitters across three benchmarks, and more buyer-centric rules and normalisation are planned for v1.0. Endpoints shows you the throughput-versus-responsiveness trade-off across concurrency levels. It does not show you your margin.
How Do You Read an MLPerf Inference Result Without Over-Reading It for Cost Decisions?
Read an MLPerf Inference result as a verified measurement of one system under stated conditions, and check those conditions before putting any weight on the number. This checklist covers what to confirm:
- Division: closed results compare like with like; open results may use a different model and are generally not comparable.
- Availability category: Available, Preview or Research/Development/Internal (RDI), which tells you whether you can actually buy or rent the system.
- Scenario: Server for online serving, Offline for batch. An Offline figure has little bearing on latency-constrained serving.
- Latency category: interactive, conversational or server limits, and how they compare with your own percentile and threshold.
- Model and accuracy floor: the reference model and quality target, not your model or your data.
- Harness and stack: whether the submission used LoadGen++ or the API-centric harness, which moves the measurement closer to a deployed serving stack.
- System scale: node and accelerator count. In v6.0, 10% of submitted systems had more than ten nodes, and the largest had 72 nodes and 288 accelerators.
- Power: wall power measured for that run only, not a TDP or a rating.
- Audit status: a submission is more likely to be audited when its performance is inconsistent with the hardware, reviewers lack insight into how it was achieved, or its hardware and software are not reasonably available to the public.
LynxBenchAI measures throughput, latency, energy and cost per unit of work across devices. For the measurement side of reading hardware numbers, see LynxBenchAI’s methodology and published results.
Which MLPerf Figures Transfer to Your Cost-per-Request Model and Which Must Be Re-Measured on Your Serving Path?
What transfers is relative standing under identical constraints, plus the measurement discipline. What must be re-measured is anything that depends on your model, your traffic, your SLO or your prices.
| Item | Transfers? | Why |
|---|---|---|
| Ranking of systems on the same closed benchmark and scenario | Partly, as a shortlist | Same model, data and quality target across submissions |
| Server-versus-Offline gap for a system | As a signal of batching sensitivity | The gap reflects how each system copes with latency constraints and suboptimal batching |
| Run discipline: minimum durations, query counts, confidence | Yes, adopt it | Runs last at least 60 seconds to capture power-management equilibrium, and query counts are chosen for 99% confidence in tail bounds |
| Absolute throughput at the benchmark’s latency limit | No, re-measure | Constraint effects extrapolate poorly across models |
| Latency at your percentile and metric definition | No, re-measure | MLPerf’s percentiles and timing boundaries are its own |
| Throughput at your offered load and concurrency | No, re-measure | Realised cost depends on offered request rate |
| Effect of cross-request caching | No, measure separately | Excluded by the rules |
| Cost per request | No, compute it | MLCommons guarantees only provisioned-power normalisation; price-based figures are derived |
How Do You Use MLCommons Measurement Discipline as the Baseline for a Defensible Config-Selection Decision?
Use MLPerf to shortlist systems and to borrow its method. Then make the decision on a cost-per-request figure that you measure on your own serving path at your own SLO.
TechnoLynx audits the production serving stack for cost and latency improvements before recommending model replacement. Batching, caching, routing, quantisation, kernel work and runtime choice sit below the model layer, can be tuned independently, and replacing the model is the most expensive lever available.
NVIDIA’s published sizing method follows the same order: benchmark each deployment unit for throughput and latency under load, combine that with the latency requirement and expected peak demand to size the deployment, then derive cost from the sizing. In order:
- Shortlist from MLPerf. Take closed-division Server (or Endpoints) results for the benchmark nearest your workload, restricted to Available systems.
- Fix your SLO in your own terms. Choose a percentile, a metric (TTFT, TPOT or end-to-end) and a threshold, stated with the conditions under which they hold.
- Sweep load on your serving path. Measure throughput and tail latency across offered rates or concurrency levels for each candidate configuration, with runs long enough to reach steady state.
- Discard operating points that breach the SLO. Among the points that pass, the highest-throughput one is the most economical. Endpoints recommends the same filtering for its published curves.
- Convert to cost. Yearly server cost is the purchase cost divided by the depreciation period, plus yearly hosting and licensing. Cost per request is that yearly cost divided by the requests one server handles in a year, adjusted for real uptime.
- Treat the SLA as a price. A tighter latency target lowers the load you can sustain and raises cost per token, so test the SLO itself as a variable.
As a worked form of step 5 (the inputs are placeholders, not figures): if a configuration sustains R requests per second at your SLO, one server handles R × 31,536,000 × u requests a year at uptime fraction u. Cost per request is then the yearly server cost divided by that number. The figure that matters is R at your SLO, which is exactly what MLPerf cannot give you.
Which optimisations pay off is decided by profiling in context, not by best-practice rules read elsewhere. Profiling tooling is covered in profiling GPU kernels to find the real bottleneck. TechnoLynx’s generative and agentic AI engineering builds LLM features into products. For complete before-and-after comparisons, see the cost-per-request benchmarking examples.
One question stays open. Endpoints v1.0 promises more buyer-centric rules and normalisation, and rolling submissions are planned. Until a published curve carries your model, your concurrency and your prices, the margin calculation stays your own work.
FAQ
What Is the Difference Between p95 and p99 Latency?
p95 latency is the value 95% of requests come in under, and p99 is the value 99% come in under. High percentiles show a plausible worst case, while the median shows the typical request.
Why Not Just Use Average Latency?
An average can hide a long tail. In the SRE book’s example, a typical request takes about 50 ms while 5% of requests are 20 times slower.