An MLPerf submission tells you a config can hit a throughput number under MLPerf’s rules. It does not tell you what that config will cost per request on your traffic, under your latency SLO, with your batching regime. Those two statements sound close. They are not, and the gap between them is where most benchmark-led hardware decisions quietly lose money.

MLPerf presents itself as a scoreboard, and in a narrow sense it is one. Submissions land in a public table: tokens-per-second, queries-per-second, offline and server scenarios, all run against a fixed accuracy target under standardised rules from MLCommons. The number is real and it is auditable. The failure isn’t in the benchmark — it’s in reading the top-line figure off a submission and treating it as a proxy for what the same config will do inside your stack. That substitution is the whole problem, and it is worth being precise about why it happens.

What does an MLPerf submission actually measure?

Strip away the scoreboard framing and an MLPerf inference result is a tightly scoped statement: this hardware, running this software stack, tuned by this submitter, sustained this throughput while holding a fixed accuracy target under one of the defined scenarios. Every clause in that sentence carries weight.

The accuracy target is fixed by the benchmark, not by you. MLPerf inference tasks define a quality bar — a percentage of the reference model’s accuracy that a submission must clear to be valid. That keeps submissions honest, but it also means the number tells you nothing about whether the model’s quality is right for your task. A config that passes the MLPerf accuracy gate can still be wrong for your domain, because domain fit is a task-specific eval question, not a throughput one. We cover that boundary in more depth in what an MLPerf result tells you (and what it can’t) in an LLM procurement eval.

The software stack is vendor-tuned. Submitters optimise aggressively and legitimately — kernel selection, quantisation to the precision the accuracy target permits, batching tuned to the scenario, sometimes custom TensorRT or vendor-specific graph compilation. This is fair within MLPerf’s rules, but it means the reported number reflects an expert-tuned config, not the out-of-the-box behaviour you’ll see when you deploy the same silicon through your own serving runtime. Reproducing an MLPerf figure on your own infrastructure is itself an engineering project.

And the throughput is measured under a specific batching regime chosen to maximise the score. That regime may bear no resemblance to how requests actually arrive in your service.

How do MLPerf’s offline and server scenarios differ?

The two scenarios that matter most for serving decisions are offline and server, and confusing them is one of the more common reading errors.

Offline assumes all queries are available at once. The system is free to batch maximally, pack the accelerator, and report the highest sustained throughput the hardware can deliver. It answers the question: how much total work can this config grind through if latency doesn’t matter?

Server models requests arriving as a Poisson stream and enforces a latency constraint — a bounded fraction of queries must complete within a target latency. This is closer to a production serving config, because real traffic arrives over time and real users have latency expectations. But even server-scenario numbers are measured against MLPerf’s chosen latency bound and arrival pattern, not your p95 SLO and your traffic mix.

Here’s the reading rule that follows: an offline number is an upper bound on hardware capability, useful for ranking silicon, useless for capacity planning under an SLO. A server number is closer to reality but still a reference-condition figure. Neither is a serving-economics metric.

Scenario	What it models	What it’s good for	What it does not tell you
Offline	All queries available at once, batch maximally	Ranking peak hardware capability	Throughput under any latency constraint
Server	Poisson arrivals, MLPerf latency bound	Approximate sustained serving throughput	Your p95 SLO, your traffic mix, your batch dynamics
Both	Fixed accuracy target, vendor-tuned stack	Qualifying a hardware-and-software config	Cost-per-request, cost-per-token on your workload

Where do MLPerf results stop short?

The divergence point is specific: MLPerf tells you a config can hit a throughput number under its rules. It is not measured in cost-per-request, it is not measured under your latency SLO, and it is not measured on your traffic mix. Those are exactly the quantities you bill against.

Three structural gaps produce the divergence, and they are worth naming because each one is fixable only by measuring on your own workload — not by reading harder.

The first is the batching gap. MLPerf’s throughput comes from a batching regime tuned for the score. Your delivered throughput depends on how requests actually arrive — bursty, uneven, mixed prompt lengths — and on how your serving runtime forms batches under those conditions. Continuous batching in a runtime like vLLM or SGLang behaves very differently from the static batches an offline submission may use. This is an observed pattern across the serving engagements we work on: delivered throughput under real arrival patterns commonly lands well below the offline headline, and the size of that gap is config-specific, not a fixed discount you can assume.

The second is the precision gap. The MLPerf accuracy target permits a level of quantisation. Your task’s quality bar may not tolerate the same precision, and dropping to FP8 or INT8 to chase the benchmark number can degrade output quality in ways the MLPerf accuracy gate never tested — because it tested a different task. We treat numerical precision as a first-class trade-off, not a free throughput lever.

The third is the cost gap. A throughput number is not a cost number. Cost-per-request depends on the accelerator’s hourly cost, the utilisation you actually sustain, the p95 latency your SLO forces you to hold headroom for, and the share of prefill versus decode in your traffic. Two configs with identical MLPerf server numbers can have materially different cost-per-request once those factors land, because one holds its throughput at your latency bound and the other collapses. This is the same margin metric the broader machine-learning model metrics that actually decide a serving config work is built around.

How should you use MLPerf results — shortlist, not decision?

Used correctly, MLPerf shortlists. Used as a headline, it substitutes a lab number for the margin metric you actually bill in. The correct workflow is a two-stage one, and keeping the stages separate is the discipline that protects the decision.

Stage one is qualification. Read MLPerf to answer: which hardware-and-software configs are capable of the class of throughput my workload needs? This is a filter. It legitimately narrows a long list of accelerators to a short list of candidates, and MLPerf is genuinely good at this because the reference conditions are standardised across submitters. A config that can’t clear your rough throughput class under MLPerf’s favourable conditions certainly won’t clear it under yours.

Stage two is decision. Take the shortlist and measure the serving-economics metrics — cost-per-request, cost-per-token, p95 latency — against your own deployed path. This is where the gap between benchmark throughput and delivered throughput gets made explicit, one config at a time, on your traffic. This is precisely the work the [inference cost-cut pack](Inference Cost-Cut Pack) runs: it takes the qualified configs and produces a real cost-per-request before/after against your deployed path, rather than a tokens-per-second headline.

The reason the stages must stay separate is that they answer different questions with different evidence classes. Stage one runs on MLPerf’s published, auditable numbers. Stage two runs on measurements from your own infrastructure — the only place cost-per-request is real. Collapsing them means letting a reference-condition benchmark decide a serving-economics question it never measured.

How do you map an MLPerf datapoint to a cost-per-request range?

The bridge from an MLPerf number to a decision you can defend is a projected cost-per-request range, not a point estimate. You build it by taking the MLPerf server throughput as a ceiling, applying a delivered-throughput discount from your own arrival pattern, then dividing accelerator cost by the realistic sustained request rate at your p95 latency bound.

Here is a worked example with the assumptions made explicit. Suppose an MLPerf server submission reports a config sustaining a given queries-per-second at a 2-second latency bound. Your SLO is a tighter p95, and your traffic is burstier than a clean Poisson stream. In configurations like this we typically project delivered throughput as a band — a plausible low and high fraction of the reference number — rather than a single figure, because the true discount only resolves once measured. Dividing your accelerator’s hourly cost across that throughput band yields a cost-per-request range.

Input	Source	Evidence class
MLPerf server throughput	Published submission	benchmark
Delivered-throughput fraction	Your arrival pattern, measured or projected	observed-pattern
Accelerator hourly cost	Cloud/on-prem list price	benchmark
p95 latency bound	Your SLO	your requirement
Projected cost-per-request range	Computed from the above	observed-pattern

The output is a shortlist where each MLPerf datapoint maps to a cost-per-request band for your deployment, with the benchmark-to-delivered gap visible rather than hidden inside a headline. That range is a decision-grade artifact; a single tokens-per-second number is not.

Closing the delivered-throughput gap is ultimately a profiling problem on your own hardware. Reference-condition throughput becomes real per-config latency and utilisation only when you measure it, and that measurement is GPU profiling on your own workload territory — where MLPerf’s number becomes a starting hypothesis rather than an answer. For teams running production inference on SaaS-scale traffic, the AI infrastructure and SaaS practice is where that reconciliation between benchmark and bill happens.

FAQ

How does mlperf results actually work?

MLPerf results are published submissions from MLCommons showing that a specific hardware-and-software config sustained a given throughput — tokens-per-second or queries-per-second — while holding a fixed accuracy target under a defined scenario. In practice, a result means “this config, tuned by this submitter, can hit this number under these rules.” It qualifies hardware capability; it does not decide a deployment.

What does an MLPerf submission actually measure — which scenarios, accuracy targets, and rules sit behind a single throughput number?

A submission measures sustained throughput at a fixed accuracy target — a percentage of the reference model’s quality that the submission must clear to be valid — under a chosen scenario (offline or server), with a vendor-tuned software stack. Every number reflects an expert-tuned config, a benchmark-chosen batching regime, and a benchmark-chosen accuracy bar. None of those are your task’s quality bar or your traffic.

How do MLPerf’s offline and server scenarios differ, and which one is closer to a production serving config?

Offline assumes all queries are available at once and batches maximally, reporting peak hardware capability with no latency constraint. Server models Poisson arrivals under a latency bound and is closer to a production serving config. Server is the more realistic of the two, but it still uses MLPerf’s chosen latency bound and arrival pattern, not your p95 SLO or traffic mix.

Where do MLPerf results stop short — why can’t a leaderboard throughput number tell you your cost-per-request or p95 latency on your traffic?

Because the number is measured under a batching regime, accuracy target, and arrival pattern chosen by the benchmark, not by you. Three gaps follow: your batching differs from the tuned regime, your quality bar may not tolerate the benchmark’s precision, and a throughput figure is not a cost figure. Cost-per-request depends on utilisation, accelerator cost, and the latency headroom your SLO forces — none of which MLPerf measures.

How should you use MLPerf results as a shortlist input rather than a config decision, and what still has to be measured on your own workload?

Use MLPerf to qualify which configs are capable of your rough throughput class — that is a legitimate filter. Then measure cost-per-request, cost-per-token, and p95 latency against your own deployed path to decide. The serving-economics metrics only become real on your infrastructure, so config selection stays in stage two, on your workload, not on the leaderboard.

How do you map an MLPerf datapoint to a projected cost-per-request range for your deployment?

Take the MLPerf server throughput as a ceiling, apply a delivered-throughput fraction derived from your own arrival pattern, and divide accelerator hourly cost by the realistic sustained request rate at your p95 latency bound. Because the delivered fraction is uncertain until measured, express the result as a cost-per-request range, not a point estimate. The band, with the benchmark-to-delivered gap made explicit, is the decision-grade artifact.

What does a table comparing an MLPerf reference number against delivered throughput on your traffic look like?

It lists the MLPerf server throughput (benchmark class), the delivered-throughput fraction from your arrival pattern (observed-pattern class), your accelerator hourly cost, your p95 SLO, and the computed cost-per-request range. The point of the layout is to keep the reference number and the delivered number in separate columns so the gap between them is visible rather than collapsed into a single headline.

Where a benchmark stops and a bill begins is the whole question. MLPerf answers whether a config can; only measurement on your own traffic answers what it costs — and if you are choosing a serving config, the second answer is the one you sign.