A vendor points at an MLPerf Inference result — a headline samples-per-second figure in the Server scenario — and the room nods. The number is real, it is neutral, and it is auditable. It is also not your cost-per-request. That gap is the whole subject of this article, because it is where most serving-config decisions quietly go wrong: the winning MLPerf submission gets chosen, deployed, and then loses money per request once your batching, concurrency, and hardware-rental economics land on top of it. MLCommons and its MLPerf Inference suite are the closest thing the industry has to a neutral benchmark standard. That is exactly why the failure is so common — the number is trustworthy enough that people stop interrogating it too early. Reading MLPerf well means knowing precisely what it measures, and being disciplined about the one thing it deliberately does not carry: the economics of your product’s serving path at your p95 latency target. How does MLCommons work in practice? MLCommons is a nonprofit engineering consortium — model vendors, hardware makers, cloud providers, and academics — that builds and governs shared machine-learning benchmarks. Its best-known output for serving decisions is the MLPerf Inference suite, a set of standardised tasks (image classification, object detection, language modelling, recommendation, and, in recent rounds, large-language-model workloads) run under tightly specified rules so that a result from one submitter can be compared against another without arguing about setup. The practical meaning is this: MLPerf is a fair-comparison instrument, not a deployment estimator. Its value comes from normalisation. Every submitter runs the same model, the same dataset, and the same accuracy target, using a common load generator called LoadGen that issues queries according to a fixed distribution. Because the workload and the latency constraint are pinned, the throughput numbers become directly comparable. The cost of that fairness is that MLPerf strips out everything specific to your product — and your cost-per-request lives entirely in that stripped-out layer. We see this pattern regularly in infrastructure reviews: a team treats a strong MLPerf placement as a procurement conclusion when it is really a well-measured starting point. The measurement discipline is genuine and worth borrowing. The conclusion has to be re-derived. What does MLPerf Inference actually measure, and what do the Server and Offline scenarios mean? MLPerf Inference reports latency-bound throughput under a small number of scenarios, and the two that matter most for serving decisions are Server and Offline. Offline assumes all queries are available at once. LoadGen fires them as fast as the system can absorb, and the metric is raw throughput — samples per second — with no per-query latency ceiling beyond finishing the batch. This models batch or asynchronous workloads where nobody is waiting on an individual response. Server models an online service. Queries arrive as a Poisson process at a target rate, and the submission is only valid if it meets a latency constraint at a specified percentile — typically a 99th-percentile bound defined per benchmark. The reported metric is the highest query rate the system sustains while still honouring that tail-latency rule. The Server scenario is the one people reach for when they want to reason about a live product, and it is the closest MLPerf gets to your world. But notice what is fixed: the latency constraint is set by the benchmark, not by your SLO. The arrival distribution is Poisson at a chosen rate, not your real traffic shape. The model, precision target, and accuracy floor are the benchmark’s, not necessarily your fine-tuned deployment’s. MLPerf answers “how many queries per second can this executor sustain under this latency rule?” — a genuinely useful, genuinely hard measurement. It does not answer “what does one of my requests cost?” For a fuller walk-through of how these numbers read in a purchasing context, our piece on what MLPerf Inference numbers mean for model procurement covers the procurement-eval angle this hub deliberately leaves to it. How do you read an MLPerf Inference result without over-reading it for cost decisions? Read it as three separate facts, and resist collapsing them into one verdict. First, the accuracy target. MLPerf submissions must clear a defined accuracy floor (for example, a minimum on the reference dataset), so a throughput number is only meaningful at that accuracy. A config that is fast because it quantised past your quality bar is not comparable to yours. Second, the scenario. An Offline number and a Server number for the same hardware are not interchangeable; the Server figure already paid the tail-latency tax, the Offline figure did not. Quoting the Offline throughput as if it predicts online capacity is the single most common over-read we encounter. Third, the executor. The result belongs to a hardware-plus-software tuple — a specific GPU, a specific serving stack, a specific set of optimisations submitted for that round. Swap the runtime, the quantisation, or the batching policy and the number moves. This is the same reason machine learning compilers change cost-per-request in production: the software layer is not a fixed multiplier on the hardware spec. Held separately, these three facts tell you which configs are worth measuring further. Collapsed into a single “config X won MLPerf, buy config X” verdict, they mislead. Where does MLPerf stop short of a cost-per-request, margin-based comparison? MLPerf normalises the workload and the latency constraint so vendors can be compared fairly. That normalisation is precisely what removes cost-per-request from the frame. Your per-request cost is a function of four things MLPerf does not carry: Batching and concurrency on your traffic. MLPerf’s Poisson arrivals at a fixed target rate rarely match your bursty, session-shaped load. The effective batch size your scheduler achieves under real traffic drives GPU utilisation, and utilisation drives cost. Your latency SLO, not the benchmark’s. If your product’s p95 is tighter than the benchmark’s tail constraint, the sustainable throughput on your path is lower than the MLPerf figure — sometimes substantially. If it is looser, you may be leaving throughput on the table. Pricing and rental economics. MLPerf reports performance, not dollars. A config that tops the throughput chart on a premium accelerator can lose per-request margin to a cheaper executor once the hourly rental rate is applied. Your model and prompt distribution. LLM serving cost is dominated by input and output token counts, which the benchmark fixes and your workload does not. The honest framing: MLPerf gives you latency-bound throughput per config; it leaves the gross-margin delta uncomputed. That delta is the number the business actually decides on. Quick-answer: what transfers from MLPerf, and what you must re-measure MLPerf-derived figure Transfers directly? What you must re-measure on your serving path Relative ordering of executors at a fixed accuracy Mostly — a fair-comparison signal Confirm the ranking holds at your precision and model Server-scenario throughput at the benchmark’s tail latency No Sustainable QPS at your p95/p99 SLO Offline throughput No (batch-only) Effective throughput under your real arrival shape Accuracy at the reference target Partially Accuracy on your task-specific eval set Cost per request / gross margin Not present Full derivation: utilisation × rental price ÷ sustained QPS (Evidence class: observed-pattern — this mapping reflects what we consistently see translating MLPerf-style results onto deployed serving paths, not a single published benchmark.) How does the MLPerf latency constraint relate to your product’s p95 latency SLO? They are the same kind of object — a tail-latency bound gating a throughput number — but they are almost never the same value, and the difference is not linear. Tail latency degrades sharply as a serving system approaches saturation, so a modest tightening of the percentile bound can cost a large fraction of sustainable throughput. If the benchmark validates at a 99th-percentile bound that is looser than your p95 target, the MLPerf throughput overstates what you can sustain. If your SLO is looser, the benchmark understates it. Either way, the only way to know the magnitude is to re-run the load against your constraint. This is where GPU profiling for per-config utilisation and latency supplies the measurements that turn an MLPerf-style result into a cost-per-request figure — you profile the config under your arrival shape and your SLO, then read utilisation and sustained QPS off the deployed path rather than off the benchmark report. For teams building serving infrastructure as a product, the economics of that translation sit at the centre of the AI infrastructure and SaaS work — where a config that “wins” a benchmark and a config that protects margin are frequently not the same config. How do you use MLCommons measurement discipline as the baseline for a defensible config-selection decision? Borrow the discipline; re-derive the decision. MLPerf’s real gift is methodological: pinned accuracy targets, a common load generator, declared scenarios, and reproducible submissions. That is the right shape for any comparison, including yours. The move is to keep that rigour and swap in your own constants. A defensible config-selection sequence: Shortlist from MLPerf. Use the relative executor ordering at the accuracy floor closest to yours to narrow candidates. This is what MLPerf is genuinely good for. Restate the constraint. Replace the benchmark’s tail-latency rule with your actual p95/p99 SLO. Restate the workload. Replace Poisson-at-fixed-rate with your measured arrival shape and token distribution. Measure sustained throughput per config under those constants, on the deployed serving path — not the benchmark harness. Apply price. Divide utilisation-adjusted cost by sustained QPS to get cost-per-request, then compute the gross-margin delta between candidates. The output is a config choice justified by margin, not by a samples-per-second headline that never mapped to margin. That translation — from fair-comparison baseline to before/after cost-per-request on your own serving path — is exactly the work the [inference cost-cut pack](Inference Cost-Cut Pack) is built to do. FAQ What’s worth understanding about mlcommons first? MLCommons is a nonprofit engineering consortium that builds and governs shared ML benchmarks, most notably the MLPerf Inference suite. In practice it is a fair-comparison instrument: it pins the model, dataset, accuracy target, and load pattern so submissions can be compared neutrally. It is not a deployment estimator, because the normalisation that makes comparison fair also strips out everything specific to your product’s economics. What does MLPerf Inference actually measure, and what do the Server and Offline scenarios mean? MLPerf Inference measures latency-bound throughput at a fixed accuracy target. Offline assumes all queries are available at once and reports raw throughput with no per-query latency ceiling — modelling batch workloads. Server models an online service with queries arriving as a Poisson process, reporting the highest query rate sustained while meeting a specified tail-latency percentile constraint set by the benchmark, not by your SLO. How do you read an MLPerf Inference result without over-reading it for cost decisions? Read it as three separate facts: the accuracy floor the throughput was achieved at, the scenario (Offline and Server numbers are not interchangeable), and the hardware-plus-software executor the result belongs to. Held separately these facts tell you which configs are worth measuring further. Collapsed into a single “config X won, buy config X” verdict, they mislead. Where does MLPerf stop short of a cost-per-request, margin-based comparison? MLPerf normalises the workload and latency constraint for fairness, which is exactly what removes cost-per-request from the frame. It does not carry your batching and concurrency, your p95 SLO, your pricing and rental economics, or your model’s token distribution. It gives you latency-bound throughput per config and leaves the gross-margin delta — the number the business decides on — uncomputed. Which MLPerf figures transfer to your cost-per-request model and which must be re-measured on your serving path? The relative ordering of executors at a fixed accuracy mostly transfers as a fair-comparison signal. Server- and Offline-scenario throughput, accuracy at the reference target, and cost per request do not transfer directly — they must be re-measured at your precision, your SLO, your arrival shape, and your prices. Cost per request is never present in an MLPerf result and must be derived in full. How does the MLPerf latency constraint relate to your product’s p95 latency SLO? Both are tail-latency bounds gating a throughput number, but they are rarely the same value and the relationship is non-linear. Because tail latency degrades sharply near saturation, a tighter percentile bound can cost a large fraction of sustainable throughput. If the benchmark’s bound is looser than your p95, its throughput overstates what you can sustain; the only way to know the magnitude is to re-run the load against your own constraint. How do you use MLCommons measurement discipline as the baseline for a defensible config-selection decision? Borrow the discipline and re-derive the decision. Keep MLPerf’s rigour — pinned accuracy targets, a common load generator, declared scenarios, reproducible runs — but swap in your own constants. Shortlist candidates from MLPerf’s ordering, restate the constraint as your SLO and the workload as your traffic, measure sustained throughput per config on the deployed path, then apply price to get a cost-per-request comparison that justifies the choice on margin. The samples-per-second headline is the easiest part of the decision to trust and the easiest to over-trust. The number that survives contact with production is the one you re-derive on your own serving path, at your own p95, with your own prices attached — and that number is the one a config selection should stand or fall on.
