A MLPerf Inference headline number — queries per second, latency at a target percentile, a peak-throughput figure — is not a verdict on which accelerator or serving stack you should buy. It is a comparability signal, produced under fixed conditions that may share very little with your deployment. The gap between those two readings is where most hardware-selection mistakes are made. That distinction matters because MLPerf Inference is one of the most disciplined benchmark suites in the industry, and precisely because it is disciplined, it encodes assumptions that are easy to miss. The suite fixes the model, fixes the accuracy target, fixes the scenario, and then measures. Every one of those fixed choices is a variable in your own deployment. When a committee lifts a submitted queries/sec figure straight into a buying decision, it is quietly assuming its own model, its own input distribution, its own accuracy tolerance, and its own concurrency shape all match the submission. They rarely do. What MLPerf Inference scenarios actually measure MLPerf Inference defines four scenarios, and each one models a different way requests arrive at a system. Reading the numbers honestly starts with knowing which scenario a headline came from, because the same hardware produces very different figures across them. Offline measures maximum throughput when all inputs are available at once and latency does not matter — a batch-processing shape. This is the number that looks biggest and travels furthest in marketing. Server measures throughput while holding a latency bound at a target percentile, modelling requests that arrive as a Poisson stream. This is the closest analogue to an online API serving concurrent users. Single-stream measures latency for one query at a time — a shape that matches a single interactive session with no batching headroom. Multi-stream measures how many concurrent streams a system can sustain under a per-stream latency bound, modelling something closer to fixed-count sensor or camera inputs. The mistake is treating any single scenario as the performance of the hardware. A given accelerator can top the Offline throughput chart and still be a poor fit for a latency-sensitive Server workload, because the batching that maximizes Offline throughput is exactly what inflates tail latency under Server conditions. Standardized scenarios are what make cross-submission comparison possible in the first place — the same discipline that makes benchmarks comparable at all on LynxBench AI’s methodology account. But comparability across submissions is not the same as portability to your production concurrency shape. Your job is to identify which scenario your workload resembles, then read only the figures produced under that scenario. Which parts of a submitted config you have to check first Every MLPerf submission ships with a config that pins down the conditions the number was produced under. A headline figure detached from its config is close to meaningless. Before a number earns any weight in a procurement discussion, four fields need reading. Config field What it changes Why the headline misleads without it Numerical precision INT8, FP8, FP16, or mixed — precision changes both throughput and accuracy An INT8 throughput number is not comparable to an FP16 one; the fast figure may run a precision you can’t accept Batch size Larger batches raise throughput and raise tail latency together An Offline-friendly batch may violate your p95 latency bound entirely Accuracy target MLPerf allows a “high accuracy” and a relaxed variant per model The faster submission may be the relaxed-accuracy one, invisible in a raw QPS chart Model version Reference model and version are fixed per round The reference model is almost never the fine-tuned model you will actually deploy Reading a MLPerf headline without opening the config is the same failure LynxBench AI names when it explains why spec-sheet reading fails for AI hardware: a single number, stripped of the conditions that produced it, gets treated as a promise. The submission is honest. The reading is not. Precision deserves particular care. In the configurations we have reviewed for clients, the throughput delta between an FP16 and an INT8 submission on the same silicon is often large enough to reorder a shortlist — and it is entirely a function of a quantisation choice the buyer may or may not be willing to make. That is an observed pattern across procurement reviews, not a benchmarked constant. The point stands regardless of the exact multiple: precision is a first-class variable, not a footnote. How to convert standardized numbers into cost-per-decision The reason a procurement committee runs benchmarks at all is to answer two operational questions: will this meet latency under my real load, and what does each decision cost. MLPerf gives you inputs to both, but not the answers directly. You have to do the conversion, and the conversion is where the calibration to your conditions happens. Start with the Server-scenario figure that matches your concurrency shape, at the precision and accuracy target you can actually accept. That gives you a sustained queries/sec under a latency bound. From there: Cost-per-decision is the amortized cost of the accelerator (plus its share of host, power, and rack) divided by the sustained throughput at your accuracy target — not the Offline peak. A system that is cheaper per hour but slower at your required precision can be more expensive per decision. p95 latency under real load comes from the Server-scenario percentile bound, but only if your request arrival pattern resembles the Poisson stream MLPerf uses. Bursty, correlated, or long-context traffic will not behave like the submission, and the tail will be worse than the headline. This is the same discipline that separates a leaderboard rank from a deployment prediction, which we cover in more depth in why an inference benchmark is not your workload evaluation. The benchmark bounds the plausible envelope; your calibration turns it into a number you can defend to a finance reviewer. Where these numbers belong in an evidence pack — and where they mislead A procurement-grade evaluation pack has distinct sections, and MLPerf and hardware benchmark results belong to exactly one region of it: the operational-feasibility and cost-per-decision reasoning. They populate the latency-under-load and cost columns. They do not populate the task-accuracy section, and they cannot populate the failure-mode section. This is the boundary most often crossed. A MLPerf submission runs a reference model to a reference accuracy target on a reference dataset. It says nothing about whether your fine-tuned model, on your input distribution, hits the accuracy your use case needs, and it says nothing about how the system behaves on the inputs that break it. Those questions are answered by task-specific evaluation and failure-mode analysis — a different part of the pack entirely, closer to what we describe in what benchmark suites prove and where they fall short for procurement. A hardware benchmark that is asked to certify task accuracy is being asked a question it was never built to answer. Used inside their proper section, MLPerf and hardware suites do real work: they bound the plausible performance envelope and they expose vendor tuning. When a submission uses an aggressive batch size or a relaxed accuracy variant to top a chart, the config makes that visible to anyone who reads it. That transparency is a feature — it is what lets you discount a headline whose scenario or precision does not match your deployment. How to tabulate hardware options for a like-for-like committee comparison The single most useful thing you can do for a decision committee is normalize every candidate to the same conditions before anyone compares them. A raw table of vendor headline figures compares numbers that were produced under different scenarios, precisions, and accuracy targets — which is to say it compares nothing. Here is the shape of a defensible comparison. Column Rule for filling it Scenario Fixed across all rows to the one matching your workload (usually Server) Precision Fixed to the precision you will actually run in production Accuracy target Fixed to your required target, using the matching MLPerf accuracy variant Sustained QPS Read from the matching scenario/precision submission, not the Offline peak p95 latency The percentile bound under the fixed scenario, flagged if your traffic isn’t Poisson Cost-per-decision Amortized system cost ÷ sustained QPS at your accuracy target Config gaps Every place the submission diverges from your deployment, named explicitly The last column is the one that keeps the table honest. Any cell where a candidate has no submission matching your fixed conditions is a gap the committee must fill with its own measurement, not a number to be borrowed from a nearby scenario. A comparison built this way lets a committee weigh accelerator and serving options on genuinely like-for-like terms and discount headline numbers whose conditions don’t hold. It also sits naturally inside the broader AI governance and trust approach, where every number in the pack is traceable to the conditions that produced it. FAQ How do you read MLPerf and hardware inference benchmarks honestly for a real deployment decision? Read them as a comparability signal produced under fixed conditions, not as a promise about your production. Identify which scenario matches your workload, check the precision and accuracy target in the config, and convert the standardized figures into cost-per-decision and p95 latency under your own concurrency shape before letting any number influence a buying decision. What do MLPerf Inference scenarios (Offline, Server, Single-stream, Multi-stream) actually measure, and how do they map to production concurrency shapes? Offline measures maximum throughput with no latency constraint (batch processing); Server measures throughput under a latency percentile bound with Poisson arrivals (online concurrent serving); Single-stream measures one-query-at-a-time latency (a single interactive session); Multi-stream measures sustained concurrent streams under a per-stream bound (fixed-count sensor or camera inputs). Match your traffic to a scenario and read only that scenario’s figures. Which parts of a submitted MLPerf config must you check before trusting a headline number? Precision (INT8 vs FP8 vs FP16), batch size, accuracy target (high-accuracy vs relaxed variant), and model version. Each changes the headline substantially, and a fast figure often reflects an aggressive precision, a large batch, or a relaxed accuracy target you may not be able to accept in production. Where do MLPerf and hardware benchmark results belong in a procurement-grade evaluation pack, and where do they mislead? They belong in the operational-feasibility and cost-per-decision sections — the latency-under-load and cost columns. They mislead when carried into the task-accuracy or failure-mode sections, because a reference-model submission says nothing about whether your fine-tuned model on your inputs hits your required accuracy or how it behaves on breaking inputs. How do you convert standardized queries/sec and percentile latency into cost-per-decision and p95 latency under your actual load? Take the Server-scenario throughput at the precision and accuracy target you can accept, then divide the amortized system cost (accelerator plus its share of host, power, and rack) by that sustained throughput to get cost-per-decision. For p95 latency, use the scenario’s percentile bound only if your arrival pattern resembles Poisson traffic; bursty or long-context load will run a worse tail than the headline. Why can’t a MLPerf or vendor hardware benchmark answer task-specific accuracy or failure-mode questions on its own? Because it runs a fixed reference model to a fixed reference accuracy target on a fixed reference dataset. None of those are your fine-tuned model, your accuracy requirement, or your input distribution, so it cannot certify task accuracy — and it exercises no adversarial or edge-case inputs, so it cannot characterize failure modes. Those questions require separate task-specific and failure-mode evaluation. How should hardware and serving options be tabulated so a committee can compare them on a like-for-like basis? Fix scenario, precision, and accuracy target to your production conditions across every row, then read sustained QPS, p95 latency, and cost-per-decision from the matching submissions rather than from Offline peaks. Add a config-gaps column naming every place a candidate diverges from your deployment, so the committee fills those gaps with its own measurement instead of borrowing a mismatched number. The question worth carrying into the room The useful discipline is not distrust of MLPerf — it is a well-run suite whose numbers mean exactly what its scenarios encode. The discipline is asking, for every headline that reaches a decision: under what scenario, at what precision, against what accuracy target, and how far is that from my deployment? When the config answers those questions and the gap is small, the number earns its weight. When the gap is large, the number becomes a bound on the envelope, not a verdict — which is precisely the reading that keeps a hardware benchmark inside the operational-feasibility section of the pack, where it belongs, and out of the accuracy claims it was never built to support.
