A device posts a leading tokens-per-second number in MLPerf Client, and someone reads it as a verdict: this machine “runs AI.” That single figure hides the question that actually decides whether your workload is viable on the hardware in front of you. MLPerf Client measures generative-model inference on consumer-class hardware — laptops, desktops, and their integrated NPUs and GPUs — under a controlled set of conditions. It is a comparison of inference paths, not a promise about the specific agentic or generative system you plan to ship.

The confusion is understandable. The benchmark reports numbers that look like a spec sheet, and spec sheets invite a single-glance reading. But the same device scores differently depending on whether it is asked to emit a first token quickly or to sustain a long stream of them. Reading MLPerf Client well means separating those two behaviours and matching each to your actual call pattern.

What does MLPerf Client actually measure?

MLPerf Client is the consumer-hardware branch of the MLPerf family maintained by MLCommons. Where MLPerf Inference targets datacenter and edge accelerators, MLPerf Client runs large-language-model inference workloads on the kind of silicon that ships in a laptop or workstation: x86 and Arm CPUs, discrete and integrated GPUs, and dedicated NPUs. It exercises a defined model — recent versions have used quantised builds of models such as Llama-family and Phi-family checkpoints — through a runtime chosen for the hardware path being tested, and it records how fast tokens come out.

The workload is fixed on purpose. A benchmark only means something if the thing being measured is the same across every device. MLPerf Client pins the model, the quantisation format, and the prompt structure so that a score reflects the hardware-and-software path rather than a lucky choice of smaller model. That control is exactly what makes it a fair comparison — and exactly why it cannot answer the question “will my app feel fast,” because your app is not the pinned workload.

Two measurements carry most of the signal:

• Time-to-first-token (TTFT) — how long from submitting the prompt until the first output token appears. This captures prompt processing (the prefill phase), where the model reads and encodes the input before it can generate anything.
• Sustained tokens per second — how fast the model emits output once generation is under way (the decode phase). This is the number people quote as the headline.

These two numbers describe different phases of the same inference, and a device can be strong at one and weak at the other. That divergence is the whole point of reading the benchmark carefully.

Why time-to-first-token and sustained throughput are not the same number

Prefill and decode stress hardware differently. Prefill is a largely compute-bound, highly parallel pass over the whole prompt — the model processes every input token at once, which rewards raw matrix-multiply throughput. Decode is autoregressive: each output token depends on the previous one, so the model generates them one at a time, and the bottleneck shifts toward memory bandwidth and the cost of moving the key-value cache. A GPU with abundant compute can post a fast prefill and therefore a low TTFT; a device with high memory bandwidth but modest compute can lead on sustained decode while lagging on first token. There is no law that says the two rank the same across devices.

This is where the naive single-number reading breaks. If you compare two laptops on sustained tokens per second alone, you might pick the throughput leader and discover that it takes noticeably longer to start responding. For a chat-style interaction where the user waits for the first words to appear, TTFT is often the perceptually dominant metric — the stream can be slower once it starts and still feel responsive, because humans read at a bounded rate. The distinction between these phases is the same one that governs latency budgeting for hosted inference; we walk through how prompt length drives both in our guide to estimating tokens for latency and cost budgets.

The evidence class here matters. When we say prefill is compute-bound and decode is bandwidth-bound, that is a well-established structural property of transformer inference (observed pattern, consistent across the inference stacks we work with), not a figure lifted from a specific MLPerf Client run. The exact TTFT and throughput numbers you should trust are the ones in the published results table for the device and software version you are evaluating.

How MLPerf Client works in practice, and which hardware paths it covers

Running MLPerf Client means installing the harness, pointing it at a hardware execution path, and letting it drive the pinned workload while it records timings. The harness abstracts over several backends so that the same benchmark logic can target different silicon:

• CPU — a portable baseline that runs almost anywhere, useful as a floor reference.
• GPU — discrete NVIDIA, AMD, and Intel GPUs, and integrated GPUs, typically through vendor runtimes and acceleration libraries.
• NPU — the dedicated neural accelerators now shipping in laptop SoCs, reached through vendor execution providers.

The reason MLPerf Client covers all three is that the interesting comparison on consumer hardware is not just “which GPU is fastest” but “which execution path on this device gives the best inference behaviour for a given model.” An NPU may win on energy while a discrete GPU wins on raw decode speed; a device’s score is only meaningful once you know which path produced it. This is why the software stack is as much a part of the result as the chip — a point that generalises well beyond this one benchmark, and one we cover for constrained targets in model optimization for edge inference. If you are weighing whether a use case even fits on target silicon, the broader feasibility picture is where our generative AI work starts.

How to read MLPerf Client scores when scoping a real project

The move that turns a benchmark number into an engineering decision is mapping the workload’s call pattern onto the phase it stresses. Here is a decision rubric we use when reading these scores for on-device feasibility.

Reading rubric: match the metric to the workload

Workload pattern	Dominant phase	Metric to weight	What a good score buys you
Agentic loop, many short model calls	Prefill per call	Time-to-first-token	Low per-step latency that does not compound across the loop
Long-form generation (drafting, summarisation)	Decode	Sustained tokens/second	Fast completion of long outputs
Interactive chat, short prompts	Prefill, then bounded decode	TTFT primary, throughput secondary	Responsive start; stream keeps up with reading speed
Batch processing of many prompts offline	Decode-heavy, latency-tolerant	Sustained tokens/second	Total wall-clock throughput
RAG with long retrieved context	Prefill-heavy	Time-to-first-token	Tolerable delay before the answer begins

The rubric is extractable on its own, but the judgement behind it is simple: find the phase your workload spends most of its user-visible time in, and weight the corresponding MLPerf Client metric. A benchmark score read against the wrong phase is not wrong — it is just answering a question you did not ask.

Two conditions temper every row. First, quantisation: MLPerf Client’s pinned quantised model may not match the precision you intend to ship, and a device’s relative standing can shift with precision. Second, prompt length: TTFT scales with input length because prefill processes the whole prompt, so a benchmark run on short prompts under-represents a RAG workload that feeds thousands of tokens of context per call.

Why a throughput leader can still be the wrong choice for an agentic loop

Consider an on-device agent that plans a task by making a sequence of short model calls in an orchestration loop — classify the request, pick a tool, format arguments, check a result, decide the next step. Each call is short, so decode time is small; the dominant cost per step is the time to process the prompt and emit the first token. Now suppose you picked a device because it topped the sustained-throughput chart, but it has a mediocre TTFT.

Work through the arithmetic as an illustrative example. If an interaction runs, say, ten sequential model calls, and the chosen device adds roughly a few hundred milliseconds of extra first-token latency per call versus a TTFT-leading device, that gap multiplies by ten and lands as seconds of added user-visible delay for a single interaction — even though the device “won” the benchmark you looked at. The throughput advantage barely helps, because each call generates only a handful of output tokens. This is the compounding effect the ROI framing warns about: in an agentic loop, latency per step is paid on every step.

The same reasoning applies in reverse. A long-form drafting tool that generates a thousand-token document per request lives almost entirely in decode; TTFT is a rounding error, and the sustained-throughput leader is the right pick. The device did not change — the workload did. When these decisions sit on a real deployment, the interplay of latency and throughput is exactly what we unpack in the context of hosted models in reading latency and throughput for real-time GenAI.

FAQ

How does MLPerf Client actually work?

MLPerf Client, maintained by MLCommons, runs a pinned generative-model inference workload on consumer hardware — CPU, GPU, or NPU — and records how fast tokens are produced. In practice it is a controlled comparison of inference paths across devices, so a score tells you how one device’s hardware-and-software stack handles a fixed model, not how your specific application will behave.

What does MLPerf Client actually measure — and what does it not tell you about your workload?

It measures time-to-first-token and sustained tokens per second for a fixed model, quantisation, and prompt structure. It does not tell you how your own model, precision, prompt lengths, or call pattern will perform, because those differ from the pinned benchmark workload — the control that makes the comparison fair is also what limits its portability to your project.

How do time-to-first-token and sustained tokens/second differ, and why does the distinction matter for agentic versus generative use cases?

Time-to-first-token captures the compute-bound prefill phase; sustained tokens per second captures the bandwidth-bound decode phase, and a device can lead on one while lagging on the other. Agentic loops with many short calls are dominated by first-token latency, while long-form generation is dominated by sustained throughput, so the same device is the right or wrong choice depending on which metric your workload actually stresses.

How do you run MLPerf Client, and what hardware paths (CPU, GPU, NPU) does it cover?

You install the harness, select an execution path, and let it drive the pinned workload while recording timings. It covers CPU as a portable baseline, discrete and integrated GPUs through vendor runtimes, and dedicated NPUs through vendor execution providers — so a result is only meaningful once you know which path produced it.

How should you read MLPerf Client scores when scoping on-device inference for a real project?

Identify the phase your workload spends most of its user-visible time in, then weight the corresponding metric: TTFT for interactive and agentic patterns, sustained throughput for long-form or batch generation. Temper every reading with two caveats — the benchmark’s quantisation may not match your shipping precision, and TTFT scales with prompt length, so short-prompt runs under-represent long-context workloads.

Why can a device that leads on throughput still be the wrong choice for an agentic orchestration loop?

In an agentic loop, each short model call pays first-token latency, and that cost compounds across the sequence of calls in a single interaction. A device that tops the sustained-throughput chart but has mediocre TTFT can add seconds of user-visible delay per interaction, while its throughput advantage barely helps because each call generates only a handful of tokens.

MLPerf Client is a good instrument used against the wrong question more often than it is a bad instrument. Before you read a single score, decide which phase your workload lives in — because that decision, not the headline number, is what determines whether the hardware in front of you can carry the system you intend to build.