A vendor hands you a benchmark: 4,200 tokens per second, 92% on a public accuracy suite. The number is real. It was measured. And it almost certainly will not be the number you see once the model meets your production load. That gap — between a benchmark figure and what actually happens under your own conditions — is the single most common reason a procurement committee approves a model that then underperforms in production. The benchmark isn’t wrong. It’s answering a different question than the one you’re asking. What does working with an inference benchmark involve in practice? An inference benchmark measures how a model behaves on a fixed, standardised setup. Someone chooses a hardware target — say, a single H100 or an 8-GPU node — pins a batch size, fixes a sequence length, and runs a defined prompt set through the model. The result is a throughput figure (tokens per second, sometimes requests per second) and, separately, an accuracy score against a known dataset. The value of that discipline is real: it makes different models comparable under identical rules. When you read that one model scores higher than another on the same suite, on the same hardware, at the same precision, you’re reading a genuine signal. That’s why MLPerf-style suites and public leaderboards exist, and it’s worth understanding what MLPerf and hardware inference benchmarks honestly tell you before deployment. The problem is not the measurement. The problem is what people do with it. A benchmark is a controlled experiment. Treating its output as a production forecast is the mistake — and it’s an easy mistake to make, because the number looks authoritative and precise. What conditions does a published inference benchmark fix that differ from my production setup? Every published number carries a set of frozen assumptions. Those assumptions are the fine print, and they rarely match a live system. The usual divergences: Prompt distribution. Benchmarks use a curated prompt set with a known length profile. Your traffic has its own distribution — longer context windows, retrieval-augmented prompts, structured system messages — and prompt shape drives both compute cost and output quality. Batch size and concurrency. A throughput headline is often measured at a batch size chosen to maximise the number, not at the concurrency your service actually runs. Real request arrival is bursty; benchmark batching is orderly. Sequence length. A benchmark might fix output at 128 or 256 tokens. If your workload generates 800-token structured responses, the per-request cost and latency change materially. Precision. A published figure is measured at a specific numerical precision. If the candidate you’re evaluating is quantised and the benchmark was run at a higher precision, you’re comparing two different models — a distinction we unpack in what extreme quantisation means for your procurement evaluation. Latency budget and failure tolerance. Benchmarks report averages or peaks. Your service lives or dies on a tail — p95 or p99 latency — and on what happens when the queue backs up. None of these are edge cases. They are the ordinary conditions of a production deployment, and each one moves the number. Why can’t a benchmark’s throughput or accuracy number decide whether a model fits my workload? Because the benchmark measured a setup that is not yours. The published throughput was achieved at a batch size and sequence length picked to look good. The accuracy score was computed on a prompt set that is not your prompt set. When those change — and they always change — both figures move. Here’s the reframe we apply in every procurement engagement: a public inference benchmark is a starting hypothesis, not a decision. It tells you a candidate is worth testing. It does not tell you the candidate will hold up. The only figure a committee can defend is one measured under the buyer’s own load. This is the same discipline that applies to reading a leaderboard rank — a topic we cover in when leaderboard rank doesn’t predict task accuracy for LLM classification. This isn’t skepticism for its own sake. A benchmark is useful signal, and dismissing it wastes information. The failure is binary thinking: either “the leaderboard says so, therefore it’s true” or “benchmarks are meaningless.” The correct posture sits between them — use the benchmark to shortlist, then re-measure to decide. How far can throughput, latency and accuracy move between a benchmark condition and real load? Enough that a decision made on the published number is unsafe. We don’t quote a universal shift figure because there isn’t one — the movement depends entirely on how far your conditions diverge from the benchmark’s. But the directions are predictable, and naming them is what lets a committee plan a re-measurement. What moves, and why (a diagnostic view) Metric Benchmark condition What shifts under real load Why it moves Throughput (tokens/sec) Optimal batch, fixed short output Typically drops under bursty, mixed-length traffic Batching efficiency falls when request shapes vary Throughput (requests/sec) Steady arrival Drops when concurrency exceeds the tested point Queueing and memory pressure at higher concurrency Latency (average) Reported as mean or peak The tail (p95/p99) is what your SLA cares about Averages hide the queue behaviour that breaks SLAs Accuracy Curated public prompt set Moves — up or down — on your prompt distribution Task and domain shift versus the benchmark’s dataset Cost-per-decision Rarely reported Emerges only once throughput at your load is known Cost = compute time ÷ useful decisions at real load The table is a planning tool, not a prediction. It says: these are the axes on which the public number will move, so these are the axes you must measure yourself. Notice that cost-per-decision doesn’t even appear on most benchmarks — and it’s the figure a committee most needs to compare candidates like-for-like. How should a benchmark claim be re-validated inside a procurement-grade evidence pack? Treat the vendor’s benchmark as a claim to be tested, not a fact to be filed. Inside an evidence pack, a re-validation reports the same model on the buyer’s own workload: Reconstruct the load. Replay a representative sample of real prompts — the actual length and structure distribution, not a synthetic set — at the concurrency the service will run. Measure throughput at that load. Report tokens/sec and requests/sec at the real concurrency, not the batch size that flatters the model. Measure latency at the target percentile. p95 and p99, against the SLA. Averages are not decision-grade for a latency-bound service. Measure the accuracy delta. Run the candidate on a labelled slice of the buyer’s own task and record how far accuracy sits from the published benchmark condition. Compute cost-per-decision at that load. Compute cost divided by useful decisions, so the committee sees the economic figure the leaderboard never published. The output is a set of measured figures alongside the public number, so a reviewer can see exactly how far the headline moved. That’s the procurement-evidence lens: a benchmark is a claim that must be re-validated against the buyer’s workload before it enters the record. It sits inside the broader discipline of AI governance and trust, where every external claim is treated as something to verify rather than accept. How do I turn benchmark figures into a cost-per-decision a committee can compare like-for-like? Cost-per-decision is the figure that makes candidates comparable, and almost no public benchmark reports it. To build it, you need the throughput measured at your concurrency, the hardware cost of that throughput, and a count of the useful decisions produced. A model that is faster in tokens per second but requires more retries, longer prompts, or a larger accelerator can easily lose on cost-per-decision to a slower-looking candidate. This is why the vertical procurement-eval methodology matters: inference throughput, latency percentiles, and cost-per-decision are only meaningful when measured against the buyer’s actual infrastructure. That measurement work — the part where the numbers become decision-grade — is a distinct discipline from reading the benchmark itself, and it’s where a like-for-like comparison is actually assembled. When is a published benchmark a good enough proxy, and when must I re-measure on my own workload? A published benchmark is a reasonable proxy when your conditions closely resemble the benchmark’s: similar hardware, similar prompt shapes, generous latency budget, and a task that overlaps heavily with the benchmark dataset. In that narrow case the number is likely to hold, and re-measurement mostly confirms it. You must re-measure when any of the divergences are large: your prompts are much longer or more structured, your concurrency is high, your latency budget is tight, your task is domain-specific, or the candidate is quantised relative to the benchmark condition. In those cases the published number is a hypothesis and nothing more. FAQ What should you know about an inference benchmark in practice? An inference benchmark runs a defined prompt set through a model on a fixed hardware target at a pinned batch size, sequence length, and precision, then reports throughput and an accuracy score. It makes models comparable under identical rules, which is genuine signal. In practice it means a controlled experiment — not a forecast of how the model will behave on your traffic. What conditions does a published inference benchmark fix that differ from my production setup? It fixes the prompt distribution, batch size, concurrency, sequence length, precision, and latency assumptions. Production traffic differs on every one of these: your prompts have their own length profile, your concurrency is bursty, and your latency budget lives on the tail. Each frozen assumption is a place where the published number diverges from what you’ll see. Why can’t a benchmark’s throughput or accuracy number decide whether a model fits my workload? Because the number was measured on a setup that isn’t yours, and both throughput and accuracy move when the conditions change. The benchmark tells you a candidate is worth testing; it doesn’t tell you the candidate will hold up under your load. Treat it as a starting hypothesis, then re-measure to decide. How far can throughput, latency and accuracy move between a benchmark condition and real load? There is no universal shift figure — the movement depends on how far your conditions diverge from the benchmark’s. But the directions are predictable: throughput typically drops under bursty mixed-length traffic, tail latency (p95/p99) is what breaks SLAs, and accuracy moves either way on your prompt distribution. Naming these axes is what lets you plan a re-measurement. How should a benchmark claim be re-validated inside a procurement-grade evidence pack? Reconstruct the real load, measure throughput at your concurrency (tokens/sec and requests/sec), measure latency at the target percentile, record the accuracy delta versus the published condition, and compute cost-per-decision at that load. Report those measured figures alongside the public number so a reviewer can see exactly how far the headline moved. How do I turn benchmark figures into a cost-per-decision a committee can compare like-for-like? Combine throughput measured at your concurrency, the hardware cost of that throughput, and a count of useful decisions produced. A model that looks faster in tokens per second can lose on cost-per-decision if it needs retries, longer prompts, or a larger accelerator. Cost-per-decision is the figure that makes candidates comparable, and public benchmarks almost never report it. When is a published benchmark a good enough proxy, and when must I re-measure on my own workload? It’s a good enough proxy when your conditions closely resemble the benchmark’s — similar hardware, prompt shapes, a generous latency budget, and an overlapping task. Re-measure when any divergence is large: long or structured prompts, high concurrency, a tight latency budget, a domain-specific task, or a quantised candidate. In those cases the published number is a hypothesis, not a decision. Most procurement disputes over model choice trace back to one question that never got asked: whose load produced this number? Answer that before the committee votes, and the leaderboard becomes what it was always meant to be — a place to start, not a place to stop.
