The capacity model was right — on paper
An infrastructure team sizes a GPU cluster from published benchmark throughput. The arithmetic is clean: N accelerators, each delivering X tokens per second, total capacity N×X. They provision, deploy, and find that sustained throughput sits well below the planning number. The cluster is undersized, SLAs are at risk, and the next procurement cycle is months away.
The planning wasn’t careless, and the benchmark number was not fabricated. The error was structural. The model was fed a peak figure — what a device delivers at boost clocks, under clean conditions, briefly — where it needed a steady-state figure. Those are different quantities, and the distance between them is where capacity plans quietly fail.
One clarification before anything else, because it shapes every number below. “Steady state” here names the regime a deployed fleet runs in — the thing capacity is sized against. It is not a condition a measurement can assert. What a benchmark run can honestly declare is narrower: completed iterations inside a continuous timed window, after a discarded warm-up, at a workload size already raised until throughput plateaus. That declaration is useful precisely because it is modest. It does not claim the machine’s thermals or clocks had settled inside the window, and it does not claim to have watched a production fleet.
Why peak throughput is the wrong input to a capacity model
We have written elsewhere about how peak and steady-state performance diverge and the physical mechanisms behind the divergence. For planning purposes the consequence collapses to one line: peak throughput answers a question nobody sizing a fleet is asking.
Peak describes what the hardware can do for a moment — inside the boost window, before clock settling, with no concurrent tenant. Capacity planning describes what the infrastructure must deliver continuously, for hours or weeks, with thermal drift, memory pressure, neighbouring workloads, and everything else that pushes real output below the headline.
The figure worth putting under a cost is the sustained one: completed iterations inside a continuous timed window, after a discarded warm-up, at a batch size raised until throughput plateaus. That sentence is deliberately mechanical. Every clause in it is something a run can state and a reader can check.
Predictability beats maxima
In capacity planning, consistency is worth more than an occasional high-water mark. A planner does not need the best throughput a system ever reached. They need the worst throughput it reliably holds.
A node that averages 1,000 tokens per second but drops to 700 during thermal settling or garbage-collection pauses has an effective planning capacity of 700, not 1,000. Size to the average and a meaningful share of requests will miss their latency target during the dips — and those dips are not anomalies, they are part of the operating envelope.
This is why the useful figure is often a low percentile: the throughput sustained at least 90–95% of the time. It absorbs normal variation rather than averaging it away, and it gives a sizing basis that survives contact with an SLA.
A related trap is reading utilization as a monotone good. Very high sustained GPU utilization can mean two opposite things — a healthy, efficiently packed node, or a system that has run out of headroom and is about to hit a wall. The number alone does not distinguish them. In our experience the pairing is what informs: a node pinned near 100% while its throughput floor sags is heading somewhere bad, whereas the same utilization with a stable floor is simply efficient. Plan against the floor; use utilization to explain why the floor moved, not as the sizing input itself.
Cost efficiency is a sustained-throughput property
Cost comparisons built on peak throughput produce confident wrong answers. Device A peaks at 1,200 tokens per second, device B at 1,000, and A costs 20% more. The peak arithmetic says A wins on throughput per dollar.
Now sustain both in a dense node. If A settles at 950 and B at 900 — because B draws less power and throttles less — the 20% premium has bought roughly a 5% sustained advantage, not a 20% one. The ranking has not necessarily flipped, but the decision has changed shape entirely.
Power compounds it. A device drawing 700W to hold 950 tokens per second costs more per token in electricity than one drawing 400W to hold 900, and at fleet scale across a multi-year deployment that gap stops being a rounding error. Note also what a rated TDP is not: it is a thermal design figure, not a measurement of what the box pulled while doing the work. Energy per unit of sustained work has to be measured under the same window as the throughput it divides, or the two numbers do not belong in the same fraction.
Cross-class comparisons — GPU against TPU, say — inherit all of this and add a failure mode of their own. The tempting shortcut is peak-FLOPS-per-dollar, which flattens two different architectures into one transient and predicts almost nothing about production cost. A defensible comparison runs the same workload on each class, at a declared precision and a saturated batch size, inside a declared timed window, and divides each device’s sustained figure by its all-in cost: capital plus power at that measured draw.
Which score goes under the dollar
Two rules keep the arithmetic honest, and both are about which number you divide.
First, use the category score that matches the work being sized, not an aggregate. A fleet is sized for training, or for inference, or for general compute — and a device strong at one is routinely weak at another. Category scores are therefore read separately. The aggregate figure LynxBenchAI reports is an unbounded ordinal aggregate with no reference-device normalisation: it orders devices, it does not measure a physical quantity, and dividing it by a price yields nothing you can defend in a budget meeting.
Second, keep every number inside one release name. A 26Q3 figure and a 27Q1 figure are not points on the same axis; catalogues change, harnesses change, and the comparison silently stops being a comparison. A cost model that mixes releases is a cost model with an undeclared variable in it.
The public leaderboard helps here in a specific, limited way: it gives a device a population to sit in. A planning assumption can be checked against runs other people produced with the same instrument, on their own hardware, rather than against a single vendor slide.
How do you translate a sustained figure into infrastructure sizing?
Throughput alone does not size a fleet. Four further inputs do most of the work:
Demand profile. What is the expected arrival rate and its variance? Peak demand has to be served out of sustained capacity, not out of a boost window.
Headroom. How much spare capacity do bursts, failover, and maintenance need? Headroom calculated against a peak figure yields actual headroom smaller than intended — the error stacks on top of the sizing error.
Workload trajectory. If model size, sequence length, or request volume is projected to grow, size against the trajectory. As we discuss in the context of choosing between throughput and latency optimization targets, the metric that constrains you today may not be the one that constrains you next year.
Scaling efficiency. Multi-GPU and multi-node scaling is never linear. Communication overhead through NCCL, load imbalance, and scheduler behaviour all reduce aggregate output relative to single-device numbers, so scaling efficiency belongs in the model as a measured discount rather than an assumption.
Capacity planning inputs that peak benchmarks miss
| Planning input | What it requires | Why peak numbers fail | Evidence class |
|---|---|---|---|
| Sustained throughput | Completed iterations in a continuous timed window, after a discarded warm-up, at a saturated batch size | Peak captures a boost-clock transient, not the regime being sized | benchmark (declared window, single release) |
| Throughput floor | The low-percentile throughput held across the window, not the mean | Averages absorb dips from GC, thermal drift, and memory pressure | benchmark |
| Scaling efficiency | Aggregate multi-device output compared against linear extrapolation | Interconnect and scheduling overhead are invisible in single-device figures | observed pattern across our engagements; not a published rate |
| Energy per unit of work | Measured draw inside the same window as the throughput it divides | Rated TDP is a thermal design figure, not a measurement of work done | benchmark (paired measurement required) |
| Category fit | The category score matching training, inference, or compute | An unbounded ordinal aggregate divided by a price is not a cost figure | methodology constraint |
The budgeting conversation
Procurement eventually needs a defensible sentence: “we need N devices at $X each.” The pressure runs toward the flattering input, because the peak figure produces a lower device count, a smaller budget ask, and an easier slide.
The alternative is to present the model on sustained figures with explicit uncertainty ranges and a named release. It takes more nuance and it asks the room to sit with a range instead of a point. It also produces the outcome everyone actually wanted: a cluster that meets its SLA, no emergency procurement, and a planning team whose numbers survived production.
Sustained-throughput thinking is an operations discipline as much as a planning one — what the SRE book teaches about running production AI is the applied counterpart to sizing against the floor rather than the peak.
A sustained-throughput-per-dollar comparison needs a sustained-throughput numerator, and the released 26Q3 LynxBenchAI benchmark produces one on hardware a planner already has: pip install lynxbench-ai, 15–30 minutes, NVIDIA, AMD, Intel, or CPU under one methodology, reported per precision with a declared window. What it does not produce is a throughput figure for your workload — it covers a fixed model catalogue at fixed precisions, and no score substitutes for measuring the thing you actually run.
So the open question for anyone building a cost model this quarter is narrower than it looks: which category score is your fleet actually being sized against, and is every number in the model drawn from the same release name?
