Run a load test, report peak tokens-per-second, rank the configs by the biggest number, done. That is what most teams mean when they say they ran an AI performance benchmark — and it is exactly why the benchmark rarely changes a serving decision. The number gets produced, someone screenshots it, and then the deployment ships on whatever config was already in place. The measurement never reached the P&L, because it was never measured in a unit the P&L understands. The phrase “AI performance benchmark” sounds like a single, well-defined thing. It is not. “Performance” resolves to at least four different quantities — speed, throughput, latency, and cost — and which one you put on the y-axis decides whether the benchmark decides anything. Pick throughput and you get a headline. Pick cost-at-a-fixed-latency and you get a margin decision. Those are not two views of the same result. They frequently rank the configurations in opposite orders. What does “performance” actually mean for an AI benchmark? Start by separating the four quantities people collapse into one word. Speed — how fast a single request completes with nothing else in flight. Time-to-first-token, tokens-per-second for one stream. Useful for a demo, misleading for a fleet. Throughput — how many tokens or requests the system moves per second at some batch size and concurrency. This is the number that grows when you turn the batch size up. Latency — the distribution of response times under load. p50 is the comfortable case; p95 and p99 are what your users actually feel and what your SLO is written against. Cost — dollars per request or per thousand tokens, given a rented or owned accelerator running at a known utilisation. The trap is that throughput and latency move against each other. Larger batches raise throughput and, past a point, wreck the tail latency. So a benchmark that maximises throughput is quietly choosing to let p95 drift — and a benchmark that never states its latency budget is not comparing configs on a level field at all. It is comparing them at whatever latency each config happened to produce, which is no comparison. The reframe that makes the number decide something: treat performance as cost-at-constraint. Fix the latency budget the product actually requires, then measure cost-per-request and cost-per-token at that fixed p95. Everything else is a diagnostic on the way to that figure. Why peak throughput is a misleading production metric Peak throughput is real, it is just answering a question nobody in production is asking. Peak assumes you can run the batch size that maximises accelerator occupancy. In a serving path with a p95 latency SLO, you usually cannot — the batch size that maxes throughput is the same batch size that pushes your tail past the budget. Once concurrency and latency limits force the config down to a viable batch size, the throughput advantage that won the isolated benchmark can evaporate. We see this pattern regularly when a team hands over a “winning” config: it won because it was measured at a batch size the SLO will never permit. Rank by peak and you promote the config that degrades most under the constraint you forgot to apply. This is the same disconnect the MLPerf discussion runs into — a standard suite reports throughput and latency-bounded results, but the leap from those to your cost-per-request is left to you, which is where MLCommons and MLPerf Inference stop short of cost-per-request becomes the operative gap. The mechanism is not exotic. Batching amortises the fixed cost of a forward pass across more sequences, so tokens-per-second climbs with batch size — until the queueing delay and the longer decode step per batch start inflating the tail. There is a batch size where throughput is still rising but p95 has already crossed your budget. A peak-throughput benchmark reports the point to the right of that line. A cost-at-latency benchmark reports the point on the line. Those are different configs. How do you fix p95 latency so the comparison is fair? Fixing the latency budget first is the single change that turns a benchmark into a decision. The procedure is boring, which is the point: State the SLO before you run anything. Get the product’s real requirement — e.g. “p95 end-to-end response under 2 seconds for a 400-token completion.” Write it down. It is now a constraint, not an output. Sweep batch size and concurrency per config, recording throughput and the full latency distribution at each point, not just the mean. Discard every operating point that violates the p95 SLO. What remains is the feasible region for that config. Within the feasible region, find the maximum sustained throughput — the highest load the config carries while staying inside the budget. Convert that to cost. Divide the accelerator’s hourly cost by the requests-per-hour it sustains at that feasible point. That is cost-per-request, and it is comparable across configs because every config was held to the same p95. The precision of steps 2 and 5 depends on measurement you probably already have from profiling. Per-config utilisation, memory-bandwidth saturation, and the latency breakdown between prefill and decode are what let you trust that a config is genuinely SLO-feasible rather than borderline. That is where GPU profiling supplies the raw inputs — the same per-config utilisation and latency traces described in our work on GPU profiling for production serving paths are what turn a raw performance number into a defensible cost-at-latency comparison. What a cost-per-request benchmark table looks like Here is the structure. The numbers are illustrative — plug in your own measurements — but the columns are the ones that make the result a decision rather than a headline. Config Peak throughput (tok/s) p95 @ peak Feasible batch (p95 < 2s) Sustained req/hr @ SLO Cost-per-request A — large batch, FP16 4,200 3.9 s 8 2,100 $0.0071 B — moderate batch, FP16 3,100 1.8 s 16 2,600 $0.0057 C — moderate batch, FP8 3,400 1.6 s 20 3,200 $0.0047 (Illustrative worked example; assumes one accelerator at a fixed hourly rate, 400-token completions, 2 s p95 budget.) Config A wins the peak-throughput column and loses the decision. It only hits 4,200 tok/s at a batch size whose p95 is nearly 4 seconds — outside the SLO — so its feasible operating point is far lower and its cost-per-request is the worst of the three. Config C, which never topped the throughput chart, has headroom under the latency budget and the lowest cost-per-request. If you had ranked by the first column you would have shipped the most expensive option. The precision choice that gets Config C there is its own subject; our note on how machine learning compilers cut cost-per-request covers the kernel-level side of the same trade. The last column is the one you can hand to a finance partner without translation. Cost-per-request is already the unit gross margin is expressed in, so a benchmark that ends there reads as a margin decision: a defensible per-request cost, with a before and after. That is the framing our [inference cost-cut sprint](Inference Cost-Cut Pack) applies when it benchmarks a buyer’s deployed serving path — the whole point is to produce that cost-per-request delta on the config you actually run, which for SaaS teams is the number that moves the AI-infrastructure SaaS unit economics. How does this relate to MLPerf and other standard suites? It complements them; it does not replace them. MLPerf Inference is genuinely useful — it gives you an audited, reproducible number under declared conditions, including latency-bounded scenarios, and it is the right reference for “is this hardware roughly where the vendor claims.” What it cannot do is price your request. Its latency targets, batch policies, and model versions are its own; your SLO, your prompt distribution, and your accelerator rental rate are yours. The standard suite answers a portability-and-sanity question. The cost-at-latency benchmark answers a margin question. Run the standard suite to trust the platform; run your own cost-at-latency measurement to decide the config. Two claims worth extracting and keeping: A production AI performance benchmark is only valid if the p95 latency budget is fixed before configs are compared — otherwise you are comparing configs at different latencies, which is not a comparison (observed pattern across serving engagements; not a published benchmark). Cost-per-request at a fixed p95 latency is the only performance metric expressed in the same unit as gross margin, which is why it survives the handoff from the platform team to the business while a tokens-per-second figure does not. How do you translate the benchmark number into a decision? The translation is mechanical once the benchmark was built correctly. You already have, per config, a cost-per-request measured inside the SLO. The decision is: pick the config with the lowest cost-per-request among those that meet the latency budget and any other hard constraints (context length, quality floor, memory headroom for the model). If two configs tie on cost, break the tie on latency margin — the one with more headroom under p95 is the safer production choice. What makes this defensible rather than a preference is that every step is a constraint someone signed off on: the SLO came from the product, the quality floor from the eval, the cost from the accelerator’s actual rate. The benchmark is not asserting a winner in the abstract; it is reporting which config is cheapest given the constraints you declared. That is a decision a reviewer can audit, which is the difference between a number that decides and a number that gets screenshotted. For the model-selection side of the same problem — comparing candidates rather than serving configs — see how to compare AI models on a level field. FAQ How should you think about an AI performance benchmark in practice? An AI performance benchmark runs a workload against one or more serving configurations and records how the system behaves under load. In practice it only means something if it fixes the constraint you care about first — usually a p95 latency budget — and then reports cost, because an unconstrained benchmark compares configs at whatever latency each happened to produce, which decides nothing. What does “performance” actually mean for an AI benchmark — speed, throughput, latency, or cost? It resolves to all four, and which one you measure changes the ranking. Speed is single-request completion time; throughput is tokens or requests moved per second at some batch size; latency is the response-time distribution under load; cost is dollars per request. The metric that survives the handoff to the business is cost-per-request at a fixed p95 latency, because that is the unit margin is already expressed in. Why is peak throughput a misleading performance metric for a production serving decision? Peak throughput is measured at the batch size that maximises accelerator occupancy, but that batch size usually pushes tail latency past a production SLO. Once concurrency and latency limits force the config down to a viable batch size, the throughput advantage that won the isolated benchmark can evaporate — so ranking by peak promotes the config that degrades most under the constraint you forgot to apply. How do you fix p95 latency so an AI performance benchmark is a fair comparison across configs? State the SLO before running anything, sweep batch size and concurrency while recording the full latency distribution, discard every operating point that violates the p95 budget, find the maximum sustained throughput inside the feasible region, and convert that to cost-per-request. Every config is then held to the same latency, so the comparison is level. What does a performance benchmark table that ranks configs by cost-per-request look like? Its columns include peak throughput, p95 at peak, the feasible batch size under the SLO, sustained requests-per-hour at the budget, and cost-per-request. The cost-per-request column is the decision column; a config can top the peak-throughput column yet have the worst cost-per-request because its peak sits outside the latency budget. How does a cost-at-latency performance benchmark relate to standard suites like MLPerf Inference? It complements them. MLPerf Inference gives an audited, reproducible number under its own declared conditions and is the right reference for trusting a platform’s hardware. It cannot price your request, because your SLO, prompt distribution, and accelerator rate are yours — so you run the standard suite to trust the platform and your own cost-at-latency measurement to decide the config. How do you translate a benchmark’s performance number into a defensible deployment decision? Pick the config with the lowest cost-per-request among those that meet the latency budget and any hard constraints such as context length or quality floor, breaking ties on latency margin. It is defensible because every step is a constraint someone signed off on — the SLO from the product, the quality floor from the eval, the cost from the accelerator’s actual rate — so a reviewer can audit the choice rather than take a preference on faith. The open question in most benchmarks is not which number is biggest — it is which constraint the number was measured against. Fix the latency budget first, express the result as cost-per-request, and the benchmark stops being a headline and starts being the workload-selection decision it was supposed to inform.
