You picked a GPU, spun up a serving runtime, fired a batch of requests, and read the tokens-per-second the console printed. That number is real. It is also not a purchasing decision, and treating it like one is where most config selections quietly go wrong. A “simple bench” for AI inference is genuinely useful. It answers “does this config run, and roughly how fast” in a few minutes, with almost no setup. What it does not answer — and what it is routinely asked to answer — is “which config should we ship.” The gap between those two questions is not pedantry. It is the difference between a number worth minutes and a number worth a serving contract. What a simple inference benchmark actually measures Strip the tooling away and a simple bench does one thing: it sends some requests to one serving configuration and reports how fast the tokens came back. Whether you run it through a hand-rolled asyncio loop against a vLLM endpoint, a wrk-style load driver in front of an SGLang server, or the built-in throughput script that ships with most runtimes, the shape is the same. One config. One reading. One number. That number is a throughput measurement — usually output tokens per second, sometimes requests per second. It is a true measurement of what happened. The problem is everything the measurement leaves unstated. The latency floor. Was that throughput achieved at a p50 of 40 ms per token or a p95 of 800 ms? A simple bench rarely pins this, so the throughput figure floats free of the latency users would actually experience. The concurrency. Throughput at batch size 1 and throughput at batch size 256 are different physical regimes on the same GPU. If concurrency drifted during the run, the headline number is an average over conditions you did not record. The request mix. Short prompts with long generations stress decode; long prompts with short generations stress prefill. A simple bench measures whatever mix you happened to send, which is almost never your production distribution. None of this makes a simple bench wrong. It makes it unqualified. The reading is accurate for the conditions that applied — you just did not write those conditions down, so you cannot reproduce them or compare against them. Why the tokens-per-second number can’t select a serving config directly Here is the naive move: run the simple bench across three configs, see that config B printed the highest tokens/sec, and ship config B. It feels like the number decided. It didn’t. The reason is that a simple bench measures one config under one set of unstated conditions. It can rank configs against each other — but only under whatever conditions happened to apply during each run, and those conditions were not held constant. If config B ran at higher concurrency than config A because its scheduler backed up differently, its throughput advantage may vanish the moment you pin both to the same p95 latency target. You measured B winning a race whose rules you never fixed. This is the same class of error we see in public leaderboards, where a single aggregate hides the conditions that produced it. In an AI performance benchmark you want the number to actually decide cost, the conditions are the whole point — and a simple bench, by construction, leaves them implicit. A throughput ranking is a real signal. It is just a relative signal under uncontrolled conditions. Promoting it to an absolute selection is the upgrade that breaks: you take an observed pattern from one uncontrolled run and treat it as a benchmark-grade decision input. That substitution is where money leaks. What you must fix before a simple bench result becomes comparable To turn a simple bench into a defensible comparison, you fix the three things it left implicit. The order matters — pin the constraint first, then read the cost. Pin p95 latency. Choose the latency your product actually requires (time-to-first-token and inter-token latency both matter for streaming) and treat it as a hard constraint, not an output. Now every config is measured at the same service level, so throughput differences mean something. Hold the workload constant. Freeze the request mix — prompt-length distribution, generation-length distribution, arrival pattern — to something that reflects production. Replay a captured trace if you have one. The same trace against every config is what makes the comparison like-for-like. Report cost-per-request, not raw throughput. Convert throughput at the pinned latency into cost. Divide the hourly cost of the instance by the sustained requests-per-second it delivers at the p95 target. Now the axis of comparison is money, which is the axis the decision actually turns on. Once those three are fixed, you are no longer running a simple bench. You are running the minimal version of a spec-organised benchmark — and that is exactly the structure that structuring serving configs so cost-per-request comparisons are fair is built to enforce. The value of naming it that way is that the conditions become reviewable: someone else can check that the p95 target, the trace, and the pricing were held constant across configs. Worked example: from tokens/sec to cost-per-request The following is illustrative — the numbers are chosen to show the mechanism, not to represent any specific hardware. Assume two configs on hypothetical instances, benchmarked against the same replayed trace. Config Instance $/hr (assumed) Raw throughput (req/s) Throughput at p95 ≤ 500 ms Cost per request A — throughput-tuned, large batch $12.00 48 22 $0.000152 B — latency-tuned, smaller batch $12.00 31 29 $0.000115 Read the middle column and config A wins by 55%: it is the “fastest,” and a raw simple bench would have shipped it. Read the p95-constrained column and it collapses — A only sustains 22 req/s inside the latency budget because its large batches blow past the p95 target under load, while B holds 29. On cost-per-request at the service level the product actually needs, B is roughly 24% cheaper. That inversion — throughput-optimal and cost-optimal being different configs — is precisely the gross-margin delta a simple bench hides. The raw number rewards the config that looks fast in an unconstrained sprint; the cost-per-request table rewards the config that is cheap under the constraint you have to honour in production. If you want to sanity-check the per-token side of that arithmetic before you build the table, an LLM token calculator for estimating inference cost per request makes the token-to-dollar conversion explicit. When a simple bench is good enough — and when it isn’t A simple bench earns its keep. It is the right tool when the stakes are low and the question is coarse. Good enough: confirming a new runtime version starts and serves. Smoke-testing that a model loads on a given GPU at all. Getting an order-of-magnitude feel before you invest in a controlled harness. Catching a config that is catastrophically slow — a broken tensor-parallel layout, a runtime falling back to a slow kernel path. Not good enough: selecting between two viable configs. Justifying a GPU purchase or an instance-type migration. Any comparison where a 20–30% cost-per-request delta changes the answer — which, at production volume, is most of them. The dividing line is whether the decision can survive the conditions being unstated. If a coarse “runs / doesn’t run, fast / slow” answer is all you need, the simple bench is complete and you should stop there. If a config-selection or purchasing decision hangs on it, the unstated conditions are load-bearing, and you have to make them explicit before the number is safe to act on. Throughput alone is a poor selector; the machine-learning model metrics that actually decide a serving config are latency-constrained cost figures, not headline tokens/sec. It is also worth noting what the throughput number cannot tell you on its own even when conditions are fixed: why one config is faster. That interpretation needs the per-config GPU utilisation and latency breakdown that profiling supplies — a throughput figure and a utilisation trace answer different halves of the same question. FAQ What’s worth understanding about simple bench ai first? A simple bench sends a batch of requests to one serving configuration and reports how fast the tokens came back, usually as tokens-per-second. In practice it is a first pulse-check: it tells you a config runs and roughly how fast, in a few minutes and with almost no setup. It does not tell you which config to ship, because it measures a single config under conditions it never pins down. What does a simple inference benchmark actually measure, and what does it leave unstated? It measures throughput — output tokens or requests per second — for one config during one run. What it leaves unstated is the latency floor (p50/p95), the concurrency the throughput was achieved at, and the request mix that was sent. The reading is accurate for the conditions that applied; the problem is those conditions were not recorded, so the number floats free of the service level users would experience. Why can’t a simple bench’s tokens-per-second number be used directly to select a serving config? Because a simple bench measures one config under unstated conditions, so it can only rank configs against each other under whatever conditions happened to apply — conditions that were not held constant across runs. A throughput advantage measured at higher concurrency can vanish once every config is pinned to the same p95 latency target. Promoting a relative, uncontrolled reading to an absolute selection is the substitution that leaks money. What conditions (latency floor, concurrency, request mix) must you fix before a simple bench result becomes comparable? Three: pin p95 latency and treat it as a hard constraint rather than an output; hold the request mix and arrival pattern constant, ideally by replaying a captured production trace; and report cost-per-request at that pinned latency instead of raw throughput. Fixing those three makes every config a like-for-like comparison at the same service level. How do you extend a simple bench into a controlled cost-per-request comparison? Run each config against the same frozen trace at the same p95 target, then convert the sustained requests-per-second delivered inside that latency budget into cost by dividing the instance’s hourly price by that rate. This turns a raw tokens/sec figure into like-for-like cost-per-request and cost-per-token numbers, which is what exposes gross-margin deltas between throughput-optimal and cost-optimal configs. When is a simple bench good enough, and when do you need a full spec-organised benchmark? A simple bench is good enough for coarse, low-stakes questions: does the runtime start, does the model load, is this config catastrophically slow. You need a full spec-organised benchmark whenever a config-selection or purchasing decision hangs on the result — any comparison where a 20–30% cost-per-request delta would change the answer, which at production volume is most of them. How does a simple bench feed the cost-cut before/after and a defensible config-selection decision? The simple bench is the raw first reading. Turning it into a defensible decision means organising the specs — fixing p95 latency, the workload, and the pricing — so the cost-per-request comparison against the buyer’s deployed serving path is fair. That fixed, like-for-like table is what supports both the before/after cost-cut narrative and the config choice that stands up to review. A simple bench is the right instrument for a small question, and the wrong one for a large one. The discipline is not “always build the full harness” — it is knowing, before you act on the number, whether the decision can survive the conditions being left unstated. When it can’t, the move is to make those conditions explicit and re-read the result as cost-per-request; the Inference Cost-Cut Pack exists to turn that raw first reading into the fair before/after comparison against whatever serving path you already run.
