You can run a clean inference benchmark and still make a wrong decision — because the spec that framed the run let two configs differ on more than one axis. The number came out; the comparison was never fair. The moment you declare Config A the winner because its cost-per-request is lower, you have to be able to say what else differed between A and B. If the answer is “batch size, and also concurrency, and a slightly different latency floor,” the margin you saw is an artefact of the setup, not a real property of the config. Spec organisation is where that fairness is either built in or lost. Before anyone hits run, someone decides which serving configs are candidates, what stays pinned, and what units the results report in. Treat that as bookkeeping and you get a table that looks comparable and isn’t. Treat it as experiment design and the cost-per-request column becomes the only thing that moves — which is exactly what makes a config-selection decision defensible. How does spec org work? “Spec org” is shorthand for how a benchmark specification is structured before it runs. It is the set of decisions that turn a vague intent (“compare these GPUs for our serving workload”) into a runnable, comparable experiment. Three decisions carry almost all the weight: which axes are candidate variables you’re deliberately sweeping, which axes are fixed constraints held identical across every run, and which units the results table reports. The naive reading treats a spec as a list of things to try. You name a few GPU types, point the harness at a model, and let the numbers sort themselves. The problem is that a serving stack has many knobs — batch size, concurrency, max sequence length, tensor-parallel degree, KV-cache allocation, the target latency floor — and if the spec doesn’t pin them, each config drifts to whatever the runtime defaults to under load. Two configs then end up measured under two different operating points, and the resulting cost-per-request numbers are not on the same axis. In practice, a well-organised spec reads like a controlled experiment. One column varies (the candidate config). Everything that could confound the cost comparison is written down as a held constraint. The output is expressed in a unit that survives the comparison: cost-per-request, with cost-per-token alongside it when output length varies. That discipline is the same one that governs spec-ing the compute behind a production AI feature — you are deciding, up front, what the number is allowed to mean. What must a benchmark spec hold constant to keep the comparison fair? The fairness of a cost-per-request comparison comes from holding the demand side constant while you vary only the supply side. If the workload distribution shifts between configs, the cost delta you measure is contaminated. So the spec must pin the things that describe what the system is being asked to do, and leave free only the things that describe how it’s provisioned. Concretely, the request mix has to be identical across every config: the same distribution of input lengths, the same distribution of requested output lengths, the same arrival pattern, the same prompt corpus. If one run replays production traffic with a heavy tail of long prompts and another uses a synthetic uniform distribution, the prefill cost differs for reasons that have nothing to do with the config under test. This is why RAG workload patterns matter to what a benchmark actually proves — retrieval-augmented traffic has a distinctive length profile that has to be replayed identically, not approximated. The second thing to hold constant is the quality bar. If Config A quietly serves at a lower numerical precision or a smaller model variant, it will look cheaper because it is doing less work, not because it is more efficient at the same work. Pin the model, pin the precision, pin any decoding parameters that change output quality, and record them in the spec as constraints rather than sweep axes. How do you encode a p95 latency constraint so configs are compared at the same latency floor? This is the single most decisive organisational choice, and it’s the one most benchmark tables get wrong. Throughput and latency trade against each other continuously — you can almost always push more requests through a GPU by allowing larger batches, at the cost of tail latency. So a table that reports throughput or cost without pinning latency is reporting configs sitting at different points on that trade-off curve. The “cheapest” one is often just the one you let run slowest. The fix is to make latency a constraint, not an output. Pick the operationally meaningful percentile — usually p95 latency, sometimes p99 for user-facing paths — set it to the value your SLO actually requires, and require every config to be tuned so it meets that floor before its cost is recorded. In practice this means sweeping batch size and concurrency within each config until p95 sits at the target, then reading off the cost at that operating point. The batch size that satisfies the constraint will differ per config — that’s expected and correct, because batch size is a tuning knob in service of the constraint, not an independent variable you’re comparing. Encoding it in the spec looks like a fixed field — p95_latency_target_ms: 250 — that applies to every row, plus a per-config record of the batch/concurrency setting that achieved it. The GPU-level utilisation and latency measurements that let you confirm each config actually hit the floor come from GPU profiling of the per-config serving path; without those measurements the constraint is asserted rather than verified. What columns and units belong in a well-organised cost-per-request table? A defensible table separates the constraints (identical down every row) from the candidate variable (the config) from the outputs (what you’re comparing). Constraints should be visible so a reader can audit that nothing drifted. The primary output is cost-per-request; cost-per-token belongs alongside it whenever output length varies across the request mix, because a config that is cheaper per request but generates longer outputs may be more expensive per token. Cost-per-request benchmark spec table (illustrative structure) Column Role Example Config ID Candidate variable A100-tp2, H100-tp1, L40S-tp4 GPU type / count Candidate variable 2×A100 80GB p95 latency target (ms) Fixed constraint 250 (identical every row) Request mix Fixed constraint prod-replay-2026Q2 (identical every row) Model + precision Fixed constraint Llama-3-70B, FP8 (identical every row) Batch size @ constraint Tuned to satisfy p95 32 (differs per row — expected) Measured p95 (ms) Verification 244 (must be ≤ target) Sustained throughput (req/s) Intermediate 41 Cost-per-request (USD) Primary output 0.0031 Cost-per-token (USD) Secondary output 0.0000042 The three constraint rows carrying “identical every row” are what make the table fair. If any of them varies between configs, the cost-per-request column is comparing apples to a differently-provisioned orange. The measured-p95 verification column is not decoration: it’s the audit trail that proves each config was actually held to the floor rather than allowed to overshoot for cheaper throughput. This is observed-pattern framing — the exact columns depend on your serving stack — but the separation of constraint / candidate / output is the invariant, not the specific fields. Which axes are candidate variables versus fixed constraints? The dividing line is a question about intent: is this an axis you are trying to select over, or an axis whose variation would confound the selection? GPU type and count are almost always candidate variables — that’s the decision you’re making. Tensor-parallel degree and quantisation scheme are usually candidates too, when the point of the benchmark is to find the cheapest provisioning. Batch size and concurrency are the interesting case: they are neither pure candidates nor pure constraints. They are tuned in service of a constraint. You sweep them per config only to hit the p95 target, then freeze them at the value that does. The request mix, the model identity, the numerical precision (when precision is not itself the thing being selected), and the latency floor are constraints. They describe the job, not the provisioning. A common failure is to leave concurrency as an unstated default — the runtime picks one operating point for the A100 config and a different one for the H100 config, and the cost comparison silently becomes a comparison of two operating points rather than two GPUs. The choice of serving runtime shapes these numbers as much as the hardware does, which is exactly why the runtime and its version belong in the constraint block too. How does a badly organised spec produce a misleading throughput-optimal winner? Here is the mechanism, because it’s worth seeing concretely. Suppose you benchmark two configs and forget to pin latency. Config B is allowed to run at batch size 128; Config A, for whatever reason the harness defaulted to, runs at batch size 16. Config B posts much higher throughput and therefore a lower cost-per-request. It looks like the clear winner. But B is hitting p95 latency of 900ms while A sits at 210ms. If your SLO is 250ms, B does not qualify at all — its cheap cost-per-request is purchased with tail latency you cannot ship. The spec that produced this table declared a winner that fails the constraint it never encoded. Re-run B tuned down to meet 250ms and its throughput collapses; its real, constraint-satisfying cost-per-request may be higher than A’s. The winner flipped, and the only thing that changed was that the spec finally held latency constant. This is the throughput-optimal-versus-cost-optimal confusion in its raw form. A throughput-optimal config maximises requests-per-second with no latency ceiling; a cost-optimal config minimises cost-per-request at the latency floor you actually serve. A spec that doesn’t pin the floor cannot distinguish them, so it reports the throughput-optimal config and calls it cheap. We see this pattern regularly in benchmark tables handed over during a cost review — the numbers are real, the harness is fine, and the comparison is still invalid because one axis was left free. How does this feed a defensible config-selection decision? When the spec is organised so cost-per-request is the only thing that moves, the selection decision becomes auditable. You can point at two configs, show that every constraint row is identical, show that both met the p95 floor, and state the gross-margin delta between the throughput-optimal choice and the cost-optimal choice as a real number rather than an artefact. Nobody can reopen the decision by saying “but they weren’t compared fairly,” because the constraint columns are right there proving they were. That table is also the input to a before/after. A cost-cut engagement takes the buyer’s currently deployed serving path, drops it into the same spec as one more row, and measures its cost-per-request against the candidate configs at the identical latency floor and request mix — which is what the [inference cost-cut pack](Inference Cost-Cut Pack) is built to produce for teams running production LLM serving. The same discipline underpins how we think about cost-per-request for AI-infrastructure SaaS platforms, where a defensible unit-economics story depends on the comparison being fair in the first place. FAQ What matters most about spec org in practice? Spec org is how a benchmark specification is structured before it runs: which serving configs are candidate variables, which axes are held as fixed constraints, and what units the results report. In practice it turns a vague “compare these GPUs” into a controlled experiment where one column varies (the config) and everything that could confound the cost comparison is written down and pinned. What must a benchmark spec hold constant across serving configs to keep the cost-per-request comparison fair? Hold the demand side constant: the same request mix (input/output length distributions, arrival pattern, prompt corpus), the same model and numerical precision, and the same latency floor. If the workload or the quality bar shifts between configs, the cost delta is contaminated by something other than the provisioning you meant to compare. How do you encode a p95 latency constraint into the spec so configs are compared at the same latency floor? Make latency a constraint, not an output. Set a fixed p95 target that matches your SLO, apply it to every row, and tune batch size and concurrency within each config until it meets that floor before recording cost. The batch size that satisfies the constraint will differ per config — that’s correct, because batch size is a knob in service of the constraint, not an independent axis of comparison. What columns and units belong in a well-organised cost-per-request benchmark table? Separate constraints (identical down every row: p95 target, request mix, model+precision), candidate variables (config ID, GPU type/count), and outputs (cost-per-request as primary, cost-per-token when output length varies). Include a measured-p95 verification column so a reader can audit that each config actually held to the floor rather than overshooting for cheaper throughput. Which axes (batch size, concurrency, GPU type) count as candidate variables versus fixed constraints in the spec? GPU type and count are candidate variables — they’re the decision you’re making. Request mix, model, precision, and the latency floor are constraints that describe the job. Batch size and concurrency are the special case: they’re tuned per config only to hit the latency constraint, then frozen — neither a pure candidate nor a pure constraint. How does a badly organised spec produce a misleading throughput-optimal winner? If latency isn’t pinned, one config may run at a large batch size, posting high throughput and low cost-per-request while blowing past the latency SLO. It looks cheapest but doesn’t qualify. Re-tuned to meet the floor, its throughput collapses and its real cost may exceed a rival’s — the winner flips once latency is finally held constant. How does this spec organisation feed a defensible config-selection decision and the cost-cut before/after? When cost-per-request is the only thing that moves, the selection is auditable: identical constraint rows, both configs verified at the p95 floor, and a real gross-margin delta between the throughput-optimal and cost-optimal choice. The buyer’s deployed serving path drops into the same spec as one more row, giving a like-for-like cost-per-request before/after at the identical floor and request mix. The uncomfortable question a clean spec forces is not “which config is cheapest” but “at what latency floor” — because the answer changes the winner. If you can’t state the p95 constraint every row in your benchmark was held to, you don’t yet have a cost-per-request comparison; you have a throughput table wearing a cost label.
