“Spec the computer” for a production AI feature usually collapses into picking a GPU SKU off a cloud menu and sizing it to peak load. That framing quietly answers the wrong question. The spec is not a hardware-procurement problem; it is an economics problem — and the two look identical right up until the invoice arrives. The naive version goes like this: pick the instance with the most VRAM headroom, confirm it clears the latency target on a benchmark run, and ship. It feels rigorous because there are numbers involved. But every number in that process describes what the hardware can do, not what each request costs to serve. A spec chosen for throughput headroom will pass a benchmark and still bleed margin — through idle capacity between traffic peaks, through oversized batches that inflate tail latency, or through a GPU class the model never comes close to saturating. The reframe is simple to state and harder to practise: a compute spec that cannot be traced back to a target cost-per-request, at the required p95 latency, given the real workload profile, is a procurement decision — not a production-AI decision. How does “spec-ing the computer” actually work for an AI feature? The phrase carries baggage from an era when spec-ing a machine meant matching a workload to a fixed box you would own for three years. In a cloud-served AI feature, the machine is elastic, billed by the second, and shared across a request stream whose shape changes hour to hour. So the spec stops being a single answer and becomes a mapping: from a workload profile to a serving configuration that meets a service-level objective at an acceptable unit cost. That mapping has more moving parts than the hardware line item suggests. The GPU class sets a ceiling on compute and memory bandwidth. But the model’s own arithmetic intensity, the batch size, the concurrency you allow, the KV-cache footprint per active sequence, and the serving runtime you deploy all determine how much of that ceiling you actually reach — and therefore what fraction of the hourly instance cost each request has to absorb. In our experience, the teams that get this right stop treating the GPU SKU as the decision and start treating it as one variable inside a cost function. The decision is the configuration; the SKU is a parameter you sweep. Which spec attributes actually move cost-per-request? Not all of them move it equally, and some that dominate the procurement conversation barely register in the economics. Here is the ordering we see repeatedly across serving-cost work, framed as an observed pattern rather than a benchmarked constant — your model and traffic will shift the weights. Attribute Effect on cost-per-request Why it is easy to get wrong Steady-state GPU utilisation Dominant. Idle silicon is billed silicon; low utilisation multiplies effective per-request cost. Peak-sized specs look safe but sit half-idle between traffic peaks. Effective batch size Large. Batching amortises fixed per-step cost across more requests — up to the point tail latency breaks the SLO. Oversized batches trade a good throughput number for a broken p95. GPU class vs. model saturation Large. A class the model never saturates pays for compute it cannot use. “Fastest instance” reasoning picks a SKU the workload can’t fill. Concurrency ceiling Moderate. Governs how many in-flight sequences share the KV cache and the batch. Set blind, it either starves throughput or overruns memory. VRAM / KV-cache headroom Threshold, not gradient. Below the model+cache footprint you fail; far above it you overpay. Treated as “more is better” instead of “enough plus margin”. The attribute that surprises people most is the first one. Utilisation is not a performance metric here — it is a cost multiplier. A GPU running at 30% steady-state utilisation is charging you roughly three times the per-request rate its raw throughput implies, because the idle two-thirds still shows up on the bill. That is why a spec tuned for peak-load headroom silently inflates unit economics: the headroom is exactly the capacity you pay for and rarely use. Why can a spec that benchmarks well still lose money? A benchmark measures the machine under conditions the benchmark author chose. Production measures the machine under conditions your users impose. The gap between those two is where cost-per-request leaks. Three mechanisms account for most of it. First, benchmarks usually run at saturating load — the exact condition that produces the best throughput number and the worst representation of a bursty production stream. Your feature spends most of its life below saturation, which is where utilisation, and therefore unit cost, is worst. Second, benchmark latency is often reported at the median or under a fixed batch, while your SLO lives at p95 or p99 under variable batching; a config that wins on mean latency can miss the tail by a wide margin. Third, the serving runtime and its scheduling policy — vLLM, SGLang, TensorRT-LLM, or a hand-rolled server — shape the achievable throughput-latency frontier as much as the GPU does, and a spec validated on one runtime does not transfer to another. This is the same reason a public leaderboard number can mislead a buyer: as we discuss in MLCommons and MLPerf Inference: where standard benchmarks stop short of cost-per-request, a benchmark result is a fair measurement of the wrong quantity for a unit-economics decision. It tells you the ceiling. It does not tell you where your traffic will actually sit under that ceiling. What utilisation should a right-sized spec target? There is no single correct number, because the answer trades directly against your latency headroom and traffic burstiness. But the useful framing is a band, and the band is narrower than most teams assume. A right-sized serving spec generally aims for a steady-state GPU utilisation high enough that idle billing does not dominate unit cost, yet low enough that a traffic burst does not blow the p95 SLO. In configurations we have profiled, that sweet spot tends to land well above the 30% figure that peak-sized specs drift toward, and comfortably below the near-100% that a benchmark run reports. Push utilisation too high and every burst queues, blowing the tail; leave it too low and you are renting idle silicon. This is an observed operating range, not a benchmarked target — the exact figure depends on your burst factor and how much latency slack the SLO grants you. The practical move is to decide the SLO first, then find the highest utilisation that still holds it under realistic burst, and only then choose the smallest GPU class that reaches that utilisation. That inverts the usual order, which starts from the SKU and hopes the economics work out. A checklist: validating a compute spec against a cost-per-request SLO Before signing off on a spec, run it against these. Each one is a place we have seen a “benchmarked-fine” configuration fail in production. Is there a stated cost-per-request target? If the spec cannot be traced to one, it is a procurement decision wearing an engineering costume. Is the workload profile real? Request-size distribution, arrival pattern, burst factor — measured from traffic, not assumed. Is latency validated at p95/p99, not the mean? Under the variable batching the config will actually use. What is projected steady-state utilisation, at typical load and not peak? Idle capacity is the largest hidden cost. Was the runtime fixed? The GPU spec is only meaningful paired with a named serving stack — a point developed in the SGLang DeepSeek-V3 cost-per-request serving benchmark walkthrough. Was a smaller class ruled out on economics, not reflex? “Fastest available” is rarely “cheapest per request”. Does the spec survive a doubling of traffic without a re-architecture? Or is it a point solution to today’s load? A spec that clears all seven is defensible. One that skips the first two is a guess with a receipt. Turning the profile into a defensible number The step that makes any of this real is profiling the deployed serving path. You cannot compute a credible cost-per-request from spec sheets and a benchmark; you compute it from measured throughput and latency under your own traffic, at your own utilisation, on your own runtime. That is where a GPU-level workload profile — the subject of our work on profiling GPU workloads — turns a compute spec from an assertion into a calculation. The machine learning compilers piece covers a related lever: the software stack can move the same hardware to a cheaper point on the curve before you touch the SKU. This is the core of the [inference cost-cut engagement](Inference Cost-Cut Pack) — profiling the serving path so the compute spec can be checked against your actual cost-per-request rather than a benchmark number. For teams building AI features as a product, the broader context lives on our AI infrastructure for SaaS page, where the spec question is really a gross-margin question in disguise. FAQ What’s worth understanding about spec computer first? For a production AI feature, “spec-ing the computer” is not choosing a box to own — it is mapping a workload profile to a serving configuration that meets a latency SLO at an acceptable cost-per-request. The GPU class is one variable in that mapping; batch size, concurrency, KV-cache footprint, and the serving runtime all shape how much of the hardware you actually use. In practice the decision is the configuration, and the SKU is a parameter you sweep against a cost target. Which compute spec attributes most affect cost-per-request? Steady-state GPU utilisation dominates, because idle silicon is still billed and low utilisation multiplies effective per-request cost. Effective batch size and whether the model actually saturates the chosen GPU class come next, followed by the concurrency ceiling. VRAM behaves as a threshold — enough plus margin — rather than a “more is better” gradient, so it rarely deserves the weight the procurement conversation gives it. How do we choose a GPU instance without over-provisioning? Invert the usual order. Fix the cost-per-request and latency SLO first, measure the real workload profile, then find the highest steady-state utilisation that still holds the SLO under realistic bursts, and only then pick the smallest GPU class that reaches it. Starting from “fastest available” and hoping the economics work out is how peak-sized specs end up half-idle and over budget. Why can a spec that benchmarks well still inflate cost-per-request? Benchmarks run under conditions the author chose — usually saturating load and median latency — while production runs under bursty traffic measured at p95 or p99. Your feature spends most of its life below saturation, exactly where utilisation and unit cost are worst. The serving runtime and its scheduler also shape the achievable frontier as much as the GPU does, so a benchmark number describes the ceiling, not where your traffic sits under it. How do we validate a compute spec against a cost-per-request SLO? Confirm the spec traces to a stated cost-per-request target, uses a real workload profile, validates latency at p95/p99 under variable batching, projects steady-state utilisation at typical (not peak) load, and fixes the serving runtime. Then check whether a smaller GPU class was ruled out on economics rather than reflex. The only way to produce a credible number is to profile the deployed serving path under your own traffic, not to read it off a spec sheet. What utilisation level should a right-sized spec target at steady-state? There is no universal figure, because it trades against latency headroom and burstiness — but the useful target is a band, not a point. It should sit well above the ~30% that peak-sized specs drift toward, so idle billing doesn’t dominate, and comfortably below the near-100% a benchmark reports, so a traffic burst doesn’t blow the p95 SLO. This is an observed operating range from serving-cost work, not a benchmarked constant; the exact figure depends on your burst factor and SLO slack. The uncomfortable question to sit with is not “which instance is fastest” but “at what point does buying more headroom start costing me margin I will never recover in throughput?” That is the boundary a benchmark cannot draw for you — only your own traffic can.
