A team turns on speculative decoding, watches tokens-per-second climb by a third on their staging benchmark, and ships it. Two weeks later the cloud bill hasn’t moved. The technique worked exactly as advertised — and it still didn’t touch their unit economics. That gap is the whole story of speculative inference, often shortened to spec int. The mechanism is elegant and the throughput gains are real on the right traffic. But “faster tokens on a benchmark” and “cheaper served requests in production” are two different claims, and the distance between them is where most teams lose the money they thought they were saving. What should you know about spec int in practice? Speculative inference runs two models instead of one. A small, cheap draft model proposes a short burst of tokens — say four or eight at a time — guessing what the large target model would produce. The target model then verifies all of those proposed tokens in a single forward pass, in parallel, rather than generating them one at a time. Every proposed token that matches what the target would have produced anyway gets accepted for free; the first mismatch is corrected, and the cycle repeats. The reason this can be faster is structural. Autoregressive decoding on a large model is memory-bandwidth-bound: each token requires reading the full weight set from HBM, and the arithmetic units sit mostly idle waiting on memory. A single verification pass over several draft tokens reuses that same expensive weight read to check multiple positions at once. You are amortising the bandwidth cost of the target model across several tokens instead of one. Critically, the output is identical to what the target model would have produced on its own. Spec int is a lossless decoding acceleration — it does not change the distribution of outputs, unlike quantisation or switching to a smaller model. That property is what makes it attractive: you keep the target model’s quality and, in theory, pay less per token. The published work on speculative decoding (the technique appears in vLLM, TensorRT-LLM, and SGLang serving stacks) frames it precisely this way — same output, fewer sequential target-model steps. What speculative inference actually costs you Here is where the benchmark-versus-bill gap opens. Spec int is not free. You are now running two models, and the draft model’s forward passes consume GPU time whether or not its guesses land. The economics reduce to one number that dominates everything else: the acceptance rate — the fraction of proposed draft tokens the target model accepts. When the draft model guesses well, you get several accepted tokens per target verification pass, and the cost of running the draft model is small relative to the saved target-model steps. When it guesses poorly, you pay for the draft model’s passes and still have to run the target model nearly token-by-token, because rejections force re-verification. At that point spec int is pure overhead — you have made the request more expensive, not less. This is the mechanism the naive “bolt it on and watch tokens-per-second climb” approach misses. Tokens-per-second on a curated staging prompt tells you the acceptance rate was high on that prompt. It says nothing about the acceptance rate on the traffic that dominates your production volume. What is the acceptance rate, and why does it decide the outcome? The acceptance rate is the single lever that turns spec int from a cost win into a cost sink. It depends on how well the draft model predicts the target model’s output, and that predictability varies enormously by request class. On text that both models find “easy” — boilerplate, common phrasing, structured formats the draft model has seen — acceptance rates are high and the technique pays off. On out-of-distribution traffic, rare domains, or adversarial inputs, the draft model’s guesses diverge from the target early and often. Acceptance collapses, and the draft-model overhead you added starts eating the gain. This divergence is not a corner case; it is the normal shape of real traffic, which is why a synthetic benchmark systematically overstates the benefit. The cost model is straightforward to reason about once you frame it around acceptance: Acceptance rate on the request class Draft-model overhead Net effect on cost-per-request High (draft closely tracks target) Small relative to tokens saved Cost-per-request drops — genuine win Moderate Roughly cancels saved target steps Roughly break-even; measure carefully Low (out-of-distribution traffic) Larger than tokens saved Cost-per-request rises — spec int is a loss (Directional cost structure, not a benchmarked table — the exact crossover points depend on draft/target size ratio, batch size, and hardware. Treat it as a reasoning frame, observed-pattern class.) The point of the table is not the thresholds. It is that the same feature can sit in any of the three rows depending on which request class you point it at — and production traffic is a mix of all three. How does spec int affect cost-per-request rather than raw latency? Latency and cost-per-request are correlated but not the same, and conflating them is the most common way teams misjudge spec int. Spec int reliably reduces latency for a single request when acceptance is decent, because fewer sequential target-model steps means the answer streams out faster. That is a genuine user-experience win and worth having on its own terms. But cost-per-request is a throughput-and-utilisation question: how many requests can one GPU serve per unit time, and what does that GPU cost. Speculative decoding interacts awkwardly with verification batching — the technique’s efficiency changes when you are already serving many concurrent requests at high batch sizes, because a large batch was already keeping the arithmetic units busy and hiding the memory-bandwidth stall that spec int exploits. In other words, spec int’s biggest wins come exactly in the regime where you have spare compute and a bandwidth bottleneck — low-to-moderate concurrency. At high concurrency, where cost-per-request is often already best because batching amortises the weight reads, the marginal benefit shrinks and the draft-model overhead can dominate. A team optimising for the demo (one user, low batch) sees a big number; the same config at production concurrency behaves differently. This is the same reasoning we apply to other decoding-layer optimisations. Our write-up of how lookahead decoding cuts inference cost-per-request walks through a parallel case where a throughput trick’s real economic value depends on the serving regime, not the headline speedup, and SGLang’s prefill/decode split shows how the serving architecture itself reshapes where cost lives. How do you measure whether spec int improved your unit economics? You cannot answer this from a benchmark. You answer it by measuring acceptance rate segmented by the request classes that dominate your actual volume, then translating that into cost-per-request. Checklist: proving spec int on production traffic Segment your traffic into the request classes that make up the bulk of production volume — not a uniform average, and not a synthetic prompt set. Weight by volume. Measure acceptance rate per class, live or on replayed production traffic. This is the number that decides everything downstream. Measure tokens-per-second per GPU at your real production concurrency and batch size, with and without spec int — not at batch size one. Compute cost-per-request from tokens-per-second-per-GPU and the GPU’s hourly cost, for each traffic-weighted request class. Compare the volume-weighted cost-per-request delta, not the best-case class. Spec int can win on one class and lose on another; only the weighted total touches your margin. Watch the tails — request classes with low acceptance may be small in volume but can dominate latency SLOs and skew average cost if they force many rejections. The relevant number that comes out of this is the per-request cost delta on the traffic mix that actually dominates production, and the resulting change in contribution margin. That is the only figure that answers whether spec int belongs in your serving path. This kind of measurement is exactly what a disciplined serving-path optimisation looks like — our [inference cost-cut pack](Inference Cost-Cut Pack) evaluates spec int against a buyer’s measured cost-per-request rather than adopting it because a benchmark improved. And it starts with profiling: on a real serving path, the acceptance rate and per-request cost delta only surface once you profile the serving path end to end, not the model in isolation. When is spec int the wrong optimisation? Spec int is one tool among several for lowering cost-per-request, and it is not always the right one. If your traffic is heavily out-of-distribution or highly varied, acceptance rates will be low and the draft-model overhead will outweigh the gain — a smaller, cheaper target model or a distilled variant may cut cost more reliably, at some quality cost. If your service already runs at high concurrency with large batches, the memory-bandwidth stall spec int exploits is already mostly hidden, and quantisation or a graph-compiling runtime will likely move the number more. And if the draft model itself is not cheap enough relative to the target — the size ratio matters — the whole scheme collapses. The honest framing is that spec int is a lossless latency accelerator that sometimes also cuts cost-per-request, in a specific regime, on a specific traffic profile. Treating it as a general cost lever is the mistake. It sits alongside quantisation, batching strategy, target-model selection, and runtime choice in a decision that has to be made against measured numbers, not technique reputation. For the broader picture of how these serving-path choices trade off for cost-sensitive AI products, see our work on AI infrastructure for SaaS. FAQ What does working with spec int involve in practice? A small draft model proposes several tokens at once, and the large target model verifies them all in a single parallel forward pass, accepting the ones that match what it would have produced. Because large-model decoding is memory-bandwidth-bound, checking multiple tokens in one weight read is cheaper than generating them one at a time. In practice it accelerates decoding without changing the output, but only pays off when the draft model’s guesses are accepted often. What is speculative inference (spec int), and how does a draft model plus a target model produce a single verified output? Speculative inference runs a cheap draft model to guess a burst of upcoming tokens, then runs the expensive target model once to verify all of them in parallel. Every draft token that matches the target’s own prediction is kept; the first mismatch is corrected and the cycle repeats. The final output is identical to what the target model would have produced alone — it is a lossless decoding acceleration, not a quality trade. How does spec int affect cost-per-request rather than just raw latency or tokens-per-second? Latency and tokens-per-second improve for a single request when acceptance is decent, but cost-per-request is a throughput-and-utilisation question. Its value depends on your serving regime: the biggest wins come at low-to-moderate concurrency where a memory-bandwidth stall exists to exploit, while at high concurrency batching already hides that stall and the draft-model overhead can dominate. A demo-scale speedup does not guarantee a production-scale cost reduction. What is the acceptance rate, and why does it decide whether spec int lowers or raises the cost of a served request? The acceptance rate is the fraction of proposed draft tokens the target model accepts. High acceptance means many tokens per verification pass and a real cost drop; low acceptance means you pay for the draft model’s passes and still run the target model nearly token-by-token, making the request more expensive. It is the single lever that turns spec int from a win into a loss. On which request classes does spec int help, and where does draft-model overhead eat the gain? It helps on text both models find predictable — boilerplate, common phrasing, structured formats the draft model tracks well — where acceptance is high. It hurts on out-of-distribution traffic, rare domains, and adversarial inputs, where the draft model diverges early, acceptance collapses, and the overhead outweighs the saved target-model steps. Real production traffic is a mix, which is why a synthetic benchmark overstates the benefit. How do you measure whether spec int actually improved your unit economics on production traffic? Segment production traffic into its dominant request classes, measure acceptance rate per class on real or replayed traffic, then measure tokens-per-second per GPU at production concurrency with and without spec int. Convert that to cost-per-request per class and compare the volume-weighted delta — not the best-case class. Only the traffic-weighted change in cost-per-request and contribution margin answers whether spec int belongs in your serving path. When is spec int the wrong optimisation compared with quantisation, batching, or a smaller target model? Spec int is the wrong tool when traffic is heavily out-of-distribution (low acceptance), when you already run at high concurrency with large batches (the bandwidth stall is already hidden), or when the draft model is not cheap enough relative to the target. In those cases quantisation, a graph-compiling runtime, better batching, or a smaller distilled target model tend to move cost-per-request more reliably. Spec int is a lossless latency accelerator that only sometimes doubles as a cost lever. The question that actually decides it Speculative inference is worth having in the toolbox, and on the right traffic it earns its place. But the number that decides whether it belongs in your serving path is not the tokens-per-second on your staging run — it is the volume-weighted cost-per-request delta on the request classes that dominate your production traffic. Measure that, segmented by class and at real concurrency, before you let a benchmark convince you the bill will fall. The technique that looks like a win in the demo can be the one quietly raising your cost-per-request in production, and only the measured serving path will tell you which one you have.
