When time-to-token slips on a production LLM, the first instinct is usually to swap in a smaller model or start a retrain. Both are expensive, and both assume the bottleneck lives in the model rather than in how you serve it. Speculative decoding in SGLang is a runtime lever that often recovers the lost latency without touching the model at all: a small draft model proposes several tokens ahead, and the target model verifies them in parallel, so you get more accepted tokens per forward pass. Configured correctly, it does not change the target model’s outputs. Enabled blind, it can make a workload slower.

That last sentence is the whole point of this article. Speculative decoding is not a free switch. It is a bet on the statistical structure of your decode path, and the bet only pays off under conditions you can measure ahead of time.

What does SGLang speculative decoding actually do?

Autoregressive decoding is sequential by construction. The model produces one token, feeds it back, produces the next, and so on. Each step is a full forward pass through a model that is frequently large enough to be memory-bandwidth-bound rather than compute-bound — you spend most of the step moving weights through HBM, not doing arithmetic. That is what “decode-bound” means, and it is why generating 200 tokens can cost far more wall-clock time than the compute alone would suggest.

Speculative decoding attacks that inefficiency directly. A small, cheap draft model runs ahead and guesses a short sequence of tokens — say four or five. The large target model then processes all of those candidate tokens in a single batched forward pass and checks each one against what it would have produced. Accepted tokens are kept; the first rejected token is corrected by the target model and the draft resumes from there. Because the verification pass is parallel over the candidate tokens, the target model can commit multiple tokens for roughly the cost of one sequential step.

The mechanism is a draft-and-verify loop, and it is closely related to the lookahead decoding approach we cover elsewhere, which reaches for the same parallelism without a separate draft model. SGLang supports speculative decoding as a runtime configuration alongside its other serving optimisations, so enabling it is a serving-path change, not a model change.

Why does draft-and-verify preserve the target model’s outputs?

This is the property that makes speculative decoding safe to reach for, and it is worth being precise about. The draft model only proposes. The target model is the sole authority on which tokens are kept. When the verification step accepts a token, it accepts it because the target model would have generated that exact token under its own sampling distribution; when it rejects one, the target model overrides it. Under the standard speculative-sampling formulation, the accepted output sequence is distributed identically to what the target model would produce on its own.

The consequence: speculative decoding is a throughput optimisation, not a quality trade-off — the target model’s output distribution is preserved by construction, not by tuning. A weaker draft model does not degrade quality; it degrades speed, because more of its guesses get rejected. That decoupling is the reason we treat this as a runtime lever rather than a modelling decision. You are changing how fast the answer arrives, not what the answer is.

There is a caveat class worth naming. The equivalence holds for the sampling procedure; it does not automatically survive sloppy configuration — mismatched tokenizers between draft and target, or a draft model trained on a different vocabulary, break the assumption. Those are integration bugs, not properties of the method, but they are exactly the kind of thing that turns a “safe” optimisation into a silent regression if you skip the before/after evaluation.

On which workloads does it help — and when does it hurt?

The economics of speculative decoding hinge on one number: the draft acceptance rate, the fraction of proposed tokens the target model keeps. When acceptance is high, each verification pass commits several tokens and you win. When acceptance is low, you pay for the draft model’s forward passes and the verification overhead while still advancing only a token or two per round — a net regression.

Acceptance rate is a property of the workload, not just the model pair. It tends to be high on predictable, low-entropy generation — structured output, code with strong local patterns, summarisation that closely tracks the source. It tends to be low on high-entropy, creative, or highly divergent generation where the small draft model simply cannot anticipate the target model’s choices.

Quick decision rubric: is your workload a speculative-decoding candidate?

Signal	Favours speculative decoding	Argues against it
Bottleneck (from profiling)	Decode-bound, memory-bandwidth-limited	Prefill-bound or compute-saturated
Output entropy	Structured, code, summarisation	Open-ended creative generation
Draft acceptance (measured on a sample)	Consistently high	Low or highly variable
Batch size at serving	Low-to-moderate concurrency	Very high concurrency (batching already fills the pass)
Draft/target vocab	Shared tokenizer	Mismatched vocab or tokenizer

The batch-size row is the one teams miss most often. At high concurrency, the target model’s forward pass is already well-utilised because many independent requests are batched together — the memory-bandwidth waste speculative decoding exploits has already been amortised, so the technique has less headroom to recover. This is an observed pattern across serving-optimisation engagements, not a fixed threshold: the crossover point depends on your model, hardware, and traffic shape, which is precisely why it has to be measured on your own path.

How do we measure whether it actually improved anything?

Because speculative decoding is judged on evidence, the measurement is not optional — it is the deliverable. The metrics that matter are the same serving KPIs an inference audit already baselines:

• Time-per-output-token (TPOT) and time-to-first-token (TTFT) — speculative decoding targets TPOT specifically; it does nothing for prefill, so if TTFT dominates your latency, this is the wrong lever.
• p95 latency under representative concurrency — averages hide the tail that users actually feel.
• Tokens-per-second throughput on the target runtime, at the batch sizes you serve in production.
• GPU utilisation — a rise in useful utilisation without a rise in cost is the signal that you converted wasted bandwidth into committed tokens.
• Draft acceptance rate — the leading indicator. If acceptance is low in your measurement, the other numbers will disappoint, and you have your answer before you ship.

The discipline here mirrors any runtime move: capture the baseline first, change one variable, re-measure under the same load profile. This is the same before/after lens we apply when mapping the serving path for performance — you cannot claim an improvement you did not baseline. Our [inference-cost-cut pack](Inference Cost-Cut Pack) exists precisely to produce that baseline and tell you whether the LLM path is decode-bound in the first place.

How does it fit alongside other SGLang runtime settings?

Speculative decoding is one lever among several, and they interact. SGLang’s prefill/decode disaggregation, which splits the two phases across execution to cut latency, addresses a different part of the pipeline — it is complementary, since disaggregation optimises where prefill runs while speculative decoding optimises how fast decode advances. Continuous batching, KV-cache management, and quantisation all change the utilisation picture that speculative decoding then works against, which is why the technique has to be evaluated in the configuration you will actually run, not in isolation.

The practical sequencing we recommend: profile to confirm the path is decode-bound, get the other runtime settings into their production shape, then layer speculative decoding on top and measure the marginal gain. Enabling everything at once and reading a single aggregate number tells you nothing about which lever earned the improvement — and leaves you unable to diagnose the regression if one of them backfires. When you are serving a specific model family, the runtime-tuning walkthroughs like serving DeepSeek-V3 on SGLang and what the runtime choice costs and saves show how these settings stack in a concrete case.

When is speculative decoding cheaper than swapping to a smaller model?

This is the decision the urgency framing points at. Both moves aim at latency, but they cost very differently.

A smaller model changes the output distribution — you accept a quality reduction, and you now own an evaluation burden to prove the smaller model is good enough for the task. That is real work, and it is irreversible in the sense that you have committed to a different model. Speculative decoding, when acceptance is high, recovers latency headroom without changing outputs and without retraining — it is reversible, it is a config change, and it preserves the model you already validated. On decode-bound workloads with high acceptance, it is almost always the cheaper first move.

The honest boundary: if your workload has structurally low draft acceptance, no amount of tuning makes speculative decoding pay, and a smaller model (or a routing strategy that sends easy queries to a cheaper model — see how model routing cuts LLM inference cost) may be the correct answer. The point is not that speculative decoding always wins. It is that it is the cheaper thing to try first, because it costs a profiling pass and a config change to find out, and it does not commit you to anything if it fails.

FAQ

How does SGLang speculative decoding work?

A small draft model proposes several tokens ahead, and the large target model verifies all of them in a single parallel forward pass, keeping the tokens it agrees with and correcting the first one it does not. In practice it means more tokens committed per pass on decode-bound workloads, cutting time-per-output-token without changing what the model produces — provided draft acceptance is high enough to offset the draft model’s overhead.

What is the draft-and-verify mechanism, and why does it preserve the target model’s outputs?

The draft model only proposes candidate tokens; the target model is the sole authority that accepts or rejects each one against its own sampling distribution. Under the standard speculative-sampling formulation, the accepted sequence is distributed identically to what the target model would have generated alone, so a weaker draft model reduces speed, not quality. The equivalence assumes correct configuration — a shared tokenizer and matched vocabulary between draft and target.

On which LLM workloads does speculative decoding actually help, and when does low draft acceptance make it slower?

It helps on decode-bound, memory-bandwidth-limited workloads with predictable, low-entropy output — structured generation, code, summarisation — where draft acceptance stays high. When acceptance is low, or at very high serving concurrency where batching already fills the forward pass, you pay for the draft passes and verification overhead while advancing only a token or two per round, producing a net regression. Acceptance is a property of the workload, so it must be measured on a representative sample.

How do we measure whether enabling it improved p95 latency, time-per-output-token, and tokens-per-second?

Baseline those metrics under a representative concurrency and load profile first, change only the speculative-decoding setting, then re-measure under the identical profile. Watch time-per-output-token and p95 latency for the intended gain, tokens-per-second and GPU utilisation for confirmation, and draft acceptance rate as the leading indicator — low acceptance predicts a disappointing result before you ship.

How does speculative decoding fit alongside other SGLang runtime settings when porting an LLM to a faster serving path?

It is complementary to prefill/decode disaggregation, continuous batching, KV-cache management, and quantisation, but it interacts with all of them because they change the utilisation picture it works against. Get the other settings into production shape, confirm the path is decode-bound, then layer speculative decoding on top and measure its marginal gain — enabling everything at once hides which lever earned the improvement.

When is speculative decoding a cheaper fix than swapping to a smaller model?

On decode-bound workloads with high draft acceptance it is almost always cheaper, because it recovers latency without changing outputs, without retraining, and reversibly — a config change rather than a model commitment. A smaller model forces a quality trade-off and a fresh evaluation burden. If draft acceptance is structurally low, though, a smaller model or a routing strategy may be the correct answer instead.

What does the audit need to confirm before we enable speculative decoding in production?

The audit’s profiler findings must confirm the LLM path is decode-bound — memory-bandwidth-limited during generation rather than prefill-bound or compute-saturated — and that measured draft acceptance on a representative sample is high enough to offset the draft model’s overhead. It also needs the baseline serving KPIs (TPOT, p95, tokens-per-second, GPU utilisation) so the before/after comparison is honest.

The open question this leaves on the table

Speculative decoding is a good example of why we treat “the LLM is too slow” as a profiling question before a modelling one. The lever is cheap to try, safe to output quality when configured correctly, and reversible — but it only pays on decode-bound paths with high draft acceptance, and those are conditions you confirm with measurement, not assumption. If your team is weighing it against a model swap or a retrain, the decision framing lives inside a broader R&D consulting and inference cost engagement, where the audit names the bottleneck before any runtime lever gets pulled. The failure class to avoid is enabling it blind on a workload whose acceptance rate never justified it — a regression you would have seen in a single profiling pass.