“We enabled SpecForge and inference got slower.” We hear a version of this more often than you would expect, and it is almost always the same misunderstanding: speculative decoding is not a switch you flip for a free speedup. SpecForge is a toolkit for training the draft model that speculative decoding depends on, and whether it reduces your decode-phase latency comes down to a single number — how often that draft model’s guesses are accepted by your target model on your actual traffic. Get the draft model wrong and you add verification work without recovering it, which pushes tail latency up, not down. That is the whole story in one paragraph, and the rest of this article unpacks why. If you take one thing away: speculative decoding trades extra compute per step for fewer steps, and it only wins when the trade lands in your favour. What is speculative decoding, and why does it need a separately trained draft model? Autoregressive LLM inference generates one token at a time. Each decode step runs a full forward pass through the target model to produce a single token, then feeds that token back in for the next step. During this decode phase the GPU is frequently underutilised — the arithmetic per step is small relative to the memory traffic of reading the weights, so you are bandwidth-bound, not compute-bound. That mismatch is the opening speculative decoding exploits. The idea: use a small, cheap draft model to propose several tokens ahead in a single pass, then have the large target model verify all of those proposed tokens in one forward pass. Because that verification pass is roughly the same cost as generating one token normally, you can accept multiple tokens for the price of one target-model step — provided the draft’s proposals match what the target would have produced anyway. Any proposed token the target rejects is discarded, and generation resumes from the last accepted position. The output is identical to what the target model would have produced alone; speculative decoding changes throughput, not the distribution. This is why the draft model cannot be arbitrary. It has to predict the target model’s next tokens with high fidelity, on your traffic, or its proposals get rejected and you have paid for a draft pass that bought you nothing. SpecForge exists to train that draft model. It supports Eagle-style draft architectures — where the draft reuses the target model’s hidden states and feature representations rather than being an independent small model — which tends to align the draft with the target far better than bolting on an off-the-shelf small LLM. The alignment is the point. A draft model trained on generic web text will diverge from a target model that has been fine-tuned for, say, your customer-support domain, and that divergence shows up directly as a lower acceptance rate. How the acceptance rate decides whether SpecForge actually helps The token acceptance rate is the fraction of draft-proposed tokens the target model accepts. It is the single variable that determines whether speculative decoding is a speedup or a tax. The intuition is simple: if the draft proposes γ tokens per step and the target accepts a high fraction of them, you advance several positions per target forward pass. If it accepts almost none, you have spent a draft pass plus a verification pass to advance one token — slower than plain decoding. Here is the mechanism made concrete, with the numbers framed as illustrative rather than measured. Worked example: when the trade lands, and when it doesn’t Assume, for illustration, a draft model that is roughly 10× cheaper per forward pass than the target, proposing 4 tokens per speculative step. The relevant comparison is expected target-model steps per output token. Scenario Acceptance rate Approx. tokens accepted / step Net effect on decode latency Well-fit draft (on-domain) ~80% ~3.2 Meaningful speedup — fewer target passes per token Moderate fit ~55% ~2.2 Modest speedup, sensitive to draft cost Poorly-fit draft (off-domain) ~25% ~1.0 Break-even at best; overhead often net-negative Mismatched draft <15% <0.7 Slower than plain decoding; tail latency rises The figures above are illustrative of the shape of the trade-off, not a benchmark of any specific model pair. The takeaway is structural: acceptance rate is not linear in usefulness. Below some threshold — which depends on your draft-to-target cost ratio and proposal length γ — the verification and draft overhead swamp the benefit, and p95/p99 latency gets worse because rejected speculations cost real GPU time. This is the divergence point the naive “just enable it” framing misses entirely. When SpecForge helps, and when it raises tail latency Speculative decoding is a decode-phase optimisation. It does nothing for the prefill phase, where you process the prompt in parallel and are already compute-bound. So the first question is whether your latency actually lives in decode. If profiling shows most of your wall-clock time is prefill — long prompts, short completions — speculative decoding is the wrong lever, and a technique like prefix caching will move the needle instead. We cover that adjacent path in what a radix cache is and where it cuts LLM inference latency. When decode dominates and generations are long, speculative decoding is squarely in scope — but three conditions still gate whether SpecForge pays off: Traffic predictability. Draft models trained with SpecForge fit a distribution. If your production traffic drifts far from the training distribution — new domains, adversarial inputs, a different language mix — acceptance rate falls and the speedup erodes. This is an observed pattern across serving deployments, not a benchmarked constant. Batch size. At large batch sizes the decode phase becomes more compute-bound, which shrinks the underutilisation gap that speculative decoding exploits. The win is largest at small-to-moderate batch sizes and low-latency single-stream serving. Draft cost discipline. The draft model has to stay cheap. An Eagle-style draft head that reuses target hidden states is inexpensive; a draft that is itself a heavy model can eat the entire benefit. The failure mode to watch for: a draft model that looks fine in aggregate throughput but degrades at the tail. Because rejected speculations are wasted work, a bimodal traffic pattern — most requests on-distribution, a minority off — can leave your p50 improved while p99 quietly regresses. If you serve to an SLO, the tail is what you are paid to protect, so measure percentiles, not just the average. The comparison worth keeping in mind is with training-free approaches. Lookahead decoding achieves speculative-style speedups without a separate trained draft model, trading a lower ceiling for zero training effort. SpecForge sits at the other end: more engineering investment, higher achievable acceptance rate when the draft is well-fit to your target and traffic. How does SpecForge fit alongside quantisation and batching? Speculative decoding is one lever in the inference-optimisation toolkit, and it composes with the others rather than replacing them. The levers attack different bottlenecks, which is exactly why stacking them works. Technique Bottleneck it attacks Changes model output? Composes with SpecForge Speculative decoding (SpecForge) Decode-phase memory-bound underutilisation No — output identical to target — Quantisation (e.g. FP4/INT8) Weight memory footprint & bandwidth Yes — small accuracy trade-off Yes; a quantised target is cheaper to verify with Continuous / dynamic batching Throughput under concurrent load No Yes, though large batches reduce the speculative win KV-cache reuse / prefix caching Prefill recomputation No Yes; targets a different phase A common and effective combination is a quantised target model plus a SpecForge-trained draft: quantisation lowers the per-pass cost, speculative decoding lowers the number of passes. If you are weighing precision trade-offs on the target, our explanation of how FP4 works and when it cuts inference cost covers the accuracy side of that decision. And when decode and prefill have genuinely different scaling behaviour, disaggregating them — as described in SGLang PD disaggregation — lets you apply speculative decoding to the decode workers specifically. Runtimes matter here too. Speculative decoding is implemented in serving frameworks like SGLang, vLLM, and TensorRT-LLM, and the quality of that implementation — how it handles the tree of speculative tokens, verification batching, and KV-cache management for rejected branches — affects the realised speedup as much as the draft model itself. The same acceptance rate can yield different wall-clock results across runtimes. Measuring the impact before and after deployment The reason SpecForge belongs in the “algorithmic optimisation” bucket — the kind that often beats adding more GPUs — is that a well-fit draft model raises tokens-per-second at the same GPU count, which lowers cost-per-1M-tokens without a hardware purchase. But that is a claim you have to verify on your own traffic, not assume. A disciplined evaluation looks like this: Baseline the target alone. Record p50/p95/p99 decode latency and cost-per-1M-tokens on a representative traffic sample, plain autoregressive decoding. Measure the draft’s acceptance rate. Run the SpecForge-trained draft against the same sample and record accepted-tokens-per-step. This is the leading indicator; if it is low, stop here. Measure wall-clock at percentiles, not just the mean. Re-run with speculative decoding enabled and compare percentiles against the baseline. Watch p99 specifically for tail regression. Recompute cost-per-token at the new throughput and same GPU count, and compare against the marginal cost of adding GPUs to hit the same latency target. These are operational measurements from your own serving stack, and they are the reference standard — no published acceptance rate transfers cleanly to your model and traffic. If profiling reveals the bottleneck is decode-phase compute rather than memory transport, speculative decoding is one of the algorithmic remedies a GPU performance audit would recommend before it recommends more silicon. The distinction between memory-bound and compute-bound decode is what tells you whether SpecForge is even the right tool. This work is a natural extension of closing the prototype-to-production gap in model serving, which we treat as a first-class engineering problem in our generative AI practice — a trained draft model is one of the concrete artifacts that gets a demo-quality LLM to production latency and cost. FAQ What does working with SpecForge involve in practice? SpecForge is a toolkit for training the small draft model that speculative decoding relies on, including Eagle-style drafts that reuse the target model’s hidden states. In practice it means you invest engineering effort up front to produce a draft model that closely predicts your target model’s outputs on your traffic, so that at serving time the target accepts most of the draft’s proposed tokens and advances several positions per forward pass. What is speculative decoding, and why does it need a separately trained draft model? Speculative decoding uses a cheap draft model to propose several tokens ahead, then verifies them all in a single target-model pass, accepting the ones the target agrees with. It needs a well-fit draft model because rejected proposals are wasted GPU work — a draft that diverges from the target produces low acceptance and can make inference slower rather than faster. SpecForge exists to train a draft that aligns tightly with a specific target and traffic distribution. How does the draft model’s token acceptance rate determine whether SpecForge actually reduces latency? Acceptance rate is the fraction of proposed tokens the target accepts, and it is the single variable that decides the outcome. High acceptance means many tokens advance per target pass, giving a real speedup; low acceptance means you pay for draft plus verification passes to advance one token, which is slower than plain decoding. Below a threshold set by your draft-to-target cost ratio, the overhead outweighs the benefit. When does SpecForge speculative decoding help, and when can it increase tail latency instead? It helps when latency lives in the decode phase, generations are long, batch sizes are small-to-moderate, and traffic stays close to the draft’s training distribution. It can raise tail latency when the draft is poorly fit or traffic drifts off-distribution, because rejected speculations cost real GPU time — this can improve p50 while quietly regressing p99, which matters most if you serve to an SLO. How does SpecForge fit alongside quantisation and batching in the inference-optimisation toolkit? Speculative decoding attacks decode-phase memory-bound underutilisation, quantisation attacks weight memory footprint, and batching attacks concurrent throughput — so they compose rather than compete. A common effective stack is a quantised target model verified against a SpecForge-trained draft, though very large batches shrink the speculative win because decode becomes more compute-bound. How do I measure the latency and cost-per-token impact of a SpecForge-trained draft model before and after deployment? Baseline the target alone at p50/p95/p99 and record cost-per-1M-tokens, then measure the draft’s accepted-tokens-per-step as a leading indicator. Re-run with speculative decoding enabled, compare percentiles (watching p99 for tail regression), and recompute cost-per-token at the new throughput and the same GPU count. These operational measurements on your own traffic are the reference standard; published acceptance rates do not transfer. Is training a draft model with SpecForge worth the engineering effort versus scaling to more GPUs? It is worth it when a well-fit draft raises tokens-per-second at the same GPU count, lowering cost-per-1M-tokens without a hardware purchase — which frequently beats adding GPUs on cost. The decision hinges on whether your bottleneck is decode-phase compute versus memory transport, and on whether your traffic is predictable enough to sustain a high acceptance rate; if either fails, the training investment may not pay back. What a second opinion would flag here Speculative decoding is one of the clearest cases where the right algorithmic change on the inference path outperforms buying more hardware — but only conditionally, and the condition is measurable. Before you commit engineering time to SpecForge, the question to answer is not “will speculative decoding make us faster” but “does our decode-phase workload, at our batch sizes and traffic distribution, support a draft-model acceptance rate high enough to beat the verification overhead — and can we prove it at p99, not just p50?” If profiling can’t yet answer that, the draft model is not the first thing to build; the profile is.
