The naive way to serve a large language model is to point every request at one GPU pool, turn on continuous batching, and trust the scheduler to hide the fact that a single request goes through two completely different kinds of work. It does not hide it. Prefill — the pass that ingests the prompt and builds the KV cache — is compute-bound and arrives in bursts. Decode — the token-by-token generation that follows — is memory-bandwidth-bound and runs for a long time. When both live on the same GPUs, one phase constantly starves the other, and you end up over-provisioning the whole cluster to keep either one happy.

SGLang’s PD disaggregation is the structural answer to that mismatch. It splits prefill and decode into separate GPU pools that you scale independently, connected by a path that moves the KV cache from the prefill workers to the decode workers. This article explains the mechanism and its trade-offs. It is not an argument that every deployment should adopt it — for a lot of workloads, co-located serving is still the right call, and the interesting engineering question is knowing which case you are in.

Why prefill and decode want different hardware

The two phases of autoregressive inference have almost opposite resource profiles, and that is the whole reason disaggregation exists.

Prefill processes the entire prompt in one shot. Every token attends to every prior token, so the arithmetic scales with the square of the sequence length, and the GPU’s matrix units run hot. It is a compute-bound operation — the bottleneck is FLOPS, and a well-fed prefill worker can push its tensor cores close to saturation. It is also bursty: a prompt lands, gets processed in a handful of forward passes, and then that request moves on.

Decode is the opposite. Each step generates one token by attending over the KV cache built during prefill. The matrix multiplications are small — batch dimension of one per request in the naive case — so the GPU spends most of its time waiting on memory rather than computing. Decode is memory-bandwidth-bound, and it is long-running: a 500-token response is 500 sequential steps, each cheap in FLOPS but each paying the full latency of reading the KV cache out of HBM. This is the same bandwidth-versus-compute distinction that shows up when you read what DGX Spark memory bandwidth actually means for inference bottlenecks — the number that governs decode is bytes-per-second, not FLOPS.

Put both on one GPU and continuous batching tries to interleave them. A long prefill for one request blocks the decode steps of everyone already generating, spiking their inter-token latency. Conversely, a pool sized to keep decode latency low sits with idle compute during the gaps between prefill bursts. In configurations we have looked at, this is the dominant source of wasted GPU time in co-located serving — not the model, not the kernels, but the fact that two workloads with incompatible profiles are fighting over the same silicon (observed pattern across profiling engagements; not a published benchmark).

How does SGLang PD disaggregation work in practice?

SGLang runs prefill and decode as two distinct sets of workers. A request first goes to a prefill worker, which ingests the prompt, runs the full forward pass, and produces the KV cache for that sequence. The KV cache is then transferred to a decode worker, which takes over and streams out tokens one step at a time until the sequence completes. The router coordinates the handoff so that from the client’s perspective it is still one request.

The point of the split is that the two pools no longer share a fate. You size the prefill pool for your prompt-ingestion throughput — how many tokens of input you need to process per second — and you size the decode pool for your concurrent-generation load — how many sequences are actively producing tokens. When traffic shifts toward longer prompts, you add prefill workers without touching decode. When it shifts toward longer outputs, you add decode workers without touching prefill. That independent scaling is the throughput multiplier, and it is why the technique earns its added architectural complexity only when your traffic mix is genuinely two-sided.

There is a lineage worth naming here. The reasoning is the same one behind the broader SGLang serving stack — see how SGLang serving throughput plays out for DeepSeek-V3 and where the edge-deployment boundary sits — and it composes with prefix-reuse techniques like the radix cache that cuts repeated LLM inference work. Disaggregation restructures where the two phases run; the radix cache restructures whether prefill has to run at all for a shared prefix. They are orthogonal levers and stack cleanly.

How is the KV cache transferred, and what does that cost?

This is the part that decides whether disaggregation is a win, so it deserves to be stated plainly: the KV cache is not free to move, and the transfer path is the real constraint on whether PD disaggregation improves throughput-per-GPU-dollar.

When a prefill worker finishes, it has produced a KV cache whose size scales with the sequence length, the number of layers, the number of attention heads, and the precision of the stored tensors. For a long prompt on a large model, that is a substantial block of memory that now has to reach the decode worker before generation can begin. SGLang moves it over the fastest interconnect available — NVLink between GPUs in the same node, or RDMA over the network fabric between nodes — because the transfer time is added directly to the time-to-first-token the user experiences.

Three things determine whether the transfer overhead stays acceptable:

• Prompt length. Short prompts produce small caches that move quickly; the overhead is negligible. Very long prompts produce large caches, and if the interconnect is slow the transfer can eat the latency budget the disaggregation was supposed to protect.
• Interconnect topology. NVLink-connected pools inside one node pay far less than pools separated by a slower fabric. This is why 800G-class NICs and the interconnect they enable matter for cross-node disaggregation — the KV transfer is exactly the traffic that saturates a thin link.
• Precision of the cached tensors. Storing the KV cache at lower precision shrinks what has to move, trading a small accuracy cost for transfer speed.

If the transfer path cannot keep up, disaggregation makes latency worse, not better. That is the failure mode to watch for, and it is why the decision hinges on measurement rather than architecture-diagram enthusiasm.

When does disaggregation win, and when does co-location?

The honest answer is that it depends on your traffic shape and your interconnect. Here is the decision surface we use.

PD disaggregation vs co-located serving

Factor	Favours PD disaggregation	Favours co-located serving
Traffic mix	Prefill and decode load vary independently (e.g. long prompts, long outputs, both bursty)	Balanced, predictable ratio of input to output tokens
Cluster scale	Large enough to run meaningful separate pools	Small — a few GPUs where splitting leaves both pools underutilised
Interconnect	NVLink within node, or fast RDMA fabric across nodes	Modest network fabric where KV transfer would dominate latency
Prompt length	Short-to-moderate prompts (cheap KV transfer)	Very long prompts where cache movement is expensive
Operational maturity	Team can run and monitor a two-pool topology	Lean ops where one pool is all you can reliably operate
Primary goal	Maximise sustained tokens/sec at a fixed SLO	Minimise architectural complexity

The measurable outcome disaggregation targets is throughput improvement at a matched latency SLO, plus reduced idle GPU time — which converts directly into a lower cost per million tokens served. The trap is comparing raw peak throughput without holding the SLO fixed; a faster number at a worse latency is not the win you were looking for. Anchoring on cost per million tokens keeps the comparison honest, the same discipline that shows up when you model how tokenisation ratios and algorithmic choices drive real GPU inference cost.

If your cluster is small, or your traffic is a steady, balanced stream, splitting into two pools often leaves both underutilised and you are better off with continuous batching on a single pool. Disaggregation is a scaling technique; it pays when there is enough load to fill separately-sized pools.

What this shares with algorithmic-redesign-then-accelerate

The reason PD disaggregation belongs in the same family as the GPU-simulation work we do is the shape of the reasoning, not the domain. Take an RF or physics simulation on the GPU: you can get a modest speedup by parallelising the loops you already have, but the real multiplier comes from restructuring the computation to match how the hardware wants to run — data layouts that keep the memory subsystem busy, kernel fusion that removes launch overhead, and separating passes that have different arithmetic intensity.

Disaggregation is that same move applied to inference serving. Parallelising what you have — bigger batches, more GPUs on one undifferentiated pool — gives you incremental gains. Restructuring the workload so that the compute-bound phase and the bandwidth-bound phase each run on hardware sized for them is the structural change that unlocks the larger throughput improvement. In both cases the lever is recognising that a single monolithic computation is really two workloads with different hardware appetites, and giving each what it needs.

That recognition is exactly what a bottleneck analysis produces, and it is why the same question — is this workload’s constraint prefill compute, decode bandwidth, or KV-cache movement — sits at the front of any serving-cost engagement.

FAQ

What matters most about SGLang PD disaggregation in practice?

SGLang runs prefill and decode on two separate sets of GPU workers. A request is processed for prompt ingestion on a prefill worker, its KV cache is transferred to a decode worker, and that worker streams out the response tokens. In practice it means you operate two pools you can size and scale independently instead of one pool that has to satisfy both phases at once.

Why are prefill and decode treated as separate phases, and how do their GPU resource profiles differ?

Prefill processes the whole prompt in one compute-heavy, bursty pass — it is FLOPS-bound and can saturate the matrix units. Decode generates one token per step over the KV cache, spending most of its time waiting on memory bandwidth, and it runs for a long time. Because their bottlenecks are opposite, co-locating them means one phase starves the other, which is the mismatch disaggregation resolves.

How is the KV cache transferred between prefill and decode pools, and what overhead does that add?

After prefill completes, the KV cache is moved to the decode worker over the fastest available link — NVLink within a node or RDMA across nodes. The overhead scales with prompt length, cache precision, and interconnect speed, and it is added directly to time-to-first-token. If the transfer path is too slow relative to the cache size, disaggregation makes latency worse rather than better.

When does PD disaggregation improve throughput-per-GPU-dollar, and when does co-located serving remain the better choice?

It wins when prefill and decode load vary independently, the cluster is large enough to run meaningful separate pools, and the interconnect is fast enough to keep KV transfer cheap. Co-located serving remains better on small clusters, with balanced predictable traffic, on modest network fabric, or when very long prompts make cache movement expensive. The measured target is throughput at a matched latency SLO, not raw peak.

How does independent scaling of prefill and decode pools change cluster sizing and cost per million tokens?

You size the prefill pool for prompt-ingestion throughput and the decode pool for concurrent generation, adding capacity to whichever phase the traffic stresses without over-provisioning the other. This reduces idle GPU time compared with a single pool forced to satisfy both, which lowers cost per million tokens served at a fixed SLO. The gain is real only when there is enough load to keep separately-sized pools busy.

What does SGLang PD disaggregation share with the algorithmic-redesign-then-accelerate approach used for GPU simulation workloads?

Both start from the same recognition: a monolithic computation is really two workloads with different hardware appetites, and modest gains come from parallelising what you have while the real multiplier comes from restructuring to match the hardware. In GPU simulation that means data layouts and kernel fusion tuned to arithmetic intensity; in serving it means running the compute-bound and bandwidth-bound phases on separately-sized pools. The lever is structural redesign, not more of the same silicon.

Where to point the question

Before committing to a two-pool topology, the decision reduces to one measurement: is the workload’s binding constraint prefill compute, decode bandwidth, or the KV-cache movement between them? Answer that and the rest — pool ratios, interconnect requirements, whether co-located serving is still fine — follows. That bottleneck question is exactly what an A1 GPU audit on the GPU engineering practice is built to answer, and it is the honest gate on whether disaggregation is worth its complexity for your traffic.