SGLang PD Disaggregation Explained: How Prefill/Decode Separation Works

PD disaggregation splits compute-bound prefill from memory-bound decode. How SGLang moves the KV cache, and when the split loses to colocation.

SGLang PD Disaggregation Explained: How Prefill/Decode Separation Works
Written by TechnoLynx Published on 11 Jul 2026

PD disaggregation (prefill/decode disaggregation) runs the two phases of LLM inference on separate GPU pools. Prefill is compute-bound: it reads the whole prompt and builds the KV cache. Decode is memory-bound: it generates tokens one at a time from that cache. Splitting them stops long prompts stalling every decode stream that shares a GPU with them, but only while the KV cache moves between the pools fast enough. SGLang implements the split by pairing a prefill server with a decode server through a handshake and moving the KV cache over RDMA, using libraries such as Mooncake and NIXL. Whether the split beats a colocated server depends on the interconnect, request load, how much multi-turn traffic you serve and which latency target is the tight one. That makes it a decision about serving topology, and turning it on will not by itself buy throughput.

Why prefill and decode fight over the same GPU

The two phases have different compute, memory and scaling characteristics. Prefill processes all prompt tokens in parallel and is compute-bound, while decode processes one token per request per step and is memory-bound, so decode benefits from batching and prefill does not. The DistServe authors (OSDI 2024) show the scale of this: a single 512-token prefill for a 13B model is enough to make an A100 near compute-bound. Decode has a different limit. Its speed is set by how fast the GPU can read model weights from memory, and that sets inter-token latency (ITL).

Put both phases on one GPU and they interfere in both directions. In DistServe’s measurements, adding one prefill job to a decode batch slows both: decodes wait for the longer prefill, and the slowdown grows with prompt length. The Nexus paper (2025) reports that, under chunked batching, mixed batches run 8 to 10 times slower than prefill-only or decode-only batches, and that this shows up even inside lightweight decode kernels. Sarathi-Serve’s authors measured naive hybrid batching raising time-between-tokens by up to 28.3 times compared with a decode-only batch.

The second problem is coupled scaling. When both phases share GPUs, the allocator has to provision for the worst case of time-to-first-token (TTFT) and time-per-output-token (TPOT) at once, which leads to over-provisioning. Disaggregation lets each phase take its own parallelism and resources. Because decode often leaves GPUs underused, several prefill instances can feed one decode instance. In DistServe’s illustrative case, giving prefill 2 GPUs and decode 1 served the model at 3.3 requests per second per GPU, 2.1 times what the colocated systems achieved.

How does SGLang hand the KV cache from prefill to decode?

SGLang’s PD disaggregation pairs a prefill server and a decode server for each request. The decode side reserves memory first, so the KV cache has somewhere to land as soon as prefill computes it. The LMSYS team describe the sequence like this:

  1. A prefill server and a decode server pair through a handshake, setting up a local sender and receiver.
  2. The decode server pre-allocates KV cache space and signals the prefill server to start the forward pass.
  3. Prefill computes the KV cache, which is transferred to the decode server.
  4. The decode server runs iterative token generation.

The transfer itself runs on a background thread, so the scheduler’s event loop stays uninterrupted. It uses RDMA queue pairs and scatter-gather elements to move non-contiguous memory chunks, and SGLang exposes APIs that plug in RDMA libraries such as Mooncake and NIXL. Decode workers register their RDMA connection details and base GPU memory pointers with prefill workers, then allocate pages for the incoming cache. Prefill writes the cache straight into decode’s GPU memory while decode polls for completion, a zero-copy GPU-to-GPU transfer with little CPU involvement. The Mooncake project lists its Transfer Engine as an officially supported SGLang backend for this exchange, across devices and across machines.

Several details in SGLang’s documentation matter once you run it:

  • When decode reserves KV slots. By default, decode reserves each request’s destination slots before prefill finishes, so a long prefill can hold capacity while decode waits. The --disaggregation-decode-allocation-policy prefill_complete flag delays the reservation until prefill signals readiness. The documentation says to compare both policies on your workload, because deferral removes the overlap between chunked prefill and KV transfer and is not a general throughput gain. This initial implementation requires Mooncake (RDMA or mooncake_tcp), attention TP and pipeline sizes of one, and it fails validation on a list of unsupported features. That list includes decoupled speculation (see speculative decoding as a decode-side runtime lever).
  • Different TP sizes on each side. When prefill and decode use different tensor parallelism, their KV layouts differ. A GPU staging buffer gathers head slices on the prefill side, sends them as one bulk RDMA transfer and scatters them on decode. SGLang reports a 2 to 5 times throughput gain over per-token slicing at high concurrency, landing within about 5% of homogeneous-TP baselines. The buffer is for GQA and MHA models, and MLA models such as DeepSeek-V2/V3 should not enable it.
  • Transport and threads. NIXL defaults to the UCX backend, and LIBFABRIC or another installed plugin can be selected with SGLANG_DISAGGREGATION_NIXL_BACKEND. Transfer worker threads and parallel queues are set through SGLANG_DISAGGREGATION_THREAD_POOL_SIZE and SGLANG_DISAGGREGATION_QUEUE_SIZE. The default timeout for receiving the KV cache is 300 seconds.

Does PD disaggregation trade time-to-first-token for inter-token latency?

In vLLM’s guide, yes. PD disaggregation protects inter-token latency, and above all its tail, and it pays for that with a transfer cost on the path to the first token. How large that payment is depends on the request load and the KV transfer medium.

The cost scales with KV size. vLLM’s disaggregated serving guide works it through: Llama-3.1-70B in BF16 stores 320 KiB of KV per token, so a 10k-token prompt hands decode about 3 GB. That takes roughly 65 ms at a 400 Gb/s line rate before any overhead, and all of it lands on TTFT. As a check, 3.2 GB × 8 bits ÷ 400 Gb/s ≈ 64 ms. The benefit shows on the other side. In the same guide, on two L40S GPUs, collocated p99 ITL jumped to 169 ms at 0.4 requests per second while the P/D setup held at 29 ms and never went above 52 ms. Median ITL was about the same in both setups. The guide attributes the collocated tail to 8k-token prefills landing on a GPU that was also decoding.

The sources disagree on how much the transfer costs. DistServe found KV transmission was less than 0.1% of total latency even for OPT-175B, with over 95% of requests delayed by under 30 ms, because its placement kept the transfer on intra-node NVLink. The semi-PD authors (2025) say the transfer is comparable to the time for one generated token, which means the second output token waits several times longer than the rest.

TaiChi’s authors (2025) give a decision rule based on latency targets. Aggregation does best under a tight TTFT target with relaxed TPOT, and disaggregation does best under strict TPOT with relaxed TTFT. Under balanced targets, neither approach alone delivers the best goodput.

When PD disaggregation loses to a colocated server

One failure case is the slow-interconnect test in vLLM’s guide. The test box had two L40S GPUs with no peer-to-peer copy support and no RDMA. Each 8k-token prompt produced about 470 MB of KV cache (56 KiB per token for Qwen2.5-7B), which took about 1.3 seconds to reach decode. At 0.2 requests per second, P/D median TTFT was 2.2 seconds against 0.7 seconds collocated. P/D therefore missed a 2-second TTFT target at every rate, even though its ITL tail stayed flat. The authors note that bandwidth alone does not explain 1.3 seconds. Most of the time went on overhead around the copy, including decode noticing a finished transfer only when it polled between its own forward steps. So line rate alone did not predict the transfer cost there. Measure the full handoff, polling included.

An energy and performance study (2026) also found the benefit is not guaranteed. The benefit depends on request load and transfer medium: under equal GPU resources, disaggregation doubles the load per GPU and degrades TTFT when high-speed interconnects are missing. At larger batch sizes, disaggregated serving wins on TPOT instead. Colocated serving runs out of KV memory, evicts and recomputes cache, and loses on TPOT. The same study found every disaggregated setup used more energy than the colocated baselines, even with independent frequency scaling per phase.

Condition Effect on the PD split Source
No RDMA, NVLink or peer-to-peer path between pools Transfer lands on TTFT; P/D can miss TTFT targets that colocation meets vLLM guide
Low request load, small batches Colocated serving can win on TTFT; P/D’s TPOT edge appears at larger batches Energy study
Small, fast-prefilling model such as Qwen3-0.6B Little interference for P/D to remove vLLM guide
Offline, throughput-oriented work with loose latency targets Chunked prefill with piggybacking may be preferable DistServe
One or a few GPUs Little room to tune allocation; a non-disaggregated system may be simpler and more efficient DistServe
Multi-turn chat and agents on a one-way transfer Prefill recomputes history and repeated transfers saturate bandwidth PPD paper
Long prompts on a fast interconnect Transfer grows linearly with length while prefill compute grows quadratically, so relative overhead shrinks DistServe
Strict TPOT, relaxed TTFT Disaggregation’s strongest regime TaiChi

Multi-turn traffic needs a closer look because it exposes limits that single-turn workloads do not reveal. In canonical designs, the KV cache moves only from prefill to decode. The decode node already holds the previous turn’s response KV, but prefill cannot reach it, so prefill recomputes the whole conversation and sends it back. The PPD authors (2026) measured a full prefill causing about 48% TPOT slowdown at batch size 200, compared with about 2% for an append-prefill that reuses cached KV. On that basis they route follow-up turns to decode nodes, which cuts KV transfer load by about 75% at an average of 3.1 turns per conversation. vLLM has a bidirectional_kv_xfer option that lets prefill pull those blocks back from decode instead of recomputing them.

Chunked prefill is a colocated alternative. It splits a prefill into compute-sized chunks and coalesces them with ongoing decodes, so each iteration’s latency stays bounded. The Sarathi-Serve authors state the trade-off plainly. Disaggregation removes interference and runs prefills at full efficiency, giving better TTFT than chunked prefill. However, it has to migrate the KV cache, which is hard without high-bandwidth interconnects, and it leaves prefill replicas’ GPU memory underused. Hao AI Lab’s retrospective on DistServe points the other way: even with chunked prefill, one large prefill can inflate TPOT by 2 to 30 times, especially under bursty load.

The split also has to be sized. An SLO-aware allocation paper (2026) derives the P:D ratio from input and output lengths and the throughput of each phase. In its worked case, a 100 ms KV transfer reduced effective prefill throughput from 28,300 to about 25,000 tokens per second. The resulting three-prefill, four-decode deployment met both a 2-second TTFT and a 20 ms TPOT target at about 4.8M tokens per minute.

How is SGLang’s PD disaggregation different from disaggregated prefill in vLLM?

The two engines orchestrate the handoff differently. The underlying idea is the same, and so are some transfer libraries. vLLM runs two instances, one for prefill and one for decode, and connects them with a connector that lets the KV consumer fetch caches from the KV producer. In the NIXL flow described in vLLM’s guide, a proxy first calls prefill with max_tokens=1 and do_remote_decode. Prefill computes the cache, holds the blocks and returns kv_transfer_params, and decode then pulls the blocks over NIXL before generating. In SGLang, decode pre-allocates its KV space during the handshake, before prefill computes, and prefill sends to it.

Measured costs in vLLM’s PD have shown up on the transfer path. In the LMCache authors’ measurements (2025), vLLM’s native PD disaggregation sent paged KV page by page when pages were scattered in GPU memory, which underused bandwidth. LMCache’s buffered transfer cut mean TTFT by 1.53 to 1.84 times on an 8K-input, 200-output workload. A newer push-based NIXL connector reverses that flow. Decode registers its target blocks in advance and a prefill-side background thread prepares and submits the write. At 8 QPS and 16k input, pull mode reached about 1,744 ms TTFT against roughly 540 ms for push, because pull posted NIXL descriptors inline on the decode worker’s engine loop.

Reading these together, SGLang’s design (decode reserves the destination early, transfer on a background thread) is structurally close to vLLM’s newer push mode. Push has a cost. The decode instance must be chosen before prefill starts, which gives up some of the load-aware routing that pull allows. Mooncake serves both engines: SGLang supports its Transfer Engine, and vLLM uses it through a MooncakeConnector.

How does PD disaggregation combine with DP attention and expert parallelism for DeepSeek-style MoE?

For large MoE models, PD disaggregation lets prefill and decode run different communication modes and different expert parallelism degrees. LMSYS name three problems with unified scheduling. Prefill batches interrupt decode. Under DP attention, one worker may run prefill while another runs decode, which raises decode latency. And DeepEP’s dispatch modes cannot both run in one scheduler. DeepEP’s normal dispatch maximises throughput for long prefill inputs but is incompatible with CUDA Graph. Low-latency dispatch supports CUDA Graph for decode but needs a fixed memory preallocation. Without disaggregation, SGLang’s auto mode cannot run both in the same communication group. With it, prefill uses normal dispatch and decode uses low-latency dispatch, both under DP attention.

The split also allows different expert placement, because expert load distributions differ between the phases. DeepSeek’s own inference system uses routed-expert EP32 for prefill and EP144 for decode. LMSYS tested each phase separately on H100s. Prefill on 4 nodes at EP32 reached 50,302 to 57,674 tokens per second per node for 4K to 1K prompts, up to 3.3 times the TP16 baseline. Decode on 9 nodes at EP72 reached 22,282 tokens per second per node for 2K inputs, 5.2 times TP16, largely because EP allowed 4 times larger batches. Expert load balancing added 1.49 times on prefill and 2.54 times on decode. Two-batch overlap needs care: in decode it gave minimal or negative gains below a batch threshold, for example -27% at 32 tokens per device.

Those figures come with a condition. LMSYS gave the phase not under test unlimited resources so that the tested nodes ran at maximum load. That makes them per-phase capacity numbers. Our walkthrough of SGLang expert parallel serving for DeepSeek and what profiling reveals goes further into the EP side.

Can prefill and decode be separated within a single GPU (intra-GPU disaggregation) instead of across separate nodes?

Yes. Two research systems partition a single GPU’s streaming multiprocessors (SMs) between the phases, which avoids cross-device KV transfer and weight replication. Nexus (2025) is a single serving engine. It uses a cost model and greedy search to pick SM partitions at runtime, with shortest-prompt-first scheduling for prefill and FCFS for decode. It applies hysteresis so that repartitioning, which is not free under Green Contexts, happens only on significant changes. semi-PD (2025) partitions SMs through CUDA MPS while keeping KV storage unified, and its authors built it on SGLang.

The results show where each approach wins. Nexus improved TTFT by up to 1.6 times over SGLang. On a mixed workload it reached 1.8 times SGLang’s throughput and 1.4 times that of vLLM-P/D, which used two GPUs. vLLM-P/D still had the best average and P95 time-between-tokens, because it fully separated the phases onto dedicated GPUs. semi-PD reduced average end-to-end latency by 1.27 to 2.58 times on DeepSeek models and served 1.55 to 1.72 times more requests within SLOs on Llama models. Its authors point out that cross-GPU disaggregation replicates weights. Llama3-8B needs at least two GPUs that way, and replicas only pay off when the request rate is high enough to fill them.

Our reading: intra-GPU separation makes most sense on a single node or a small fleet with mixed prompt lengths. Cross-node disaggregation keeps its edge where decode tail latency is the binding target and a fast interconnect exists. The results for both systems come from their own authors’ papers.

How do you evaluate SGLang PD disaggregation against production traffic?

Benchmark the split against a colocated server on the same GPUs and judge it on goodput together with tail ITL and TTFT. vLLM’s guide recommends this shape: check the transfer first, then run the same benchmark against the P/D proxy and against one collocated server with data parallelism 2 on the same two GPUs, comparing p99 ITL, TTFT and goodput. Goodput is the request rate you can sustain while requests still meet both their TTFT and ITL targets.

A configuration tuned once does not stay optimal across workloads. The PPD authors found that in 92.2% of workload-QPS combinations, the best configuration for Turn 2 TTFT differed from that for average TPOT. DistServe re-runs its placement when the workload shifts, because a plan tuned to one pattern goes stale. A run that matches production traffic follows this order:

  1. Measure the raw KV handoff on your interconnect at your longest common prompt, polling and setup included, before any end-to-end run.
  2. Fix identical GPUs, model, precision and KV cache settings for both topologies, and record the P:D ratio, TP per side, transfer backend and allocation policy.
  3. Replay production prompt and output lengths and arrival patterns, not a uniform synthetic set.
  4. Sweep request rate up to and past your expected peak, and report goodput against your actual TTFT and ITL targets.
  5. For chat or agent traffic, measure later turns separately from turn one.

How should prompt-length distribution and concurrency be represented in the eval to expose tail-latency behaviour under long-context requests?

Tail latency is where the topologies differed, so the eval has to include the tail deliberately. Long prompts carry two separate costs. Transfer size grows linearly with prompt length. Colocated interference grows faster: the PPD authors measured full-prefill interference reaching a 3 to 4 times slowdown at 32K tokens. Concurrency compounds both. vLLM’s pull-mode TTFT stayed close to push at low load and short prompts, and it pulled apart only as QPS and input length rose together. So an eval should keep the long-context tail of your real distribution at its production frequency and run it under concurrent load. A short-prompt average hides exactly the regime where the topologies differ. For multi-turn work, vLLM’s guide advises measuring TTFT on turn five, not turn one.

Whether a PD win carries over to the next model or GPU candidate depends on the run conditions behind it. Those include KV bytes per token, interconnect type, request load and SLOs. Treating those conditions and the captured results as first-class parts of the test is what the LLM evaluation framework run conditions and evidence layers are for. They let you re-run the comparison instead of trusting one number. For turning such runs into a purchasing or deployment decision, our cost-per-request inference benchmarking examples show how the comparison is framed.

For teams, the open question is whether their own interconnect and traffic mix sit on the side of the table where disaggregation wins. Only a replay of their own traffic answers that.

FAQ

What does chunked prefill do?

Chunked prefill splits a long prompt’s prefill into compute-sized chunks over several iterations, batching each chunk with ongoing decodes so new requests join without pausing generation. It bounds per-iteration latency at the cost of somewhat slower prefills than unchunked ones.

What is time to first token latency?

Time to first token (TTFT) is the time from a request arriving to the first output token being generated by the prefill phase. In disaggregated serving it includes any KV cache transfer that has to finish before decode can start.

What is inter-token latency?

Inter-token latency (ITL) is the gap between successive output tokens during decode. It is set mainly by how fast the GPU can read model weights from memory.

Back See Blogs
arrow icon