Teams find the SGLang PD disaggregation flag, turn it on, benchmark it once against a fixed prompt set, and report the higher aggregate tokens-per-second. That number is real. It is also almost useless as a serving-architecture decision, because it tells you the split was faster on that sample — not whether it survives your traffic.

PD disaggregation is not a throughput switch. It is a serving-topology decision whose payoff depends on run conditions you have to name explicitly: the prompt-length distribution your users actually produce, batch composition, KV-cache transfer cost between prefill and decode workers, and the prefill-to-decode ratio your concurrency generates. Flip it on against a synthetic prompt set and you get a leaderboard number. Fix the run conditions to production traffic and you get an answer you can defend in a serving-architecture review.

What is prefill, what is decode, and why would you separate them?

An LLM inference request runs in two phases with fundamentally different compute profiles. Prefill processes the entire input prompt in one forward pass, building the KV cache for every token in context. It is compute-bound and highly parallel — a long prompt saturates the GPU’s matrix units. Decode then generates output tokens one at a time, each step reading the growing KV cache and producing a single next token. It is memory-bandwidth-bound and latency-sensitive, because you cannot generate token N+1 until token N is done.

These two phases want different things from hardware. Prefill wants raw FLOPs and tolerates batching. Decode wants memory bandwidth and low per-step latency, and it starves the GPU’s compute units because each step touches only one token’s worth of new work. When prefill and decode share the same worker, a long prompt’s prefill pass stalls every decode step queued behind it — the classic head-of-line blocking that spikes inter-token latency under mixed traffic.

PD disaggregation splits the two phases onto separate GPU pools. Prefill workers do nothing but process prompts and emit KV caches; decode workers do nothing but generate tokens from KV caches transferred to them. In SGLang, this is a serving-topology choice layered on top of its RadixAttention KV-cache management and continuous batching. The intuition is clean: stop prefill from blocking decode, and scale each pool to the load it actually sees. The cost — and it is a real cost — is that the KV cache now has to move across the network from prefill worker to decode worker for every request.

If you want the serving-context framing for how SGLang fits alongside other runtime choices before you reach for the disaggregation flag, our note on serving-layer context for task-specific LLM evals with SGLang OME sets up the same run-conditions discipline this article applies to the PD split.

Which latency does PD disaggregation trade for which?

This is where the throughput-switch mental model breaks. PD disaggregation does not uniformly make everything faster — it trades two latencies that a monolithic deployment couples together.

• Time-to-first-token (TTFT) is dominated by prefill latency: how long until the first output token appears. A dedicated prefill pool can be scaled and batched for this without a decode queue getting in its way.
• Inter-token latency (ITL) is the decode-side per-step time: how fast tokens stream after the first one. A dedicated decode pool no longer stalls behind long prefills, so its per-step latency becomes more predictable.
• KV-cache transfer is the new tax: the cache must move from prefill worker to decode worker before generation starts. This adds to TTFT and only pays off when the decode-side and utilisation gains exceed it.

The honest framing is that disaggregation lets you tune TTFT and ITL independently — which is valuable precisely because a monolithic server forces a single trade-off across both. But independence is not a free lunch. On short-prompt, short-output traffic, the KV-cache transfer overhead can dominate any gain, and a co-located deployment wins. The benefit grows with prompt length and concurrency, where prefill blocking would otherwise be severe.

When does the split actually win? A decision table

The following table summarises how the payoff moves with workload shape. Treat the direction of each row as an observed pattern from serving-optimisation work, not a benchmarked rate — the crossover point is workload- and hardware-specific, which is the whole point.

Workload condition	Effect on PD disaggregation payoff	Why
Long prompts, short outputs	Strongly favourable	Heavy prefill would block decode in a monolith; dedicated prefill pool absorbs it
Short prompts, long outputs	Neutral to unfavourable	Little prefill blocking to remove; KV transfer tax has no offset
High concurrency, long context	Favourable, improves tail latency	Prefill contention is worst here; separation stops head-of-line blocking
Uniform short requests	Usually unfavourable	Transfer overhead per request exceeds any decode-side gain
Fast interconnect (NVLink / InfiniBand)	Raises the ceiling	KV-cache transfer cost is lower, so the crossover moves earlier
Ethernet-only KV transfer	Lowers the ceiling	Transfer tax grows; the split needs a bigger prefill-blocking problem to justify it

The single most common evaluation mistake is testing PD disaggregation against a prompt set whose length distribution does not match production. A benchmark of uniformly long prompts will make the split look unconditionally good; a benchmark of uniformly short prompts will make it look useless. Neither reflects the mixed, bursty, long-tail traffic a real endpoint serves. This is the same trap that catches teams reading a single SGLang DeepSeek-V3 cost-per-request serving benchmark without pinning the run conditions to their own traffic.

How do you evaluate PD disaggregation so the result matches production?

The difference between a leaderboard number and a serving-architecture decision is entirely in the run conditions. An evaluation that changes the topology flag but leaves the prompt set, concurrency model, and evidence capture unspecified cannot tell you whether the win is reusable. Three layers of the evaluation determine that.

Run conditions. The prompt-length distribution has to be sampled from real traffic, not a fixed synthetic set. Concurrency has to be modelled as an arrival process — bursts of concurrent long-context requests are exactly where prefill blocking hurts, and a fixed batch size hides that behaviour. You measure TTFT and ITL as separate distributions, not a blended average, because disaggregation moves them in different directions and an aggregate throughput headline can rise while tail TTFT gets worse.

Dataset construction. The eval prompt set should preserve the joint distribution of prompt length and requested output length, because the prefill-to-decode ratio is what determines whether the split pays. A dataset that decouples them — long prompts always paired with long outputs, say — produces a ratio your traffic never generates.

Evidence capture. Every reported number needs the configuration that produced it recorded alongside it: the prefill/decode pool sizes, the interconnect, the model, the quantisation, and the traffic sample. Without that, the win is not re-runnable when you swap the model or the hardware — which is the situation most teams are actually in, since the reason you tuned serving was to compare candidates. This run-conditions-plus-evidence discipline is exactly what an evaluation spec that links task, dataset, scoring, and run conditions exists to enforce.

Making a PD disaggregation comparison re-runnable against live traffic — rather than a one-off number on a static prompt file — is the job of a monitoring-and-evaluation harness with explicit run-conditions and evidence layers. That is where our Production AI Monitoring Harness work sits: capturing the traffic distribution and the exact serving config so the split can be re-tested when the next model or hardware candidate arrives, instead of re-benchmarked from scratch with unstated assumptions.

Tail latency under long-context concurrency

One number matters more than average throughput and is the easiest to miss: tail TTFT when several long-context requests arrive at once. In a co-located deployment, three simultaneous 32k-token prompts serialise their prefills against the same worker, and every decode stream behind them stalls — p99 TTFT and p99 ITL both blow out even though average throughput looks fine. This is the failure mode disaggregation is built to remove, and it is invisible to any benchmark that sends one request at a time or uses a fixed, modest batch.

So the eval has to represent concurrency and long context together, as a joint condition, and report the tail — not the mean. If you have worked through which machine learning model metrics actually decide a serving config, the pattern is familiar: the mean is where good serving decisions go to die, and the distribution is where they get made.

FAQ

How should you think about SGLang PD disaggregation in practice?

It splits LLM inference into two dedicated worker pools: prefill workers process the input prompt and build the KV cache, and decode workers generate output tokens from that cache. In SGLang this sits on top of its RadixAttention cache management and continuous batching. In practice it means prefill work no longer blocks decode streams, at the cost of transferring the KV cache across the network between the two pools.

What is the difference between prefill and decode phases, and why separate them onto different resources?

Prefill processes the whole prompt in one compute-bound, highly parallel pass; decode generates tokens one at a time and is memory-bandwidth-bound and latency-sensitive. Because they want different things from hardware, sharing a worker means a long prefill stalls the decode steps queued behind it. Separating them lets each pool be scaled and batched for its own bottleneck.

How does PD disaggregation affect time-to-first-token versus inter-token latency, and which one does it trade for which?

It lets you tune the two independently, which a monolithic server cannot. A dedicated prefill pool can improve time-to-first-token, and a dedicated decode pool makes inter-token latency more predictable by removing prefill blocking. The trade is a new KV-cache transfer cost added to TTFT, which only pays off when the decode-side and utilisation gains exceed it.

Under what workload conditions does PD disaggregation actually improve throughput, and when does the KV-cache transfer cost outweigh the benefit?

It wins on long prompts, high concurrency, and long-context traffic, where prefill blocking would otherwise be severe. It is neutral-to-negative on short, uniform requests, where there is little blocking to remove and the transfer tax has no offset. Fast interconnects like NVLink or InfiniBand move the crossover point earlier; Ethernet-only KV transfer pushes it later.

How do you evaluate SGLang PD disaggregation so results match production traffic rather than a fixed benchmark prompt set?

Sample the prompt-length distribution from real traffic, model concurrency as a bursty arrival process rather than a fixed batch, and measure TTFT and ITL as separate distributions instead of a blended throughput average. A fixed synthetic prompt set will make the split look either unconditionally good or useless depending on its length profile, neither of which reflects real endpoints.

Which evaluation-framework layers determine whether a PD disaggregation win is reusable on the next model or hardware candidate?

Run conditions (traffic-matched prompt distribution and concurrency model), dataset construction (preserving the joint prompt-length and output-length distribution that sets the prefill-to-decode ratio), and evidence capture (recording pool sizes, interconnect, model, quantisation, and traffic sample beside every number). Without the evidence layer the win cannot be re-run when you swap model or hardware.

How should prompt-length distribution and concurrency be represented in the eval to expose tail-latency behaviour under long-context requests?

Represent them as a joint condition — bursts of concurrent long-context requests — rather than testing one request at a time or with a fixed modest batch. Report p99 TTFT and p99 ITL, not the mean, because the failure mode disaggregation removes (serialised long prefills stalling decode streams) is only visible in the tail under simultaneous long-context load.

Reading this claim correctly, going forward

The question worth carrying out of here is not “is PD disaggregation faster” — it is “does the prefill-to-decode ratio my traffic produces, at my concurrency, over my interconnect, make the KV-cache transfer tax worth paying.” That is a serving-topology question with a defensible answer only when the run conditions are pinned to production. The flag is easy to flip. The evidence that the flip survives your workload — including tail behaviour under concurrent long-context requests — is the part that takes an evaluation built to be re-run, not a one-off benchmark you have to trust on faith.