Your p95 latency slipped last week. The APM trace shows the request was slow, GPU utilisation on the dashboard reads a healthy 80%, and the obvious move is to add another GPU. That move is often wrong — and PD disaggregation is why. Under the hood, every LLM request runs in two phases that look nothing alike. Prefill processes the entire prompt in one shot: it is compute-bound, it saturates the GPU’s matrix engines, and it produces the first token. Decode then generates the rest of the output token by token, each step reading the growing KV cache back from memory. Decode is memory-bandwidth-bound and spends much of its time waiting on HBM, not on compute. SGLang’s PD disaggregation runs these two phases on separate workers so each can be tuned for its own resource profile instead of fighting for the same GPU. That is the whole idea, and it is a serving-path fix rather than a model swap. If you already run SGLang or vLLM and your decode tail latency is dragging while utilisation looks fine, the bottleneck may be phase interference — a blind spot that request-scoped monitoring cannot see. How does SGLang PD disaggregation work in practice? In a monolithic serving setup, one worker pool handles both phases. When a new prompt arrives, its prefill step competes for GPU cycles with the decode steps of every request already in flight. Prefill is bursty and compute-heavy; decode is steady and bandwidth-heavy. Schedule them on the same device and each degrades the other — a large prefill stalls active decodes, and a queue of decodes delays incoming prefills. Disaggregation splits the pool. Prefill workers receive prompts, run the one-shot forward pass, and hand off the computed KV cache. Decode workers take that cache and generate tokens. The KV cache is transferred between them — over NVLink within a node or over the network across nodes — which is the one new cost the architecture introduces. Because the phases now live on dedicated hardware, you can size and tune each independently: batch prefill aggressively for throughput, tune decode for low inter-token latency, and scale the two pools to different ratios based on your actual prompt-to-output length distribution. SGLang implements this on top of its RadixAttention scheduler and continuous batching machinery. The same runtime that serves models like DeepSeek and Llama on a single worker can route the prefill and decode stages to distinct worker groups. If you are new to how the runtime schedules and batches requests, our walkthrough of how the SGLang runtime works and what to profile covers the scheduler internals this feature builds on. Why do prefill and decode have opposite resource profiles? This is the mechanical heart of the whole argument, so it is worth being precise. The divergence is not a quirk of one implementation; it falls out of the transformer inference math. Prefill computes attention and feed-forward layers over the entire prompt at once. With a prompt of length N, that is a batch of N positions moving through dense matrix multiplications — high arithmetic intensity, so the GPU’s compute units stay busy and memory bandwidth is rarely the limit. A single long prefill can push a modern accelerator close to its FLOPS ceiling. Decode is the opposite. Each step processes exactly one new token but must attend to every previous token, which means reading the full KV cache from HBM on every iteration. The matrix multiplications shrink to matrix-vector products with low arithmetic intensity, so the accelerator spends its time waiting on memory rather than computing. This is the classic memory-bandwidth wall, and it is the same failure class we describe in when memory bandwidth is the real inference bottleneck. Decode throughput is governed by how fast you can stream the KV cache, not by how many FLOPS the chip advertises. Run both on one GPU and you are asking a single device to be simultaneously compute-optimal and bandwidth-optimal. It cannot be. A compute-heavy prefill burst monopolises the SMs while queued decodes sit idle; a wall of decodes keeps memory saturated while a new prefill waits its turn. The interference is structural — which is exactly why splitting the phases helps. Prefill vs decode at a glance Property Prefill phase Decode phase Work per request One-shot, whole prompt Token by token Primary bottleneck Compute (matrix engines) Memory bandwidth (HBM / KV cache) Arithmetic intensity High Low Batching benefit High throughput per batch Bounded by bandwidth Latency contribution Time to first token Inter-token latency, tail What to tune for Throughput Low per-token latency The table is the quick answer: two phases, opposite profiles, one device that cannot serve both optimally at once. When does PD disaggregation help — and when does it add overhead? Disaggregation is not free, and treating it as a universal win is the mistake that mirrors the naive “just add a GPU” reflex. The KV cache transfer between prefill and decode workers costs latency and interconnect bandwidth. Whether that cost is worth paying depends on your workload shape. It tends to help when: Prompts are long relative to outputs (RAG, long-context summarisation, agents), so prefill is heavy and its interference with decode is severe. You run at scale with enough concurrent requests that phase contention is chronic rather than occasional. Your interconnect is fast — NVLink within a node, or a well-provisioned fabric across nodes — so the KV cache handoff is cheap relative to the contention it removes. It tends to hurt when: Outputs dominate prompts and prefill is a small fraction of the work, so there is little contention to eliminate. Request volume is low and a single monolithic worker is already underutilised — you would be paying transfer overhead to solve a problem you do not have. The interconnect is the constraint, so the KV cache transfer becomes the new bottleneck. This is a genuine trade-off, not a checkbox. It sits alongside other SGLang runtime levers such as speculative decoding for faster inference, and the right combination depends on measurement, not on which feature has the newest release note. In our experience across production LLM serving engagements, teams reach for disaggregation before they have confirmed prefill/decode contention is actually the dominant cost — an observed pattern, not a benchmarked rate, but a consistent one. How does an APM dashboard miss prefill/decode contention? This is the observability gap that sends teams down the wrong path. Application performance monitoring instruments the request boundary: it tells you the call arrived, spent 1,400 ms in the model-serving service, and returned. It is request-scoped by design. It cannot see that 200 ms of that time was time-to-first-token stalled behind a prefill queue, and another 900 ms was decode competing for memory bandwidth with three other requests’ prefills. GPU utilisation makes it worse. A device thrashing between compute-bound prefill and bandwidth-bound decode can report 80% utilisation while doing very little useful work — the SMs are “busy” waiting on memory. The dashboard looks healthy, the trace confirms the request was slow, and the two signals point at nothing actionable. So the team scales the deployment, which adds capacity to a contention problem instead of resolving it. Seeing the real bottleneck requires profiling inside the serving boundary — the kernel- and memory-level detail that distinguishes a compute stall from a bandwidth stall. That is where request-scoped APM ends and GPU profiling of the kernel and memory-bandwidth level begins. Mapping where each phase lives in the request path — the exercise in our guide to mapping the serving path for performance — is often the first step toward locating the contention the dashboard hid. How do you measure whether it worked? If disaggregation is a serving-path change, it has to be justified with serving-path evidence. Three metrics matter, and they map directly to what an inference audit produces. p95 latency, split by phase. Do not measure end-to-end latency alone. Separate time-to-first-token (prefill’s contribution) from inter-token latency and total decode time. Disaggregation should show up as a reduction in decode-phase tail latency once prefill stops interrupting it. If your p95 does not move, the contention was not your dominant cost. Effective GPU utilisation. Not the dashboard’s headline percentage — useful throughput per GPU. Removing phase interference typically lifts effective utilisation because each device now does work it is suited for. This is the number that reveals over-provisioning: GPUs bought to mask contention that a serving change resolves. Cost-per-request at a fixed SLA. This is the metric that turns a latency win into a business case. If disaggregation lets you hold the same SLA on fewer or better-utilised GPUs, cost-per-request falls. The unit-economics framework for AI infrastructure is what makes this comparable across configurations — latency alone does not tell you whether you saved money. Measure all three before and after. A change that cuts latency but raises cost-per-request (because the transfer overhead forced more hardware) is a change you should not ship. How does an audit decide if PD disaggregation is the right fix? PD disaggregation is one lever among several — batching, quantisation, speculative decoding, model routing, or a smaller model can each address inference cost, and they are not interchangeable. The audit’s job is to find which lever your bottleneck actually responds to. APM confirms the request is slow; profiling inside the serving boundary determines the cause. That is precisely what our [AI Inference Cost Audit](Inference Cost-Cut Pack) does: it profiles the serving path, isolates whether your cost lives in prefill contention, decode bandwidth, batching inefficiency, or numerical precision, and tells you whether disaggregation is warranted before you re-architect the deployment. Teams tuning SGLang or vLLM who cannot explain their inference cost are the ones this is built for — and it is one of the R&D engagements scoped to your problem we run when the dashboard and the trace disagree about what is slow. FAQ How does sglang pd disaggregation work in practice? SGLang PD disaggregation runs the prefill phase (one-shot prompt processing) and the decode phase (token-by-token generation) on separate worker pools instead of one shared GPU. The prefill worker computes the KV cache and hands it to a decode worker over the interconnect. In practice this lets you tune and scale each phase independently — batching prefill for throughput and decode for low inter-token latency — rather than forcing one device to serve two opposite workloads. Why do prefill and decode phases have opposite resource profiles, and why does running them together hurt latency? Prefill processes the whole prompt at once with high arithmetic intensity, so it is compute-bound and saturates the GPU’s matrix engines. Decode generates one token at a time while reading the full KV cache from memory on every step, so it is memory-bandwidth-bound. On a shared GPU, a compute-heavy prefill burst stalls active decodes and a queue of decodes delays incoming prefills — structural interference that inflates tail latency without any single request being obviously at fault. When does PD disaggregation help, and when does it add overhead that outweighs the gain? It helps when prompts are long relative to outputs, request volume is high enough for chronic contention, and the interconnect is fast enough that the KV cache handoff is cheap. It adds net overhead when outputs dominate prompts (little prefill contention to remove), volume is low (a single worker is already underutilised), or the interconnect is the constraint — in which case the transfer becomes the new bottleneck. How does an APM dashboard miss prefill/decode contention inside the serving boundary? APM is request-scoped: it reports that a call spent, say, 1,400 ms in the serving service, but not that the time split into a prefill-queue stall and bandwidth-starved decode. GPU utilisation compounds the blind spot, since a device thrashing between compute and memory work can read 80% busy while doing little useful throughput. The dashboard looks healthy, so teams scale capacity instead of resolving contention. How do you measure the effect of PD disaggregation on p95 latency, GPU utilisation, and cost-per-request? Measure phase-split p95 — separate time-to-first-token from inter-token latency — so a decode-tail improvement is visible rather than buried in an end-to-end number. Track effective (useful-throughput) GPU utilisation, not the dashboard headline, to expose over-provisioning. Then convert both into cost-per-request at a fixed SLA; a latency win that raises cost-per-request through transfer overhead is not a win worth shipping. How does an inference cost audit determine whether PD disaggregation is the right fix versus batching or quantisation? The audit profiles inside the serving boundary to locate where cost actually lives — prefill contention, decode bandwidth, batching inefficiency, or numerical precision — because these levers are not interchangeable. Only that phase-level evidence shows whether disaggregation addresses your dominant bottleneck or whether batching, quantisation, or a model change would do more for less re-architecture. How does SGLang’s PD disaggregation compare to running prefill and decode on a single monolithic worker in vLLM? A monolithic worker (the default in many vLLM deployments) schedules both phases on the same GPU, which is simpler and cheaper at low scale but exposes the phases to mutual interference under load. SGLang’s disaggregated path removes that interference at the cost of a KV cache transfer between pools, so it wins when contention is chronic and the interconnect is fast, and loses when volume is low or outputs dominate prompts. The real question is not “should we adopt PD disaggregation” but “does profiling inside the serving boundary show that prefill and decode are actually competing for the same GPU” — because that is the one thing an APM trace, however detailed, was never built to answer.
