A single GPU pool serving a large language model is quietly doing two jobs at once, and the two jobs want opposite things from the hardware. The prefill phase — reading the prompt, building the KV cache — saturates compute. The decode phase — generating tokens one at a time — barely touches the compute units and instead pins memory bandwidth. When both share the same GPUs, one of them is always the reason the other is waiting. SGLang PD disaggregation splits those two phases onto separate pools so each runs on hardware suited to its bottleneck, and the reason it matters is narrow and measurable: it can lower cost-per-request when your prompt and generation lengths are imbalanced, and it can make things worse when they are not. That last clause is where most of the confusion lives. Teams read “disaggregation cuts cost” as a general truth and reach for it as a default. It is not a default. It is a lever whose payoff depends entirely on your traffic mix and on whether the routing and KV-cache-transfer overhead it introduces stays smaller than the utilisation it reclaims. This article grounds the mechanism carefully enough that you can decide the question for your own serving path rather than inherit someone else’s answer. Why prefill and decode have different hardware bottlenecks Start with what each phase actually does, because the cost argument falls straight out of the mechanics. Prefill processes the entire input prompt in one forward pass. Every token in the prompt attends to every prior token, and the model computes all of the key and value projections that will populate the KV cache. This is a large matrix-multiply workload — dense, parallel, and compute-bound. On a modern accelerator running under CUDA with fused attention kernels such as FlashAttention, prefill will happily drive the tensor cores toward their sustained ceiling. The longer the prompt, the more compute prefill consumes. Decode is the opposite shape. It generates one token per step, and each step reads the full KV cache back out of high-bandwidth memory to compute attention against it. The arithmetic per step is tiny; the memory traffic is not. Decode is memory-bandwidth-bound, and it spends most of a GPU’s compute capacity idle while it waits on HBM. The longer the generation, the more decode steps you pay for, each one dominated by memory movement rather than math. Put those two profiles on the same GPU pool and you get a scheduling problem with no good answer. A batch heavy on prefill leaves memory bandwidth underused; a batch heavy on decode leaves the tensor cores underused. The pool’s average utilisation looks respectable, but it is an average of two workloads each starving a different resource. This is the failure class disaggregation targets: co-located serving wastes GPU-seconds on whichever phase idles the resource the other phase needs. The insight that the unit of measurement is the executor — hardware plus software stack, not the chip’s spec sheet — is one we return to often, because it is the same reasoning that makes profiling the serving path the precondition for any of this. How SGLang PD disaggregation actually works SGLang PD disaggregation runs prefill and decode as separate worker pools connected by a KV-cache transfer path. A request arrives, gets routed to a prefill worker, which processes the prompt and produces the KV cache. That cache is then transferred to a decode worker, which streams out the generated tokens. The two pools can be sized independently, scheduled independently, and — critically — placed on hardware chosen for their respective bottlenecks. Because prefill is compute-bound and decode is memory-bound, you can allocate more compute-dense configurations to prefill and more memory-dense (or simply more numerous, smaller) configurations to decode. You can also batch each phase according to its own optimal shape instead of compromising on a batch geometry that half-suits both. SGLang’s scheduler handles the routing and manages the cache handoff; the serving-layer choices that shape these numbers are the same ones discussed in SGLang’s serving runtime and its effect on eval numbers. The expert framing here is to stop treating the mixed workload’s utilisation as a fixed given. Instead, treat prefill cost and decode cost as two distinct components of the per-request bill. Once they are separated, you can size and schedule each against its own bottleneck and measure the delta — rather than assuming the split pays for itself. Does separating prefill and decode always lower cost-per-request? No, and the honest answer is the useful one. Disaggregation introduces overhead that co-located serving does not carry: the KV cache has to be transferred from a prefill worker to a decode worker over the interconnect, and a routing layer has to decide where each request goes and track its state across the handoff. That transfer cost scales with the size of the KV cache, which scales with prompt length. If prompts are short and generations are long, the cache is small and cheap to move, and decode dominates the bill — a favourable case. If prompts are enormous, the cache is large, and moving it can eat much of what you reclaimed. The decision therefore turns on the imbalance in your traffic. The value of disaggregation comes from asymmetry: when one phase would otherwise leave a resource idle on the shared pool, giving it its own right-sized pool reclaims those GPU-seconds. When prefill and decode load are roughly balanced and the shared pool is already near saturation on both resources, there is little idle capacity to reclaim, and the transfer plus routing overhead is pure cost. Decision table: when does the split pay off? Traffic pattern Co-located behaviour Disaggregation outcome Short prompts, long generations (e.g. chat, agents) Decode-bound; tensor cores idle during long decode runs Usually lower cost-per-request — small KV cache, cheap transfer, decode pool right-sized Long prompts, short generations (e.g. summarisation, classification over long context) Prefill-bound; memory bandwidth idle Can help, but large KV-cache transfer eats into the saving — measure the transfer cost first Balanced prompt and generation lengths at high concurrency Both resources near saturation; little idle capacity Often worse — routing and transfer overhead exceeds reclaimed GPU-seconds Highly variable, bursty mix across both phases Head-of-line blocking; one phase stalls the other Frequently helps — independent scheduling absorbs the variance Evidence class: observed-pattern — these directions reflect how the mechanism behaves across serving-path work we have profiled, not a single published benchmark. The crossover point for your workload has to be measured, not assumed. The overhead that can cancel out the savings Two costs decide whether disaggregation earns its complexity, and both are concrete enough to measure before you commit. The first is KV-cache transfer. Every disaggregated request moves its cache from prefill to decode. On a fast interconnect — NVLink within a node, or a high-bandwidth fabric across nodes — this is cheap for small caches and tolerable for medium ones. Over PCIe or a slower network, or for very long prompts, it becomes a real line item. The transfer cost is roughly proportional to prompt length times model KV size per token, so a workload with a few very long prompts can pay disproportionately. The second is routing and state management. The scheduler has to place each request, track it through the handoff, and keep both pools fed. Under bursty or highly skewed load this coordination is where latency tails and scheduling inefficiency can creep in. It is not free, and it is the part most easily underestimated in a napkin calculation that only counts GPU-seconds. The rule that keeps teams honest: disaggregation only earns its complexity when the per-request cost reduction survives the added routing and KV-cache-transfer overhead. If profiling shows a 20% reduction in decode-pool idle time but the transfer and routing cost adds back most of that on your interconnect, you have added operational surface for no margin gain. The same discipline that governs whether a machine-learning compiler cuts cost-per-request applies here — the lever is real, but the net effect is workload-specific and has to be measured on the deployed path. How to measure whether it actually improved contribution margin Cost-per-request is the number that matters, and it decomposes cleanly once you have separated the phases. The measurable quantities are the GPU-seconds consumed per prefill, the GPU-seconds consumed per decode, and the effective utilisation of each pool once separated. Multiply GPU-seconds by your GPU-hour cost and you have the compute component of each request; add the transfer and routing overhead and you have the disaggregated total to compare against the co-located baseline. Checklist: quantifying the disaggregation decision Profile the co-located baseline first. Measure prefill GPU-seconds, decode GPU-seconds, and per-resource utilisation on your current shared pool under real traffic. Without this you cannot compute a delta. This is the step where prefill/decode imbalance either shows up or doesn’t. Characterise your traffic mix. Record the distribution of prompt lengths and generation lengths. The asymmetry between them is the single strongest predictor of whether disaggregation helps. Estimate KV-cache transfer cost. Compute cache size per request (prompt tokens × per-token KV bytes) and the transfer time on your actual interconnect topology — NVLink, InfiniBand, or PCIe changes the answer. Run the disaggregated config on the same traffic. Measure the same GPU-seconds-per-phase and per-pool utilisation, plus the routing and transfer overhead as separate line items. Compare cost-per-request, not utilisation. A prettier utilisation number that does not move cost-per-request is a distraction. The denominator of contribution margin is the target. The bridge between the abstract mechanism and a real decision is measurement on the serving path. Prefill/decode imbalance is not something you can eyeball from a model card; it emerges from your prompts, your generation lengths, and your concurrency. That is why the disaggregation decision is downstream of profiling — a point we make at length in the walkthrough of SGLang serving benchmarks and cost-per-request, and it is the same discipline that turns the broader question of how much a production AI feature actually costs per request into a defensible number. For teams profiling their AI serving costs and weighing serving-path levers like this one, our [AI infrastructure and cost-optimisation work](AI infrastructure and SaaS) starts from the same premise: measure the deployed path, then decide. The rest of the platform is documented on the TechnoLynx site. FAQ How does sglang pd disaggregation actually work? SGLang PD disaggregation runs the prefill phase (processing the prompt and building the KV cache) and the decode phase (generating tokens) on separate worker pools connected by a KV-cache transfer path. In practice this lets you size, schedule, and place each phase on hardware suited to its bottleneck — compute-dense for prefill, memory-dense for decode — instead of compromising on one shared pool that half-suits both. What is the difference between the prefill and decode phases, and why do they have different hardware bottlenecks? Prefill processes the whole prompt in one dense forward pass, so it is compute-bound and saturates the tensor cores. Decode generates one token at a time, reading the full KV cache from memory each step, so it is memory-bandwidth-bound and leaves compute largely idle. Because they stress opposite resources, running both on one pool means one phase is always starving the resource the other needs. How does separating prefill and decode change cost-per-request on a production serving path? Separation lets each pool run near the ceiling of its own bottleneck instead of averaging two half-idle workloads. When prompt and generation lengths are imbalanced, this reclaims the GPU-seconds a co-located pool would waste on the idle resource, lowering the denominator of cost-per-request. The gain is only real if it survives the transfer and routing overhead disaggregation adds. What overhead does disaggregation add (KV-cache transfer, routing), and when does it cancel out the savings? It adds KV-cache transfer — moving the cache from prefill worker to decode worker, a cost that scales with prompt length and interconnect speed — and a routing layer that places and tracks each request across the handoff. When prompts are very long (large caches) or the interconnect is slow, or when the workload is balanced and the shared pool is already saturated, that overhead can exceed the reclaimed capacity and cancel the saving. For what traffic mixes does PD disaggregation lower cost-per-request, and for which is co-located serving cheaper? Short-prompt, long-generation traffic (chat, agents) and bursty, variable mixes usually benefit, because the KV cache is small and there is idle capacity to reclaim. Balanced prompt-and-generation load at high concurrency, where both resources are already near saturation, is often cheaper co-located because there is little idle capacity and the transfer plus routing overhead is pure cost. How would a team measure whether disaggregation actually improved contribution margin per request? Profile the co-located baseline first — GPU-seconds per prefill, per decode, and per-pool utilisation under real traffic — then run the same traffic through the disaggregated config and measure the same quantities plus transfer and routing overhead as separate line items. Compare cost-per-request, not utilisation: a better utilisation number that does not move the cost denominator is a distraction. The delta on the deployed path, net of overhead, is the answer. The uncomfortable part of this decision is that the mechanism is easy to describe and the payoff is hard to predict. Prefill/decode imbalance lives in your traffic, not in the model card, and it moves as your usage shifts. The right posture is not “disaggregate” or “don’t” but “profile the serving path, quantify the imbalance, and let the measured cost-per-request delta — net of transfer and routing overhead — decide.” That is the failure class the inference-cost-cut work exists to catch: a serving-path lever adopted on faith instead of on measurement.
