A team stands up SGLang in front of a DeepSeek model, watches the GPU utilisation counter climb toward 95%, and concludes the serving stack is tuned. It is a reasonable conclusion. It is also, often, wrong. High utilisation tells you the GPU is busy; it does not tell you the GPU is busy doing useful work, and it says nothing about where the wall-clock time for a request actually goes. The metric SGLang reports describes its own scheduler — how many requests it batched, how full the running batch was — not the kernel-level truth of whether you are prefill-bound, memory-bound, or stalling on the host. That gap between what the framework reports and what a profiler sees is the whole subject of this article. SGLang is a strong serving engine, and DeepSeek is a demanding model to serve well. But the number that convinces a team the deployment is done — throughput, or a utilisation percentage — is measuring scheduler behaviour, and the real cost-per-token constraint frequently lives somewhere the scheduler metric cannot see. What does SGLang actually accelerate when serving DeepSeek? SGLang is not one optimisation. It is a bundle of three mechanisms that each attack a different part of the serving cost, and they do not all help the same workload equally. Confusing “SGLang is fast” with “SGLang made my workload fast” is the first mistake, because the answer depends entirely on which of the three your traffic actually exercises. The first mechanism is RadixAttention, a prefix-caching scheme that stores the KV-cache for shared prompt prefixes in a radix tree and reuses it across requests. If your traffic has heavy prefix overlap — a long shared system prompt, few-shot examples repeated across calls, multi-turn conversations — RadixAttention skips recomputing the attention state for the shared tokens. The gain here is a reduction in prefill compute, and it is proportional to how much prefix your requests share. On traffic with no shared prefix, RadixAttention does almost nothing; the cache-hit rate is the number that predicts the benefit. We cover the mechanism itself in more depth in RadixAttention and how KV-cache reuse cuts inference cost. The second is continuous batching (also called in-flight batching): instead of waiting for a fixed batch to finish before starting the next, SGLang admits new requests into the running batch as soon as slots free up during decode. This raises decode-phase GPU occupancy when requests have varied lengths, which is the normal case. The gain is throughput per GPU, and it scales with how much length variance your traffic has. The third is structured decoding — constraining generation to a grammar or JSON schema by masking invalid tokens at each step. This is a correctness and latency feature for structured output, not a raw-throughput lever, and it can occasionally add host-side overhead if the constraint engine is not on the fast path. Three mechanisms, three different beneficiaries. A profiler trace is how you find out which one your traffic is actually paying for. Why can utilisation look high while cost per token stays bad? Here is the mechanism the naive read misses. GPU utilisation — the figure nvidia-smi reports and the one SGLang surfaces — is the fraction of time at least one kernel was resident on the device. It does not distinguish a kernel doing dense matrix math near peak FLOPS from a kernel waiting on HBM reads, and it does not distinguish either from a memory-copy kernel shuffling KV-cache pages. A device can sit at 95% “utilisation” while every resident kernel is memory-bound and the compute units are mostly idle inside each kernel. For DeepSeek specifically, this matters more than for a dense model, and the reason is the architecture. DeepSeek’s larger variants use a mixture-of-experts design: each token is routed to a small subset of expert feed-forward blocks rather than passing through one dense stack. That means the arithmetic intensity per token is lower than a dense model of the same parameter count, the memory traffic is dominated by moving the right expert weights and the KV-cache, and the routing introduces load imbalance across experts. In a profiler trace, MoE serving frequently shows up as memory-bandwidth-bound gather/scatter kernels and uneven expert occupancy — not as the compute-bound GEMMs a dense-model mental model expects. We walk through those algorithmic choices in how DeepSeek inference works and what drives its GPU cost. The consequence is direct: the operationally relevant number is cost per token — tokens-per-second per GPU — not the utilisation percentage, and the two can move in opposite directions. You can raise utilisation by admitting more concurrent requests and simultaneously raise cost per token if the extra requests are thrashing the KV-cache. Utilisation is an activity signal (observed pattern across serving deployments); cost per token is the outcome you are actually paying for. The three places the bottleneck actually lives When we profile an SGLang/DeepSeek deployment, the real constraint almost always resolves to one of three regions. The point of profiling is to find which, because the fix for each is different and applying the wrong fix wastes both engineering time and capacity budget. Bottleneck region What it looks like in a trace What actually fixes it What does NOT fix it Prefill compute Long, dense GEMM kernels dominate; time scales with input length; high compute-unit occupancy Prefix caching (RadixAttention) if prefixes are shared; chunked prefill; prefill/decode separation Adding GPUs helps only if you are also throughput-limited KV-cache memory pressure Memory-bound kernels, frequent cache evictions/recompute, capacity errors under load Quantised KV-cache, larger cache budget, shorter max context, better admission control Faster compute — the GPU is waiting on HBM, not math Host-side scheduling / transfer GPU idle gaps in the timeline between kernels; CPU-side tokenisation or scheduling on the critical path; PCIe/H2D copies stalling decode More scheduler workers, async tokenisation, pinned-memory transfers, batching-policy tuning A bigger or faster GPU — the device is starved, not saturated The third row is the one teams most often overlook, because it is invisible to any GPU-centric metric. If the SGLang scheduler, request queue, or tokeniser is on the critical path, you will see periodic gaps in the GPU timeline where no kernel runs at all. That is a host-bound stall, and it caps throughput regardless of how much GPU you buy. The SGLang PD disaggregation approach — separating prefill and decode is one structural answer when prefill and decode contend for the same scheduler, but you only reach for it once a trace confirms the contention exists. How do I profile an SGLang/DeepSeek deployment? You do not need exotic tooling. The two profilers that answer the question are NVIDIA’s Nsight Systems and Nsight Compute, and they answer different halves of it. Nsight Systems gives you the timeline view: it shows CUDA kernel launches, memory copies, CPU threads, and — critically — the gaps between them. This is how you diagnose the host-side row of the table above. If the timeline shows the GPU idle while a CPU thread runs tokenisation or scheduling, the constraint is not the kernel. This is also where you see whether prefill and decode are serialising against each other. Nsight Compute goes one level down: it profiles an individual kernel and tells you whether it is compute-bound or memory-bound, what its achieved occupancy is, and how close it runs to the HBM bandwidth or FLOPS roofline. This is how you distinguish the prefill-compute row from the KV-cache-memory row — both keep the GPU “busy,” but one is limited by arithmetic and the other by memory reads, and Nsight Compute shows you which. A workable sequence: Reproduce representative load. Profile under traffic that matches production prefix-overlap and length distribution — a synthetic single-prompt loop will hide both the RadixAttention benefit and the KV-cache pressure. This step is where most profiling exercises quietly go wrong. Capture a Nsight Systems timeline first. Look for GPU idle gaps. If they dominate, you are host-bound; fix that before touching kernels. If the GPU is genuinely saturated, drop into Nsight Compute on the hottest kernels. Compute-bound prefill and memory-bound decode lead to different next steps. Correlate with SGLang’s own metrics. Its running-batch size and cache-hit rate are still useful — they tell you whether continuous batching is filling the batch and whether RadixAttention is landing. They are context for the trace, not a substitute for it. Re-measure cost per token, not utilisation, after each change. The utilisation number can rise while your actual cost rises with it. For the broader principle that a busy GPU is not a fast GPU, we lay out the observability angle in ML model metrics that explain GPU bottlenecks versus metrics that mislead. In what order should you fix things once you know the constraint? Profiling identifies the constraint; sequencing the fixes is a separate discipline. The rule is to attack the region that gates throughput right now, re-profile, and only then move on — because relieving one constraint usually exposes the next, and optimising a region that is not currently the bottleneck buys nothing. If host-bound: fix scheduling and transfer first. It is usually the cheapest change (config and worker counts, not model surgery) and it unblocks everything downstream. Adding GPUs before this is fixed spends money to keep more devices idle. If prefill-bound with prefix overlap: confirm RadixAttention is actually hitting the cache, tune the cache budget, consider chunked prefill so long prompts do not block decode. If KV-cache-memory-bound: quantise the KV-cache, cap max context to what the workload needs, and tighten admission control so you do not accept more concurrent sequences than HBM can hold. Only then evaluate scaling out. Adding GPUs helps when — and only when — you are throughput-limited after the single-node constraints are cleared. Profiling tells you whether more capacity would help before you commit the spend, which is the whole cost argument. The ROI is concrete: profiling-confirmed tuning of batching and KV-cache behaviour typically shifts sustained throughput by a large multiple versus default settings (observed pattern across serving engagements, not a published benchmark — the exact factor depends on traffic shape and model variant). More importantly, it answers the capacity question honestly. Deciding whether to buy another node is a spend decision, and our DeepSeek-on-H100 inference cost analysis shows how quickly that decision compounds. The engineering work behind all of this is what a GPU performance audit formalises — profiling the specific deployment, not the framework in the abstract, and delivering a ranked optimisation roadmap for the constraint that is actually binding. FAQ How does sglang/deepseek work? SGLang is a serving engine that runs a model like DeepSeek behind an inference API, applying RadixAttention prefix caching, continuous batching, and structured decoding to raise throughput. In practice it means the raw model is wrapped in a scheduler that decides how requests are batched and which KV-cache state is reused. The important consequence is that SGLang’s reported metrics describe that scheduler’s behaviour, not the kernel-level cost of serving your specific traffic. What does SGLang actually accelerate when serving DeepSeek models, and where does each gain come from? Three distinct mechanisms. RadixAttention caches and reuses the KV-cache for shared prompt prefixes, so its gain is proportional to prefix overlap in your traffic. Continuous batching admits new requests into the running batch during decode, raising per-GPU throughput when request lengths vary. Structured decoding constrains output to a grammar or schema — a correctness and latency feature, not a raw-throughput lever. Which one helps depends entirely on your traffic shape. Why can SGLang’s reported throughput or GPU utilisation look high while end-to-end cost per token stays worse than expected? GPU utilisation only measures the fraction of time a kernel was resident on the device; it does not distinguish compute-bound work from memory-bound stalls or KV-cache shuffling. A device can sit near 95% utilisation while compute units are mostly idle waiting on HBM. Cost per token — tokens-per-second per GPU — is the outcome you actually pay for, and it can worsen even as utilisation rises, for example when extra concurrent requests thrash the KV-cache. How do I profile an SGLang/DeepSeek deployment to tell whether the bottleneck is prefill compute, KV-cache memory, or host-side scheduling? Use Nsight Systems first for the timeline view: GPU idle gaps between kernels point to host-side scheduling or transfer stalls. If the GPU is genuinely saturated, drop into Nsight Compute on the hottest kernels to see whether they are compute-bound (prefill) or memory-bound (KV-cache). Profile under representative traffic that matches production prefix overlap and length distribution, and correlate with SGLang’s batch-size and cache-hit metrics as context. When is the SGLang serving bottleneck outside the kernel — in batching policy, request scheduling, or host-to-device transfer? Whenever the GPU timeline shows idle gaps with no kernel running. Those gaps mean the device is being starved by CPU-side tokenisation, request scheduling, or PCIe transfers on the critical path. This host-bound case is invisible to any GPU utilisation metric and caps throughput regardless of how much GPU capacity you add, which is why it must be diagnosed and fixed before scaling out. How does DeepSeek’s mixture-of-experts architecture change what shows up as the bottleneck in a profiler trace? DeepSeek’s larger variants route each token to a subset of expert feed-forward blocks, so arithmetic intensity per token is lower than a dense model of the same size and memory traffic dominates. In a trace this typically appears as memory-bandwidth-bound gather/scatter kernels and uneven expert occupancy from routing load imbalance — not the compute-bound GEMMs a dense-model mental model expects. Reading the trace with a dense-model assumption misidentifies the constraint. Once profiling identifies the real constraint, what optimisation order delivers the largest cost reduction first? Fix host-bound scheduling and transfer first — it is usually the cheapest change and unblocks everything downstream. If prefill-bound with prefix overlap, confirm RadixAttention is hitting cache and consider chunked prefill; if KV-cache-memory-bound, quantise the cache, cap max context, and tighten admission control. Only after single-node constraints are cleared should you evaluate adding GPUs, because scaling out only helps once you are genuinely throughput-limited. Profiling does not make a deployment fast on its own; it tells you which lever is worth pulling and, just as usefully, which capacity purchase to defer. The trap worth naming is the utilisation-percentage stall — a serving stack that reads as “done” on the scheduler dashboard while the profiler shows a host-starved GPU idling between kernels. That failure class is exactly what a GPU performance audit is built to surface, one trace at a time.
