A team stands up DeepSeek on SGLang, the dashboard shows a respectable aggregate tokens-per-second, and yet p95 latency is bad enough that users complain. The instinct is to blame the model or the GPU. That instinct is almost always wrong, and acting on it — swapping the model, provisioning a bigger card — is the most expensive way to not fix the problem.

A high aggregate throughput number can coexist with poor tail latency. When it does, the cause is usually somewhere in how SGLang is scheduling and caching work, not in the weights or the silicon. The single tokens-per-second figure on the dashboard is an average over a request mix that hides exactly the thing you care about. To see what is actually happening you have to understand what the runtime is doing on the serving path, then read the layer that matches your symptom rather than trusting one aggregate.

What SGLang’s runtime actually does on the serving path

SGLang is a serving runtime, not a model. When you serve DeepSeek with it, three mechanisms dominate how requests turn into tokens, and each one has its own failure surface.

RadixAttention prefix caching. SGLang keeps a radix tree of previously computed key-value cache entries keyed by token prefix. When a new request shares a prefix with something already cached — a system prompt, a few-shot preamble, a shared document — the runtime reuses that KV state instead of recomputing it. This is the single largest lever on prefill cost for workloads with repeated structure. It also means your effective performance depends heavily on how similar your requests are to each other, which no synthetic benchmark with random prompts will show you.

Continuous batching. Rather than waiting to assemble a fixed batch, SGLang admits and retires requests token by token. A request that finishes decoding leaves the batch immediately; a new one joins on the next step. This keeps the GPU busy across heterogeneous request lengths, but it also means the batch composition is constantly shifting, and a few very long generations can occupy slots that shorter requests are queuing behind.

Prefill/decode scheduling. Prefill (processing the input prompt) and decode (generating output tokens one at a time) have completely different performance profiles. Prefill is compute-bound and processes many tokens in parallel; decode is memory-bandwidth-bound and processes one token per request per step. SGLang interleaves them, and how it prioritises prefill versus decode directly shapes whether new requests start quickly or existing ones finish quickly. This is the same prefill/decode distinction that motivates prefill/decode disaggregation as a latency lever — the two stages behave so differently that separating them is sometimes worth the complexity.

Once you hold these three mechanisms in view, “the throughput is bad” stops being a single question. It becomes: is prefill slow, is decode stalling, is the cache missing, or is the hardware genuinely saturated? Those are four different problems with four different fixes, and only one of them is “get a bigger GPU.”

Why can SGLang report high aggregate throughput while p95 latency is still bad?

Aggregate tokens-per-second sums output tokens across all concurrent requests and divides by wall-clock time. A batch of thirty requests each decoding steadily can produce an impressive number even while any individual request waits a long time between its own tokens. Throughput is a fleet metric; latency is a per-request metric. They are not the same axis, and optimising one can quietly degrade the other.

The most common mechanism behind the divergence is decode-stage contention. When continuous batching packs many requests into a decode step, each step still has to run every active request’s forward pass. Add more requests and the aggregate token count rises, but the time per decode step rises too — so each individual request’s inter-token latency grows even as the fleet number looks better. Push far enough and p95 blows out while the dashboard headline still looks healthy.

The second common mechanism is prefix cache behaviour. If your production traffic has low prefix reuse — every request is a fresh, unique prompt — RadixAttention has nothing to hit, so every request pays full prefill cost. That prefill work competes with decode for GPU time under SGLang’s scheduler, and the requests already mid-generation stall waiting for the scheduler to come back to them. The aggregate throughput may still be fine because prefill produces a lot of parallel token work; the tail is bad because real users are the ones stuck behind a prefill burst.

Quick answer: A high aggregate tokens-per-second on SGLang tells you the GPU is doing a lot of token work. It does not tell you that any single request is fast. When headline throughput looks good but p95 is bad, look first at decode-step time under load and at your prefix cache hit rate — not at the model.

Which profiler signals tell you where the bottleneck is?

Diagnosing an SGLang/DeepSeek path is a layered read: each candidate bottleneck has a signal that confirms or rules it out. In our inference-audit work the discipline is to move from the request layer down to the kernel layer, stopping as soon as the symptom is explained — this is an observed pattern across the DeepSeek serving paths we have profiled, not a fixed benchmarked recipe.

Suspected bottleneck	What to profile	Signal that confirms it	Typical fix
Prefill-bound	Per-request prefill vs decode time from SGLang request traces	Large prefill share of total request latency; p95 tracks input length	Raise prefix cache hit rate; consider prefill/decode disaggregation
Cache miss	RadixAttention hit rate from SGLang scheduler metrics	Low hit rate against traffic you expected to share prefixes	Restructure prompts to share stable prefixes; size the cache to fit the working set
Decode contention	Decode-step time as concurrency rises; batch occupancy	Inter-token latency grows with batch size while GPU util is high	Cap max running requests; tune batch/scheduler limits for tail SLO
Hardware-bound	GPU occupancy and memory-bandwidth utilisation during decode	Kernels near memory-bandwidth roofline; occupancy already high	This is the genuine “bigger/faster GPU” case — but only after the three above are excluded

The order matters. Reaching for a kernel timeline before you have looked at request traces is how teams end up optimising a matmul that was never the constraint. The GPU profiling methodology we apply to a DeepSeek decode path governs how the kernel- and occupancy-level tools plug into this same layered read, so that a Nsight Systems timeline is only consulted once the request-layer evidence points at the GPU.

Which NVIDIA tools map to which question?

The mapping is not one-tool-fits-all. SGLang’s own request traces and scheduler metrics answer the prefill-vs-decode and cache-hit questions — those live above the kernel layer and no NVIDIA profiler will surface them. For the hardware-bound question, Nsight Systems gives you the timeline view: whether the GPU is genuinely busy during decode or is idling between scheduler decisions. Nsight Compute drops to the kernel level for the specific decode kernels — useful only once you have established that decode kernels, and not scheduling gaps, are where the time goes. And nvidia-smi or DCGM occupancy sampling gives the coarse “is the card even saturated” read that tells you whether a kernel-level investigation is worth starting. The tools sit at different layers; using the wrong one for the question is how a profiling session produces confident numbers about the wrong subsystem.

How prefix cache hit rate affects cost-per-request

Cost-per-request on a DeepSeek serving path is dominated by GPU-seconds consumed, and prefill is where a lot of those seconds go for prompt-heavy workloads. Because RadixAttention lets a cache hit skip recomputing the shared prefix’s KV state, hit rate maps almost directly onto how much prefill compute you pay for.

Consider an illustrative case: a RAG-style workload where every request carries a 2,000-token system prompt plus retrieved context, and 500 tokens of that prefix are stable across requests. If the stable prefix is cached, those 500 tokens of prefill are skipped on every hit. Whether that is a large or small saving depends entirely on how much of your prefix is genuinely shared and how big your generations are — a short-prompt, long-generation workload gets almost nothing from prefix caching, while a long-prompt, short-answer classification workload can be transformed by it. The point is that the lever exists in the runtime configuration and the prompt design, not in the model weights. A correctly tuned SGLang configuration shifts cost-per-request without touching DeepSeek at all.

This is why we treat the SGLang numbers as inputs to a unit-economics view rather than an answer on their own. A measured hit rate or decode-step time is only actionable once it is placed against a cost-per-request target; the unit-economics framework that decides whether a serving number is worth acting on is what turns a profiler reading into a decision. Two teams can see the same hit rate and reach opposite conclusions because their cost structures and SLOs differ.

How these measurements become an audit’s before/after baseline

Read in isolation, a single SGLang metric invites a model swap. Placed in a bottleneck map, the same data points at scheduling or cache configuration you can fix without changing the model — which is a much cheaper intervention with a measurable before/after. The metrics that anchor that baseline are per-stage latency contribution (prefill vs decode), prefix cache hit rate, GPU occupancy during decode, and cost-per-request under a real batch mix rather than a synthetic one.

The reason the real batch mix matters is that every one of the four bottlenecks above is workload-dependent. A prompt distribution with high prefix reuse and short generations profiles nothing like a distribution of unique long-form prompts, even on identical hardware and model. An audit that baselines against synthetic traffic will read the wrong bottleneck map, which is why the [AI Inference Cost Audit](Inference Cost-Cut Pack) applies the profiling instruments to the deployed serving path under production traffic. Understanding how the SGLang/DeepSeek path behaves is what lets that map read correctly for this runtime — the runtime’s mechanisms are the lens through which the numbers become interpretable.

None of this replaces the audit; the numbers are inputs to its before/after baseline, not a substitute for it. What the runtime understanding buys you is the ability to bring your own profiler output and already know which layer you are looking at.

FAQ

How does sglang/deepseek work?

SGLang is a serving runtime that turns DeepSeek’s model weights into a request-handling system. In practice it means your observed performance depends as much on SGLang’s RadixAttention prefix caching, continuous batching, and prefill/decode scheduling as on the model itself — so a disappointing throughput or latency number is often a runtime-configuration problem, not a model problem.

What does SGLang’s runtime actually do on the serving path — RadixAttention prefix caching, continuous batching, and prefill/decode scheduling?

RadixAttention keeps a tree of previously computed key-value cache entries keyed by token prefix and reuses them when requests share a prefix, cutting prefill cost. Continuous batching admits and retires requests token by token so the GPU stays busy across heterogeneous request lengths. Prefill/decode scheduling interleaves compute-bound prompt processing with memory-bandwidth-bound token generation, and how it prioritises the two shapes whether requests start quickly or finish quickly.

Which profiler signals tell me whether an SGLang/DeepSeek bottleneck is prefill, decode, cache, or hardware?

Read the layers in order: SGLang request traces show the prefill-vs-decode split, scheduler metrics show the RadixAttention hit rate, decode-step time under rising concurrency shows contention, and only then do GPU occupancy and memory-bandwidth utilisation confirm a genuine hardware limit. Stop as soon as the symptom is explained — reaching for a kernel timeline before the request-layer evidence points at the GPU is how teams optimise the wrong subsystem.

Why can SGLang report high aggregate throughput while p95 latency is still bad?

Aggregate tokens-per-second is a fleet metric that sums output tokens across all concurrent requests, while p95 is a per-request metric. Decode-step time rises as continuous batching packs in more requests, so the fleet number can climb while any single request’s inter-token latency grows; low prefix cache reuse compounds this by forcing full prefill on every request, which stalls in-flight generations behind prefill bursts.

How does prefix cache hit rate affect cost-per-request when serving DeepSeek?

Cost-per-request is dominated by GPU-seconds, and prefill consumes many of them for prompt-heavy workloads. Because RadixAttention lets a cache hit skip recomputing the shared prefix’s KV state, hit rate maps almost directly onto prefill compute paid for — so long-prompt, short-answer workloads with high prefix reuse can cut cost-per-request substantially without touching the model, while short-prompt, long-generation workloads gain little.

Which NVIDIA profiling tools map to which questions about an SGLang decode path?

SGLang’s own request traces and scheduler metrics answer the prefill-vs-decode and cache-hit questions, which live above the kernel layer. Nsight Systems gives the timeline view of whether the GPU is genuinely busy during decode; Nsight Compute drops to specific decode kernels once you know decode is the constraint; and nvidia-smi or DCGM occupancy sampling gives the coarse saturation read that tells you whether a kernel-level investigation is worth starting.

How do these SGLang measurements become an audit’s before/after baseline?

The metrics that anchor the baseline are per-stage latency contribution, prefix cache hit rate, GPU occupancy during decode, and cost-per-request under a real batch mix. Read in isolation a single metric invites a model swap; placed in a bottleneck map, the same data points at scheduling or cache fixes with a measurable before/after — the numbers are inputs to the audit, not a substitute for it.

What the next review should check

The runtime is the interpreter between your traffic and your tokens, and most SGLang/DeepSeek performance surprises live in that interpreter rather than in the model or the card. Before you conclude that DeepSeek is too slow or your GPU is too small, the question worth answering is narrower and cheaper to test: under your actual production batch mix, which stage — prefill, decode, cache, or hardware — is the one carrying the tail latency? Answer that with a layered read of request traces and kernel timelines, and the fix usually turns out to be a scheduling or cache-configuration change you can make without changing anything about the model at all.