Pick SGLang to serve a DeepSeek model and the tempting move is to accept the framework’s throughput headline and call the cost problem solved. That headline was measured on someone else’s traffic. Whether it holds on yours depends on the shape of your requests — and the only way to find out is to profile the serving path stage by stage, before and after the switch. That is the whole argument of this article. SGLang is not a free win or a free loss; it is a serving-path decision whose effects — RadixAttention prefix caching, continuous batching, scheduler behaviour — are conditional on workload shape. A profiler resolves which case you are in by showing cache hit rate, batch fill, and where p95 latency actually lands. Skip that step and an SGLang-plus-DeepSeek migration is a bet on a vendor benchmark, not a measured decision. What does “SGLang DeepSeek” actually mean in practice? The phrase describes a serving configuration: a DeepSeek model (V3, R1, or a distilled variant) running behind the SGLang runtime rather than a plainer inference server. SGLang contributes two mechanisms that matter for cost and latency, plus a scheduler that ties them together. The first is RadixAttention — a prefix cache that stores the key/value tensors for token sequences it has already processed and reuses them when a new request shares a leading prefix. If many requests begin with the same long system prompt or the same few-shot preamble, SGLang computes that prefix once and serves the cached KV state to every request that matches. The second is continuous batching, where the scheduler admits and retires requests token-by-token rather than waiting for a fixed batch to fill and drain. A request that finishes decoding early frees its slot immediately, so the GPU keeps working on live sequences instead of idling on stragglers. Both are real engineering wins under the right traffic. Neither is unconditional. That conditionality is the entire reason profiling is not optional here — it is how you separate “SGLang helped” from “SGLang was neutral” from “SGLang added overhead for no return.” What does RadixAttention change in the DeepSeek serving path, and how do you see it? Prefix caching attacks the prefill stage — the compute-heavy pass that processes the input tokens before the first output token is generated. On DeepSeek models, which are large and expensive per prefill token, removing redundant prefill work is where the biggest cost swings live. The metric that tells you whether it is working is prefix-cache hit rate: the fraction of input tokens served from cached KV state rather than recomputed. When a shared system prompt of a few thousand tokens is common across requests, the hit rate on those tokens climbs and the prefill compute they would have cost simply disappears. In configurations we have profiled, a high shared-prefix hit rate can remove a substantial share of prefill FLOPs — an observed-pattern claim, not a benchmarked constant, because the share depends entirely on how much of your input is genuinely shared. The trap is reading the aggregate hit rate without asking which tokens are hitting. A deployment can show a respectable overall hit rate that is entirely carried by a small warm set of prompts while the long tail — unique user documents, one-off contexts — hits nothing. For that tail, RadixAttention adds cache lookup and memory pressure with no prefill saving to offset it. A profiler that breaks hit rate down by prompt family, rather than reporting one number, is what surfaces this. The technique for measuring whether the cache actually moves GPU work is the same GPU profiling methodology you would apply to any kernel-level question about where compute goes. How do continuous batching and the scheduler show up in a latency breakdown? Continuous batching improves throughput by keeping the batch full, but its effect is visible only when you split latency by stage rather than staring at an average. The metric to watch is continuous-batch fill ratio — how full the running batch is, on average, across the decode loop. Under high, steady concurrency the fill ratio stays high, GPU utilisation stays high, and per-request cost falls because the fixed cost of a decode step is amortised across many sequences. Under bursty or low-concurrency traffic the fill ratio sags: the scheduler has few requests to pack, the GPU processes near-empty batches, and the throughput advantage evaporates. Average throughput can still look acceptable in that regime because the busy moments mask the idle ones. That is exactly why average is the wrong lens. Here the scheduler’s behaviour leaks into tail latency. When a long-running request holds a batch slot, shorter requests queued behind it wait, and p95 — the 95th-percentile latency — rises even while the mean stays flat. A per-stage breakdown separates queueing delay, prefill time, and decode time so you can see whether p95 is a scheduling artefact or a genuine compute cost. Splitting the two stages is also the lever behind prefill/decode disaggregation, which SGLang can apply once profiling shows the two stages contend for the same GPU. For which DeepSeek traffic shapes does SGLang cut cost, and for which does it not? This is the decision the whole exercise turns on. The divergence point is workload shape, and it is legible enough to tabulate. SGLang-for-DeepSeek fit by traffic shape Traffic characteristic Prefix-cache effect Batch-fill effect Net expectation Long shared system prompt, high concurrency High hit rate, prefill cost drops sharply High fill, cost amortised Strong win — profile to quantify Many similar few-shot prompts, steady load Moderate-to-high hit rate High fill Win — confirm hit rate holds under production prompts Unique long contexts (RAG docs), high concurrency Low hit rate, little prefill saving High fill still helps decode Mixed — batching helps, caching mostly does not Bursty, low-concurrency, low prompt reuse Low hit rate Low fill, GPU idles between bursts Marginal or negative — overhead without return Long-tail one-off requests Near-zero hit rate Depends on arrival pattern Likely negative on cache; profile before committing The table is a hypothesis generator, not a verdict. It tells you what to expect and therefore what to measure; the profiler tells you which row your traffic actually sits in. We have seen deployments assume they were in the top row — because the demo used a fixed system prompt — and land in the fourth once real, varied user traffic arrived. How do you profile an SGLang-plus-DeepSeek deployment to prove the migration was warranted? The migration is justified by a before/after comparison on the same traffic, not by a framework’s published number. Run the DeepSeek serving path under a representative replay of production requests, capture the same metrics on the plain runtime and on SGLang, and attribute the difference. Per-stage profiling checklist for an SGLang migration Prefix-cache hit rate, broken down by prompt family — not a single aggregate. This is the direct readout on whether RadixAttention removes prefill work for your prompts. Continuous-batch fill ratio across the decode loop, under the concurrency profile you actually serve, not a synthetic max-load run. p50 and p95 latency by stage — queueing, prefill, decode — so tail behaviour is separated from the mean. GPU utilisation during steady state and during bursts, to catch the idle-between-bursts pattern the average hides. Cost-per-request attribution that folds hit rate, fill ratio, and GPU-hours into one KPI you can defend to whoever signs the cloud bill. Metrics 1 through 4 are the mechanism; metric 5 is the answer. Converting per-stage profiler output into a cost-per-request number is the step that turns instrumentation into a business decision — the same unit-economics framework that governs any inference-cost comparison across runtimes or clouds. This before/after profile is precisely what an [AI Inference Cost Audit](Inference Cost-Cut Pack) produces when SGLang-with-DeepSeek is one of the serving-path configurations on the table. How does SGLang compare to a plain runtime when you look at p95 rather than throughput? Average throughput flatters continuous batching, because throughput is a sum over the batch and hides how any single request fared. p95 is a distributional statistic — it tells you what your worst-served users experience — and it can move in the opposite direction from throughput. A concrete pattern: SGLang lifts aggregate tokens-per-second on a high-concurrency DeepSeek workload, and simultaneously p95 rises for short requests because the scheduler let a long generation hold a slot. The throughput number says “faster”; the p95 number says “some users wait longer.” Both are true. Which one matters depends on your SLO, and only a stage-split latency profile lets you decide with the actual numbers in front of you rather than a single headline. The broader question of runtime-versus-runtime for DeepSeek — treated as an engineering measurement rather than a research debate — is where SGLang for DeepSeek as an engineering task picks up. FAQ How does sglang deepseek actually work? It means running a DeepSeek model behind the SGLang runtime, which adds RadixAttention prefix caching and continuous batching on top of a scheduler. In practice it is a serving-path decision: the two mechanisms cut cost and latency for some traffic shapes and add overhead for others, so the configuration only “works” once you have profiled it against your own requests. What does SGLang’s RadixAttention prefix caching change in the DeepSeek serving path, and how do you see it in a profiler? RadixAttention caches the key/value tensors for token prefixes it has already processed and reuses them when new requests share a leading prefix, cutting redundant prefill compute. You see it as the prefix-cache hit rate — and the useful view breaks that rate down by prompt family, because an aggregate number can hide a long tail of unique contexts that hit nothing. How do continuous batching and the SGLang scheduler show up in a per-stage latency breakdown? Continuous batching shows up as the batch fill ratio across the decode loop; high fill under steady concurrency amortises decode cost, while a sagging fill under bursty load signals GPU idling. The scheduler shows up in p95: a long request holding a batch slot delays shorter ones, so a stage-split breakdown of queueing, prefill, and decode is what separates a scheduling artefact from real compute cost. For which DeepSeek traffic shapes does SGLang cut cost per request, and for which does it not? It cuts cost for high-concurrency traffic with long shared prefixes — high hit rate and high batch fill both work in your favour. It is marginal or negative for bursty, low-concurrency traffic with little prompt reuse, where the cache rarely hits and the batch rarely fills. The fit-by-traffic table above is a hypothesis; the profiler confirms which row you actually occupy. How do I profile an SGLang-plus-DeepSeek deployment before and after to prove the migration was warranted? Replay representative production traffic against both the plain runtime and SGLang, and capture the same five metrics: prefix-cache hit rate by prompt family, batch fill ratio, p50/p95 latency by stage, GPU utilisation in steady state and bursts, and a cost-per-request attribution. The before/after delta on cost-per-request — not a vendor throughput headline — is the evidence that justifies the switch. How does SGLang compare to a plain runtime for DeepSeek when you look at p95 rather than average throughput? Throughput sums over the batch and hides individual request experience, so it flatters continuous batching. p95 tells you what your worst-served users see, and it can rise even as throughput improves when the scheduler lets a long generation hold a slot. Which number matters is an SLO decision, and only a stage-split latency profile lets you make it with real data. What cache-hit-rate and batch-fill metrics should I attribute cost-per-request to when serving DeepSeek on SGLang? Attribute cost to the prefix-cache hit rate (weighted by how much prefill compute each hit avoids) and the continuous-batch fill ratio (which sets how well decode-step cost is amortised), combined with GPU-hours consumed. Folding those into one cost-per-request KPI is what converts per-stage profiling into a number you can defend to whoever owns the infrastructure spend. The honest closing position is that “should we serve DeepSeek on SGLang?” is not answerable from the framework’s documentation. It is answerable from your traffic — the shared-prefix ratio, the concurrency distribution, the tail you are willing to tolerate. Name the failure class plainly: an unprofiled framework migration is an inherited benchmark masquerading as a decision. The [inference-cost-cut-pack](Inference Cost-Cut Pack) exists to replace that inherited number with a measured one.
