A team searches “sglang deepseek-v3” and arrives carrying the wrong toolkit. They have spent months compressing models for edge targets — distilling for on-device inference, quantising for CoreML and ONNX Runtime — and they reach for the same instincts here. The reflex is understandable and, in this case, misleading. SGLang is not a portability target you compress toward. It is a datacenter-side serving framework whose whole job is to raise sustained throughput on GPUs you already own. DeepSeek-V3, at hundreds of billions of parameters, is served, not embedded — so the question that dominates edge work (“how far do I shrink this before it fits?”) never comes up. That confusion is worth naming precisely, because it quietly steers infrastructure decisions in the wrong direction. If you frame DeepSeek-V3 as a compression problem, you spend your effort on distillation pipelines and export formats that will never touch the deployed system. If you frame it as a serving problem, you spend that effort on KV-cache reuse, batching behaviour, and fleet sizing — which is where the actual cost lives. The two problems look adjacent from a distance. Up close, they share almost nothing. What does “sglang deepseek-v3” mean in practice? SGLang is an inference-serving stack. You point it at a model, it stands up an endpoint, and it schedules incoming requests across GPU replicas to maximise how many tokens per second the fleet sustains under real concurrency. DeepSeek-V3 is one of the models it serves particularly well, because DeepSeek-V3 is a mixture-of-experts (MoE) model whose serving profile rewards exactly the tricks SGLang specialises in. The practical meaning of the phrase, then, is a deployment decision: given a large MoE model, how do I serve it so that the GPUs run hot on useful work rather than idling between requests? That is a datacenter question. It sits on the same side of the boundary as picking HGX vs DGX for an inference deployment or reasoning about DeepSeek inference economics on H100. It has nothing to do with whether the model runs on a phone, because it will not — and no compression choice changes that. What does SGLang actually optimise? Two mechanisms carry most of the weight, and both are throughput levers, not accuracy or size levers. RadixAttention KV-cache reuse. In transformer inference, every generated token depends on the attention key/value state of all preceding tokens — the KV cache. Naive serving recomputes or re-stores that state per request. RadixAttention organises cached prefixes in a radix tree so that requests sharing a common prefix reuse the already-computed KV state instead of paying for it again. For workloads where many requests share a system prompt, a few-shot preamble, or a document context, this is the difference between doing the work once and doing it thousands of times. We cover the mechanism in more depth in how RadixAttention reuses KV cache to raise GPU utilisation. Continuous batching. Rather than waiting to assemble a fixed batch, SGLang admits and evicts requests from the running batch token by token. A request that finishes decoding frees its slot immediately, and a waiting request takes it. This keeps the GPU’s matrix engines fed during the decode phase, where per-request work is small and idle time is the enemy. The composed effect is straightforward to state and easy to over-claim, so state it carefully: for shared-prefix workloads, RadixAttention KV-cache reuse and continuous batching can raise sustained serving throughput several-fold over naive per-request serving, and lower time-to-first-token under concurrency (observed pattern across serving-optimisation work; the multiple depends heavily on prefix-sharing ratio and concurrency, and is not a single benchmarked number). The gain is real, but it is a property of your traffic shape, not a headline figure you can port between deployments. Quick answer: what SGLang changes and what it does not Question SGLang’s answer Not SGLang’s job More tokens/sec from the same GPUs? Yes — batching + KV reuse — Lower time-to-first-token under load? Yes — continuous batching — Make the model smaller for a phone? — Edge compression (distil/quantise) Change model accuracy? No Quantisation / distillation trades this Cross-platform export (CoreML, ONNX)? No Edge deployment toolchain The table is the whole boundary in miniature. Everything in the left column is a serving decision; everything in the right column is an edge decision. They do not overlap. Why is serving DeepSeek-V3 a different problem from edge compression? Here is the divergence point, and it is architectural rather than a matter of degree. Edge compression exists because a model has to fit somewhere small and run on hardware you do not control — a phone, a browser via WASM, an embedded NPU. The engineering question there is a cross-target one: how many platforms must I support, and at what point does distilling a smaller student model beat maintaining N quantised builds? That is a genuine trade-off with a crossover point. DeepSeek-V3 does not present that trade-off, because it never fits on the edge in the first place. A model of its scale lives on GPU replicas in a datacenter and is reached over an API. There is no CoreML export in its future, no ONNX Runtime target, no distillation-vs-quantisation crossover to compute. The “how many edge platforms before distillation wins?” calculation simply does not arise — the model lives on the server side of the line by construction. This is why importing edge instincts here wastes effort. If you have done serious multi-platform edge work, you know the discipline of machine-learning model metrics for multi-platform edge — accuracy retention across quantisation levels, per-target latency budgets, build-matrix maintenance. None of that governs a served MoE model. The governing metrics flip to tokens/sec, tokens/sec/$, time-to-first-token, and inter-token latency under concurrency. We see this misframing regularly when a team’s prior work was edge-heavy: the muscle memory says “shrink it,” and the first weeks go to compression experiments that produce a smaller model nobody will deploy, because the deployment was always going to be a served endpoint. When do distillation and quantisation actually apply? Quantisation is not entirely absent from server-side serving — weight-only quantisation (for example FP8 or 4-bit weights) is used to shrink a served model’s memory footprint per replica so more of it fits in HBM and more KV cache room remains. That is a legitimate serving-side use, and it is worth separating from the edge use of the same word. The distinction is the target: Edge quantisation targets a fixed device budget and a portability format. The question is fit and cross-platform reach. Server-side quantisation targets HBM economics on GPUs you own. The question is how many replicas you need and how much batch headroom each has. Distillation — training a smaller student to mimic a larger teacher — is an edge and cost play. For DeepSeek-V3 served at scale, you generally serve the real model; distilling it into something small enough for the edge is a different product decision, not a serving-optimisation step. So the honest answer to “distillation vs quantisation” for a served MoE model is: quantisation may matter for footprint, distillation usually does not enter the serving decision at all, and neither is the cross-target portability problem edge work makes them out to be. How does DeepSeek-V3’s MoE routing affect fleet sizing? Mixture-of-experts changes the memory-versus-compute picture in a way that matters for provisioning. In a dense model, every parameter participates in every token. In an MoE model like DeepSeek-V3, a router selects a small subset of experts per token, so the active compute per token is far smaller than the total parameter count suggests — but the total parameters still have to live in GPU memory, because any expert may be selected next. The consequence for sizing: your memory footprint per replica is dominated by the full parameter set (plus KV cache), while your compute demand scales with active experts and batch size. This decouples the two axes people usually conflate. You cannot size a DeepSeek-V3 fleet by peak FLOPS alone, and you cannot size it by parameter count alone. You size it by fitting the full weights plus a realistic KV-cache budget into aggregate HBM, then measuring sustained tokens/sec under your concurrency to find the tokens/sec/$ figure. Getting this framing right is what lets a team size a GPU fleet against a throughput target rather than over-provisioning replicas out of caution — the ROI is in not buying GPUs to cover a mis-modelled bottleneck. The mechanism behind DeepSeek’s routing and where its cost lands is worth reading alongside this: how DeepSeek inference works and the algorithmic choices that drive GPU cost. A worked sizing sketch (illustrative) Assume a serving target of a fixed sustained tokens/sec and a shared-prefix ratio typical of an agent workload with a large common system prompt. Fit test. Full DeepSeek-V3 weights (optionally FP8-quantised for footprint) plus a KV-cache reservation must fit across the replica’s GPUs’ aggregate HBM. This sets the minimum GPUs per replica. Throughput test. Measure sustained tokens/sec per replica under continuous batching at your real concurrency — not a single-request microbenchmark. Prefix-reuse test. Re-measure with RadixAttention on your actual prefix-sharing pattern. If your traffic shares long prefixes, this is where the several-fold headroom shows up; if it does not, the gain shrinks toward the batching-only figure. Fleet size. Replicas needed = target tokens/sec ÷ measured per-replica sustained tokens/sec, rounded up for headroom. Every number here is measured on your traffic, not read off a spec sheet. That is the point. How do I measure the gains rather than assume them? Do not trust the headline multiple; reproduce it on your workload. Run a load generator that reflects your real prompt distribution — including how much prefix is shared — at the concurrency you expect. Record sustained tokens/sec (not the first-second burst), time-to-first-token, and inter-token latency at the tail, not just the mean. Then toggle RadixAttention and batching settings and re-measure. If your prefix-sharing ratio is low, expect a smaller gain; that is not a failure of the framework, it is the framework correctly reflecting your traffic. For a profiling-first walk through this, see what profiling reveals about real DeepSeek serving bottlenecks, and for splitting the two phases apart, SGLang PD disaggregation for prefill and decode. If you want an outside read on whether your inference pipeline is actually a server-side serving problem or a multi-platform edge one, that is precisely the line an inference-optimisation assessment on the GPU practice is built to draw — it evaluates the pipeline against its real deployment surface before any optimisation effort is committed. FAQ What should you know about sglang deepseek-v3 in practice? SGLang is a datacenter-side serving framework that stands up an inference endpoint and schedules requests across GPU replicas to maximise sustained throughput. Serving DeepSeek-V3 with it means a deployment decision — how to keep GPUs busy on useful work under concurrency — not a compression or portability decision. The model is served over an API, not embedded on a device. What does SGLang actually optimise — how do RadixAttention KV-cache reuse and continuous batching improve DeepSeek-V3 throughput? RadixAttention organises cached KV prefixes in a radix tree so requests sharing a prefix reuse already-computed attention state instead of recomputing it. Continuous batching admits and evicts requests token by token, keeping the GPU’s engines fed during decode. Together, for shared-prefix workloads, they can raise sustained throughput several-fold over naive per-request serving and lower time-to-first-token under concurrency (observed pattern; the multiple depends on prefix-sharing ratio and concurrency). Why is serving DeepSeek-V3 with SGLang a different problem from compressing a model for multi-platform edge targets? Edge compression exists to make a model fit and run on hardware you do not control, with a cross-target trade-off between distillation and per-platform quantised builds. DeepSeek-V3 never fits on the edge — at hundreds of billions of parameters it lives on datacenter GPUs and is reached over an API. So the “how many edge platforms before distillation wins?” calculation does not arise; the model is server-side by construction. When do edge compression choices like distillation vs quantisation apply, and when are they simply irrelevant to a served MoE model? Distillation and edge quantisation target fixed device budgets and portability formats — irrelevant to a served MoE model. Weight-only quantisation can still matter server-side, but for a different reason: shrinking memory footprint per replica so more fits in HBM. Distillation generally does not enter the serving decision at all; you serve the real model. How does DeepSeek-V3’s mixture-of-experts routing affect memory footprint and GPU fleet sizing under SGLang? MoE routing means only a subset of experts is active per token, so active compute is far smaller than the total parameter count suggests — but all parameters must still live in GPU memory since any expert may be selected. This decouples memory footprint (dominated by full weights plus KV cache) from compute demand (active experts × batch). You size the fleet by fitting weights plus a realistic KV budget into HBM, then measuring sustained tokens/sec. What throughput and time-to-first-token gains should I expect from SGLang for shared-prefix workloads, and how do I measure them? Expect meaningful gains when prefixes are heavily shared, and smaller gains when they are not — the framework reflects your traffic honestly. Measure by running a load generator matching your real prompt distribution and concurrency, recording sustained tokens/sec, time-to-first-token, and tail inter-token latency, then toggling RadixAttention and batching to compare. Do not port a headline multiple between deployments. The sharpest question to leave with is not “which compression path do I take?” but “which side of the boundary is my model on?” For DeepSeek-V3 the answer is fixed by its scale — it is served — and once that is settled, every downstream decision follows from tokens/sec/$ under real load rather than from a device budget that was never going to apply.
