Serving DeepSeek-V3 on SGLang: What the Runtime Choice Costs and Saves

A team running DeepSeek-V3 hits a cost ceiling. The first instinct is almost always to reach for a smaller model or a managed API, because that is the lever most people understand. It is usually the wrong first move. Before you change the model, you should know what the serving runtime is doing — because with DeepSeek-V3, the runtime often determines most of the realised cost-per-request, not the model.

SGLang is one of the runtimes that makes this visible. Its RadixAttention prefix caching, continuous batching, and the way it handles DeepSeek-V3’s mixture-of-experts routing together move the numbers that actually appear on your bill: cost-per-request, p95 latency, GPU utilisation, and effective tokens-per-second under real concurrency. Understanding what SGLang changes tells you whether “swap DeepSeek-V3” is a measured decision or a guess dressed up as one.

Why “the model is too expensive” is usually a runtime statement in disguise

The reasoning that leads to a model swap is intuitive. Costs are high, DeepSeek-V3 is a large mixture-of-experts model, so the model must be the problem. Shrink the model, shrink the bill.

The problem is that cost-per-request is a system property, not a model property. A large model served with poor batch occupancy and a cold KV cache on every request can look catastrophically expensive, while the same model served with high prefix-cache reuse and dense batching can look ordinary. The model weights did not change. The serving path did.

This matters because a model-replacement project is expensive in ways that do not show up until later: re-running evaluations, re-validating quality on your actual traffic, rebuilding prompt templates, and often discovering that the smaller model needs more retries or more careful prompting to hold quality. If profiling shows the runtime was the bottleneck, you have avoided that entire project. That is the ROI anchor here — not a headline throughput number, but the model swap you did not have to run.

What does SGLang’s RadixAttention prefix caching change for cost-per-request?

RadixAttention is SGLang’s mechanism for reusing the KV cache across requests that share a prefix. It organises cached key/value tensors in a radix tree keyed on token sequences, so when a new request begins with tokens already computed for an earlier request, the runtime reuses that cached state instead of recomputing it.

For DeepSeek-V3 workloads this is directly a cost lever, because prefill — processing the input tokens before generation starts — is often the dominant compute cost per request. Consider a workload with a long shared system prompt: a RAG assistant, a code-completion service, or a customer-support bot where every request carries the same 2,000-token instruction block. Without prefix caching, every request pays full prefill for that block. With RadixAttention, the shared prefix is computed once and reused across the batch.

The metric that connects this to your bill is the prefix-cache hit rate. When the hit rate is high, effective prefill cost per request drops sharply, and cost-per-request follows it down. When the hit rate is low — highly varied prompts, no shared structure, cache evicted under memory pressure — RadixAttention gives you little, and the runtime is not your lever. This is the kind of before-and-after that a disciplined profiling pass surfaces; our walkthrough on serving DeepSeek-V3 with SGLang and what to profile covers the instrumentation in detail.

The following is an illustrative decomposition, not a benchmark — the point is the structure, not the specific numbers.

Worked example: where cost-per-request actually goes

Assume a DeepSeek-V3 serving path handling requests with a 2,000-token shared system prompt and roughly 300 tokens of variable user input, generating 400 output tokens. Assume prefill is the dominant compute term.

Configuration	Prefix-cache hit rate	Effective prefill work/request	Relative cost-per-request
No prefix reuse, low batch occupancy	~0%	Full 2,300-token prefill	Baseline (1.0×)
RadixAttention, warm shared prefix	high	~300-token prefill only	markedly lower
RadixAttention + dense continuous batching	high	~300-token prefill, amortised decode	lowest

Illustrative framing only (no leading slash on the numbers is deliberate — these are shapes, not measured rates). The takeaway is that the same model spans a wide cost band depending purely on runtime configuration. If your current deployment sits near the baseline row, the model is not your problem yet.

How does DeepSeek-V3’s MoE routing interact with continuous batching?

DeepSeek-V3 is a mixture-of-experts model: each token is routed to a subset of expert feed-forward networks rather than passing through every parameter. This is what lets a model with a large total parameter count keep per-token compute manageable — only the activated experts do work.

Continuous batching is SGLang’s scheduling strategy for keeping the GPU busy: instead of waiting for a whole batch to finish before starting the next, it admits and retires requests token-by-token, so a request that finishes early frees its slot immediately for a waiting one. The relevant metric is batch occupancy — how full the batch actually is over time.

The interaction between the two is where it gets interesting, and where naive intuition fails. You might assume MoE routing and batching are independent. They are not. When many requests are batched together, their tokens route to different experts, and the runtime must gather activations for whichever experts are hit. If batch occupancy is high and routing is reasonably balanced, the expert compute is amortised well across the batch. If occupancy is low — few concurrent requests, poor scheduling — you pay the overhead of the MoE machinery without the amortisation benefit, and effective tokens-per-second collapses even though the model is doing “less” compute per token on paper.

That is why MoE models are unusually sensitive to the runtime. A dense model degrades gracefully at low occupancy; an MoE model can degrade sharply, because the routing overhead is a larger share of a nearly-empty batch. The prefill/decode disaggregation SGLang supports is one further lever here, letting the two phases scale independently so neither starves the other.

When is a runtime change the right lever versus replacing DeepSeek-V3?

This is the decision the whole article is built toward. Use the rubric below before committing to either path.

Decision rubric: runtime tuning vs model replacement

Prefix-cache hit rate is low but the workload has shared structure → runtime lever. You are leaving reuse on the table; fix the runtime first.
Batch occupancy is low under real concurrency → runtime lever. Continuous batching and admission control can lift utilisation without touching the model.
p95 latency is dominated by queueing, not compute → runtime lever. The model finishes fast once it starts; the wait is scheduling.
Cache hit rate is already high, batch occupancy is high, GPU utilisation is high, and cost-per-request is still above budget → now the model is a candidate. You have exhausted the runtime, and a smaller or routed model becomes a measured decision.
Quality headroom exists and traffic is genuinely low-value → model routing may beat both, independent of runtime state.

The honest position is that both levers are legitimate. What is not legitimate is pulling the model lever before you have measured the runtime one, because you cannot know whether the swap will help. In practice, across cost-audit engagements we have run, the runtime lever is checked first precisely because it is cheaper to test and cheaper to reverse (observed pattern from consulting engagements, not a benchmarked rate). A model swap is a one-way door that costs weeks; a runtime configuration change is a Tuesday.

How does cost-per-token relate to cost-per-request once batching and caching are accounted for?

Cost-per-token is the metric everyone quotes, because it is the one API pricing pages advertise. It is also the metric that hides the levers that matter. Cost-per-token treats every token as identical and independent. Batching and caching make that false.

With RadixAttention, the tokens in a shared prefix are not paid for on every request — they are paid once and amortised. With continuous batching, the marginal cost of a token depends on how full the batch is when it runs. So the same nominal cost-per-token can produce wildly different cost-per-request depending on prefix structure and concurrency.

The operationally relevant number is cost-per-request under your real traffic distribution — not a synthetic single-request measurement. That means measuring with representative prompt-sharing and representative concurrency, then reading four numbers together: prefix-cache hit rate, batch occupancy, p95 latency, and GPU utilisation. Any one of them alone will mislead you. This is the same discipline we apply when comparing runtimes generally, and it connects to the broader question of whether SGLang for DeepSeek is an engineering task rather than a research question — the answer, once you can see the metrics, is that it is squarely engineering.

FAQ

How does sglang deepseek-v3 work?

SGLang is an inference serving runtime; DeepSeek-V3 is a large mixture-of-experts model. Running them together means SGLang handles the serving path — prefix caching via RadixAttention, continuous batching, and scheduling of DeepSeek-V3’s MoE routing. In practice, this pairing determines most of your realised cost-per-request and p95 latency, which is why the runtime configuration is worth profiling before you consider changing the model.

What does SGLang’s RadixAttention prefix caching change for cost-per-request when serving DeepSeek-V3?

RadixAttention reuses the KV cache across requests that share a token prefix, so a shared system prompt is computed once instead of on every request. Because prefill often dominates per-request compute, a high prefix-cache hit rate can sharply lower cost-per-request. When prompts are highly varied and the hit rate is low, the effect is small and the runtime is not your lever.

How does DeepSeek-V3’s mixture-of-experts routing interact with SGLang’s continuous batching?

MoE routing sends each token to a subset of experts, and continuous batching keeps the GPU busy by admitting and retiring requests token-by-token. When batch occupancy is high, expert compute is amortised well across the batch; when occupancy is low, you pay the MoE routing overhead without the amortisation, and effective tokens-per-second can collapse. MoE models are therefore unusually sensitive to batch occupancy compared with dense models.

When is a runtime change to SGLang the right lever versus replacing DeepSeek-V3 with a smaller model?

Fix the runtime first when prefix-cache hit rate or batch occupancy is low, or when p95 latency is dominated by queueing rather than compute. Consider a model swap only once cache hit rate, batch occupancy, and GPU utilisation are all high and cost-per-request is still above budget. A runtime change is cheap and reversible; a model swap is a multi-week, one-way door.

What metrics should we profile before and after moving to SGLang?

Read four numbers together: prefix-cache hit rate, batch occupancy, p95 latency, and GPU utilisation, all measured under your real traffic distribution rather than synthetic single-request tests. Any one metric alone will mislead you — high GPU utilisation with a cold cache still means expensive requests. The before-and-after delta on these four tells you whether the runtime was the bottleneck.

How does cost-per-token on DeepSeek-V3 relate to cost-per-request once SGLang batching and caching are accounted for?

Cost-per-token treats every token as identical and independent, which batching and caching make false. Shared-prefix tokens are amortised across requests, and a token’s marginal cost depends on how full the batch is when it runs. The operationally relevant figure is cost-per-request under your real prompt-sharing and concurrency, not a synthetic single-request cost-per-token.

The assumption worth testing before you ship

The uncertainty that remains is workload-specific: your prefix-sharing structure and your concurrency profile decide how much SGLang’s caching and batching can save you, and no article can measure those for you. What an article can do is stop you pulling the expensive lever first. Before recommending any model change, our [AI Inference Cost Audit](Inference Cost-Cut Pack) measures whether SGLang’s caching and batching on a DeepSeek-V3 path are the real bottleneck — because “the model is too expensive” is a hypothesis, and the runtime is where you test it. If you want to talk through where your serving path sits, that starts with a conversation about how we scope engagements.