Spin up SGLang with default settings, watch the tokens-per-second counter climb, declare DeepSeek-V3 deployed. That is the reflex, and it is the wrong finish line. Peak throughput is not the number your finance team lives with, and it is not the number a customer feels when a response takes too long. The right question is narrower and harder: at the p95 latency your product actually promises, what does one DeepSeek-V3 request cost on SGLang — and does that cost survive translation into gross margin? A throughput headline does not survive that translation. A cost-per-request figure measured at a fixed latency does. This walkthrough shows how to set up the measurement so the result maps to a serving-config decision rather than a bragging-rights slide. How does SGLang DeepSeek-V3 serving work in practice? SGLang is an inference-serving runtime built around two ideas that matter here: RadixAttention, which caches and reuses the key-value state of shared prompt prefixes across requests, and continuous batching, which lets new requests join a running batch instead of waiting for the whole batch to drain. DeepSeek-V3 is a large mixture-of-experts model, so the serving story is dominated by how much of the model’s KV cache and expert activation you can keep resident in GPU memory (HBM) while still hitting your latency target. The naive mental model treats “serving DeepSeek-V3 on SGLang” as a config exercise — pick a checkpoint, set a batch size, done. In practice the runtime exposes a set of knobs — maximum running batch size, memory fraction reserved for the KV cache, tensor-parallel and expert-parallel degree, chunked-prefill size, scheduling policy — and each one trades latency against throughput against memory headroom. RadixAttention helps most when your traffic shares prefixes (system prompts, few-shot templates, RAG context reused across requests); it does little when every request is unique. Continuous batching raises utilisation but also raises tail latency, because a request that arrives mid-batch competes for decode steps with everything already in flight. None of this shows up in a peak tokens-per-second number. That number is measured with the batch saturated and latency ignored, which is precisely the operating point your SLO forbids. What serving parameters actually move cost-per-request? The parameters that change DeepSeek-V3’s cost-per-request on SGLang fall into three groups. Batching and concurrency controls (maximum running requests, chunked-prefill size) set how many requests share each GPU forward pass. Memory controls (KV-cache memory fraction, tensor/expert parallel degree) set how many concurrent sequences fit before the runtime starts rejecting or queuing. Decode controls (max new tokens, speculative decoding if enabled) set how long each request occupies the GPU. Here is the mechanism that trips people up. Raising the maximum running batch size raises throughput — more tokens per GPU-second — which lowers cost-per-token if you only look at the throughput plot. But a larger running batch lengthens the decode queue, which pushes p95 latency up. If your SLO caps p95 at, say, 800 ms to first useful response, the throughput-optimal batch size may sit well past that ceiling. The config that maximises tokens-per-second and the config that minimises cost-per-request at a fixed latency are usually not the same config, and the gap between them is money. Cost-per-request is derived, not measured directly: it is the hourly cost of the GPU instance divided by the requests-per-hour the config actually sustains at your latency target. That framing is the same one we lay out in spec-ing the compute behind a production AI feature, and it is what makes the number comparable across configs. How do you benchmark cost-per-request at a fixed p95 latency? The measurement protocol is what separates a defensible number from a headline. Fix the latency first, then sweep the config, then read cost off the sustained throughput. State the SLO. Pick the p95 latency your product requires — end-to-end, or time-to-first-token plus inter-token latency if you stream. This is an input, not a result. Fix the workload. Use a request trace that matches production: prompt-length distribution, output-length distribution, and prefix-sharing rate. A benchmark run on uniform 512-token prompts will overstate RadixAttention’s benefit if your real traffic is diverse. Structuring the run conditions so comparisons stay fair is the discipline covered in structuring serving configs so cost-per-request comparisons are fair. Sweep the candidate configs. Vary batch size and concurrency across a small grid. For each config, ramp offered load until p95 latency reaches your SLO ceiling, then record the sustained requests-per-second at that point. Derive cost. Divide the GPU instance’s hourly rate by requests-per-hour at the SLO. That is cost-per-request. Divide by mean output tokens for cost-per-token. Read the margin delta. Compare the cost-per-request of the throughput-optimal config against the cost-optimal-at-SLO config. The difference is your gross-margin delta on that feature. The per-config latency and GPU-utilisation numbers this protocol needs come from GPU profiling — the same measurement layer described in our GPU profiling for production inference work. Without accurate per-config utilisation you cannot tell whether a config is memory-bound, decode-bound, or simply queuing. Quick-answer block: throughput vs cost-per-request Peak tokens-per-second answers “how fast can this GPU go with latency ignored?” — a hardware ceiling, not an operating point. Cost-per-request at fixed p95 answers “what does one request cost at the latency my customers accept?” — the operating point your margin depends on. The two diverge because the batch size that maximises throughput usually violates the latency SLO. Optimising for the first silently degrades the second. Worked example: two SGLang DeepSeek-V3 configs at the same SLO The following table is illustrative — the shape of the reasoning, not a benchmark of your hardware. Assume a single GPU instance at a rate of $R per hour, an SLO of p95 ≤ 800 ms, and a production-like trace with moderate prefix sharing. The pattern shown here is what we consistently observe when tuning MoE serving configs; treat the ratios as directional, not as a published benchmark. Metric Config A (throughput-tuned) Config B (cost-tuned at SLO) Max running batch Large Moderate KV-cache mem fraction High Tuned to trace Peak tokens/sec Highest ~15–20% lower p95 latency at peak load Above SLO (SLO violated) At SLO ceiling Sustained req/s at 800 ms p95 Lower (must back off load) Higher Cost-per-request at SLO Higher Lower The counterintuitive result: Config A wins the throughput plot but loses on cost-per-request, because to actually hold the 800 ms SLO you have to reduce offered load until the queue drains — and at that reduced load the instance is underutilised. Config B, tuned so that the batch fills to the point where p95 just reaches the ceiling, sustains more paying requests per GPU-hour. The evidence class here is observed-pattern — a recurring shape across MoE serving engagements, not a single named benchmark result. Where does a throughput-optimal config stop mapping to margin? It stops the moment the throughput-optimal batch size pushes p95 past your SLO. Beyond that point, every additional token-per-second is bought at a latency you cannot ship, so it does not convert into servable requests. The throughput curve keeps climbing; the servable throughput curve has already flattened. Reporting the first while billing against the second is how a deployment looks efficient on a dashboard and bleeds margin in production. There is a second boundary worth naming. RadixAttention’s contribution to cost-per-request scales with your prefix-sharing rate. If a config was tuned on a synthetic trace with heavy shared prefixes and your production traffic is diverse, the cost-per-request you measured will not hold. Re-measure on a representative trace before trusting the number. This is the recurring failure we see teams walk into: a serving-config decision defended by a benchmark whose workload no longer matches reality. FAQ How does sglang deepseek-v3 work? SGLang serves DeepSeek-V3 — a large mixture-of-experts model — using RadixAttention to reuse shared prompt-prefix KV state and continuous batching to admit new requests into a running batch. In practice this means the deployment’s cost is governed by how many concurrent sequences fit in GPU memory while still meeting your latency target, not by the peak tokens-per-second the runtime can post. How do you set up SGLang to serve DeepSeek-V3, and what serving parameters affect cost-per-request? Setup involves choosing the checkpoint, tensor/expert-parallel degree, and the runtime’s batching, memory, and decode knobs. The parameters that move cost-per-request are maximum running batch size and chunked-prefill (batching), KV-cache memory fraction and parallel degree (memory), and max new tokens or speculative decoding (decode). Each trades latency against throughput against memory headroom. How do you benchmark DeepSeek-V3 on SGLang in cost-per-request at a fixed p95 latency instead of raw tokens-per-second? Fix the p95 latency SLO first, then fix a production-like request trace, then sweep candidate batch-size and concurrency configs. For each config, ramp offered load until p95 hits the SLO ceiling and record sustained requests-per-second there. Cost-per-request is the GPU instance’s hourly rate divided by requests-per-hour at that latency — a derived number, not a directly measured one. How do RadixAttention, continuous batching, and concurrency settings change cost-per-token for DeepSeek-V3 on SGLang? RadixAttention lowers cost when traffic shares prompt prefixes and does little when requests are unique. Continuous batching and larger concurrency raise GPU utilisation and throughput but lengthen the decode queue, pushing tail latency up. Cost-per-token improves with batching only up to the point where p95 latency stays inside the SLO; past that, extra tokens-per-second are not servable. What does a worked cost-per-request comparison between two SGLang DeepSeek-V3 configs look like? You compare a throughput-tuned config against a config tuned so the batch fills to exactly the SLO’s latency ceiling. The throughput-tuned config posts higher peak tokens-per-second but violates the SLO, forcing you to back off load until the instance is underutilised — raising cost-per-request. The SLO-tuned config sustains more paying requests per GPU-hour, so its cost-per-request is lower. How do you turn the SGLang DeepSeek-V3 benchmark into a defensible serving-config decision? Report cost-per-request and cost-per-token at the fixed p95 latency for each candidate config, plus the gross-margin delta between the throughput-optimal and cost-optimal configs. A decision backed by before/after cost-per-request numbers at a stated SLO survives finance and engineering review; a peak tokens-per-second figure does not. Where does a throughput-optimal SGLang config for DeepSeek-V3 stop mapping to margin? It stops the moment the throughput-optimal batch size pushes p95 latency past your SLO. Beyond that point additional throughput cannot be shipped, so it does not convert into servable requests. The second boundary is prefix-sharing rate: a config tuned on heavy shared prefixes will not hold its cost-per-request if production traffic is diverse. From walkthrough to your own stack This walkthrough shows the comparison method on a hypothetical instance. Running it against your deployed DeepSeek-V3 serving path — your GPUs, your trace, your SLO — is what produces the before/after cost-per-request that a serving-config decision can stand on. That is the shape of our Inference Cost-Cut Pack, and it is the everyday concern of the teams building AI-infrastructure and SaaS platforms. The failure to guard against is the throughput-headline deployment: a config that looks efficient on a dashboard, violates the latency it promised, and quietly carries a higher cost-per-request than a more conservative setup would have. Fix the latency first, and the number you get back is one you can put in front of finance.
