A team stands up DeepSeek-V3 on SGLang, watches the serving dashboard report a healthy throughput figure, sees GPU utilisation pinned near the top, and calls the deployment tuned. Then p95 latency comes back from the first pilot cohort and it is nowhere near the target. The dashboard said everything was fine. The users disagree. That gap is not a bug in SGLang. It is the difference between the metric the scheduler emits — how busy the engine is — and the metrics that explain what a given request actually paid for. DeepSeek-V3 is a large mixture-of-experts (MoE) model, and serving it well means reading the serving path layer by layer: RadixAttention prefix caching, continuous batching, tensor and expert parallelism placement, and KV-cache memory pressure. A single aggregate throughput number collapses all of those into one figure that can look good while a specific layer is quietly capping you. This article explains which SGLang-specific readings tell you where a DeepSeek-V3 bottleneck lives, and how to attribute p95 latency and cost-per-request to the layer that owns them rather than to “the model.” Why can SGLang report high throughput while p95 latency on DeepSeek-V3 is still bad? Throughput and tail latency measure different things, and on a continuously batched serving engine they can move in opposite directions. SGLang batches incoming requests dynamically: it packs many sequences into a single forward pass to keep the GPU fed. High aggregate token throughput means the engine is doing a lot of work per unit time. It says nothing about how long any individual request waited in the queue, how many decode steps it shared a crowded batch, or whether it landed behind a long-prefill request that stalled the schedule. A useful way to hold the distinction: throughput is a property of the engine; p95 latency is a property of a request’s journey through it. You can maximise the first while degrading the second — a fuller batch raises throughput but lengthens the per-step time every sequence in that batch experiences. When the dashboard number looks great and the tail is bad, the batch policy is usually the first place to look, not the model. This is the same reasoning we apply when mapping a serving path for performance: the aggregate number is downstream of a chain of stages, and you cannot fix a stage you have not isolated. Which parts of the SGLang serving path most affect DeepSeek-V3 latency and cost? Four layers dominate, and each fails in a recognisable way. Reading them separately is what turns “it’s fast enough” into a defensible per-layer attribution. RadixAttention prefix caching. SGLang’s RadixAttention reuses the KV cache of shared prompt prefixes across requests, so repeated system prompts or few-shot preambles do not get recomputed. The relevant reading is the prefix-cache hit rate. A high hit rate means prefill work is being amortised; a low one means every request is paying full prefill cost even when the traffic pattern should allow sharing. Continuous batching. The scheduler decides how many sequences run together and how prefill and decode interleave. Its knobs — maximum batch size, prefill/decode balance — set the throughput-versus-tail trade-off directly. Tensor and expert parallelism placement. DeepSeek-V3 is MoE, so on top of tensor parallelism (splitting a layer’s weights across GPUs) there is expert parallelism (placing different experts on different devices). Where the experts sit determines how much cross-device traffic a token’s routing decision generates. KV-cache memory pressure. The KV cache consumes GPU memory in proportion to concurrent sequences and their lengths. When it runs short of headroom, SGLang cannot admit more concurrent requests, so the effective batch size is capped by memory, not by compute. Quick reference: SGLang DeepSeek-V3 reading → likely bottleneck Symptom in the readings Likely bottleneck layer What it usually means High throughput, bad p95, batch full Continuous batching policy Batch is too aggressive for the tail SLO Low prefix-cache hit rate under repetitive traffic RadixAttention / prompt structure Prefixes not shareable or cache too small KV-cache near capacity, batch size capped below compute limit KV-cache memory headroom Memory, not GPU compute, is the ceiling High inter-GPU traffic, decode-step time dominated by comms Expert/tensor parallelism placement Expert layout generates excess all-to-all GPU utilisation high, per-kernel time surprising in profiler Kernel A specific op (attention, MoE gather) is inefficient The table is a starting hypothesis, not a verdict — the point of profiling is to confirm which row your deployment is actually in. Which profiler readings tell me whether a bottleneck is batching, KV-cache, kernel, or parallelism? SGLang’s own metrics tell you the scheduler is busy. They do not tell you why a decode step took as long as it did. To answer that you read two layers together: the serving-engine metrics SGLang exposes, and a kernel/occupancy profiler underneath. Start at the serving layer. Split latency into prefill and decode contributions. Prefill is compute-heavy and scales with prompt length; decode is memory-bandwidth-bound and runs one token at a time per sequence. If prefill dominates your p95 and your traffic has repeated prefixes, the RadixAttention hit rate is the reading to interrogate. If decode dominates, look at batch composition and KV-cache pressure. Then go under the engine. The GPU profiling methodology that governs kernel and occupancy analysis applies directly here: a profiler like NVIDIA Nsight Systems or Nsight Compute shows you per-kernel time, occupancy, and — critically for MoE — the all-to-all communication that expert routing triggers. If decode-step wall-clock time is dominated by NCCL collectives rather than matmul kernels, the bottleneck is expert-parallel placement, not the model’s arithmetic. The discriminating question at each layer: Batching — does per-step time rise with batch size while queue wait also rises? Then the batch policy is trading tail for throughput. KV-cache — is admitted concurrency capped below what compute could sustain? Then memory headroom is the ceiling, a classic case of a memory-bandwidth-bound serving path. Kernel — is a single op disproportionate in the kernel profiler relative to its FLOP share? Then a kernel is the target. Parallelism — is decode-step time dominated by inter-GPU collectives? Then expert/tensor placement owns the cost. How do I read prefix-cache hit rate and KV-cache headroom? These two readings explain most DeepSeek-V3 throughput surprises, and they are easy to misread in isolation. Prefix-cache hit rate under RadixAttention tells you how much prefill you are not paying for. In an observed pattern across serving-path work — not a benchmarked rate — traffic with long shared system prompts or few-shot templates can reach high hit rates, which collapses prefill cost per request; traffic with unique long prompts sits near zero, and every request eats full prefill. If your hit rate is low and you expected it to be high, the problem is often upstream: the prompt structure is preventing prefix sharing, or the radix cache is being evicted under memory pressure before it can be reused. That last clause is the link to the second reading. KV-cache headroom and prefix-cache effectiveness are coupled — a cache starved of memory evicts prefixes before they are reused, so a memory problem shows up first as a caching problem. Read them together. KV-cache headroom caps the concurrent sequences you can admit; when it is tight, SGLang throttles concurrency and your batch size never reaches the level your GPUs could compute. The measurable outcome you want is the concurrency at which admitted requests stop growing, and whether that ceiling is set by memory or by compute. For DeepSeek-V3 specifically, the KV cache is large because the model is large, so this ceiling arrives sooner than teams expect coming from a smaller dense model. Which raises the question worth its own section. How does profiling an MoE model like DeepSeek-V3 differ from profiling a dense model? A dense model activates all its weights for every token. An MoE model routes each token to a subset of experts, so the compute per token is smaller than the parameter count suggests — but the placement of experts across GPUs introduces a cost a dense model does not have: cross-device routing traffic. That changes what you profile. On a dense model, per-layer compute and KV-cache pressure explain most of the picture. On DeepSeek-V3 you add two MoE-specific readings. First, the expert-parallel communication: token routing generates all-to-all traffic between GPUs, and if experts are placed such that hot experts cluster on one device or routing crosses a slow interconnect hop, decode-step time balloons from communication, not compute. Second, load balance across experts: if traffic skews toward a few experts, those devices saturate while others idle, and aggregate utilisation looks healthy while the critical path is one overloaded GPU. The practical consequence: on an MoE model, GPU utilisation is an even weaker proxy for performance than usual, because a device can be busy handling routing collectives while doing little useful matmul. This is where reading SGLang’s numbers against a kernel profiler — separating collective time from compute time — earns its keep. The runtime-choice trade-offs of serving DeepSeek-V3 on SGLang and the runtime tuning walkthrough for DeepSeek-R1 both lean on this distinction; this article is the one that says which readings tell you the MoE layer is the problem. Worked example: attributing cost-per-request to a layer Assume a DeepSeek-V3 deployment where the SGLang dashboard reports strong aggregate throughput and near-full GPU utilisation, but p95 latency is roughly double the SLO. The illustrative reasoning path: Split latency by stage. Suppose decode contributes most of p95. Prefill is not the story; move on. Check batch composition. If per-step decode time rises sharply with batch size, and queue wait is also high, the batch policy is trading tail for throughput. Candidate fix: cap batch size or rebalance prefill/decode. Check KV-cache headroom. If admitted concurrency is capped well below the compute limit, memory — not the batch knob — is the ceiling. Candidate fix: reduce KV-cache footprint before adding GPUs. Check inter-GPU collectives in the kernel profiler. If decode-step wall-clock is dominated by all-to-all rather than matmul, the expert placement owns the cost. Adding GPUs without fixing placement changes nothing. The value of this sequence is not the individual fixes — it is that each candidate change is now attached to a reading, so you can predict whether it will move p95 before you spend on it. A GPU-count increase that changes nothing is the expensive failure mode; a batching or placement change grounded in a profiler reading is the cheap one. How does this profiler data become a before/after baseline? The readings above are exactly the inputs to a cost audit’s bottleneck map. Per-stage latency across prefill and decode, prefix-cache hit rate under RadixAttention, KV-cache headroom that caps batch size, and cost-per-request attributed to a serving layer — captured before a change and again after — are what turn a tuning claim into evidence. Our [AI Inference Cost Audit](Inference Cost-Cut Pack) applies these profiling instruments to an SGLang + DeepSeek-V3 serving path and places the readings against a bottleneck map, so a per-layer cost attribution is defensible rather than asserted. Whether a measured cost-per-request is worth acting on at all is a separate unit-economics judgement that sits alongside our broader R&D consulting work. FAQ What does working with sglang deepseek-v3 involve in practice? SGLang serves DeepSeek-V3 by continuously batching incoming requests, reusing shared prompt prefixes through RadixAttention, and distributing the model’s tensor and expert-parallel layers across GPUs. In practice it means DeepSeek-V3 inference runs through a multi-layer serving path where each layer — caching, batching, parallelism placement, KV-cache memory — can independently become the bottleneck. Which parts of the SGLang serving path most affect DeepSeek-V3 latency and cost? Four layers dominate: RadixAttention prefix caching (how much prefill is amortised), continuous batching (the throughput-versus-tail trade-off), tensor/expert parallelism placement (how much cross-GPU routing traffic tokens generate), and KV-cache memory pressure (what caps concurrent requests). Each fails in a recognisable way and must be read separately. Which profiler readings tell me whether a bottleneck is batching, KV-cache, kernel, or parallelism placement? Read the SGLang serving metrics and a kernel/occupancy profiler together. Rising per-step time with batch size points to batching; concurrency capped below the compute limit points to KV-cache memory; a single disproportionate op in the kernel profiler points to a kernel; decode time dominated by inter-GPU collectives points to expert/tensor placement. How do I read prefix-cache hit rate and KV-cache headroom to explain DeepSeek-V3 throughput on SGLang? Prefix-cache hit rate tells you how much prefill you are not paying for; a low rate under repetitive traffic usually means prompts are not shareable or the cache is being evicted. KV-cache headroom caps admitted concurrency, so a tight cache throttles batch size below the compute limit — and starves the prefix cache, so a memory problem often shows up first as a caching problem. Why can SGLang report high throughput while p95 latency on DeepSeek-V3 is still bad? Throughput is a property of the engine’s aggregate work rate; p95 latency is a property of an individual request’s journey through the queue and batch. A fuller batch raises throughput but lengthens the per-step time every sequence experiences, so the two can move in opposite directions — a great dashboard number can coexist with a bad tail. How does SGLang profiler data become the audit’s before/after baseline for a DeepSeek-V3 deployment? Per-stage latency, prefix-cache hit rate, KV-cache headroom, and cost-per-request attributed to a serving layer are captured before a change and again after. Placed against a bottleneck map, that pair of readings turns a tuning claim into evidence and shows whether a batching, memory, or placement change actually moved p95 — or whether an added GPU changed nothing. How does profiling an MoE model like DeepSeek-V3 differ from profiling a dense model on the same serving stack? An MoE model routes each token to a subset of experts, so it adds two readings a dense model lacks: cross-GPU routing traffic from expert placement, and load balance across experts. GPU utilisation is an even weaker proxy for performance on MoE, because a device can be busy handling routing collectives while doing little useful compute. The uncomfortable part is that none of these readings is optional if you want a cost claim to survive scrutiny. Profile your own SGLang DeepSeek-V3 serving path first — split prefill from decode, read the prefix-cache hit rate against KV-cache headroom, separate collective time from compute in a kernel profiler — and you will already know which layer to argue about before anyone quotes a throughput number at you.
