A team has spent three weeks tuning the attention kernel for a DeepSeek serving stack. Occupancy is up, the profiler looks clean, and the serving bill has barely moved. That is not a mystery — it is the expected outcome when the cost profile is dominated by architecture, not kernels. The naive read of DeepSeek inference cost treats it as a kernel problem: optimise the attention path, push occupancy, and expect the bill to fall. That approach hits a ceiling quickly. DeepSeek’s cost profile is set by architectural decisions — mixture-of-experts (MoE) routing, multi-head latent attention (MLA), and how the KV cache is laid out and batched — long before register allocation or shared-memory tiling become the binding constraint. If you want tokens-per-second-per-GPU to move after the hot kernels are already tight, the remaining speedup lives in the algorithm, not the microarchitecture. What does DeepSeek inference actually do per token? DeepSeek’s published models (V2, V3, and the R1 reasoning line) share a design that departs from a dense transformer in two ways that matter for cost. Both change how much work each GPU does per generated token, which is the quantity that ultimately sets cost-per-million-tokens. The first is sparse mixture-of-experts. Instead of routing every token through one large dense feed-forward block, the model holds a large pool of expert sub-networks and activates only a small subset per token via a learned router. The total parameter count can be very large while the active parameter count per token is a fraction of it. That sparse activation is the whole point: it decouples model capacity from per-token compute. In our experience reasoning about serving stacks, this is the single biggest reason DeepSeek’s cost curve does not behave like a dense model of similar quality — the compute you pay for per token is set by the router, not the nominal parameter count. The second is multi-head latent attention. MLA compresses the key/value representation into a lower-rank latent before caching, which shrinks the KV-cache footprint per token substantially compared with storing full-width keys and values. For long-context serving, the KV cache — not the weights — is often what fills GPU memory and throttles the batch size you can hold. Shrinking it directly buys back the memory headroom that batching depends on. Neither of these is a kernel trick. They are structural properties of the model that determine the shape of the workload your kernels then run on. Why does mixture-of-experts routing change the GPU cost profile? A dense model has a predictable, uniform per-token compute cost: every token touches the same weights. MoE breaks that uniformity, and the break is where both the opportunity and the trap live. On the opportunity side, sparse activation reduces active parameters per token by a large multiple versus a dense model of equivalent quality (an architectural property of the design, not a benchmarked serving rate for any specific deployment). Fewer active parameters per token means less matrix-multiply work per token, which is the lever that moves throughput per GPU. On the trap side, routing introduces load imbalance. If a batch of tokens routes disproportionately to a handful of experts, those experts’ GPUs saturate while others idle — the aggregate FLOPs look low, the profiler shows underutilised SMs, and the naive conclusion is “the kernel is slow.” The kernel is fine. The routing distribution across the current batch is starving half the hardware. This is a scheduling and batching problem expressed as a utilisation number, and no amount of kernel tuning will fix it. Expert parallelism is the standard structural response: experts are sharded across GPUs, and tokens are dispatched to wherever their expert lives via an all-to-all communication step. That step — typically riding NCCL over NVLink or the inter-node fabric — becomes a first-class cost. Get the placement and the dispatch batching wrong and the all-to-all dominates; get it right and the sparse compute advantage actually materialises. The same reasoning about interconnect as a first-class cost appears when an 800G NIC decides whether interconnect is your bottleneck. Which parts are algorithmic levers versus kernel tuning? This is the distinction the GPU performance audit draws for every serving stack we look at: each intervention is either an algorithmic lever that changes the shape of the work, or a micro-level lever that makes a fixed shape run faster. The two have completely different ceilings, and confusing them is the most common way teams waste optimisation weeks. Lever Class What it changes Ceiling behaviour MoE routing / expert placement Algorithmic Active params per token; load balance across GPUs High — reshapes the workload KV-cache layout (MLA, paging) Algorithmic Memory per token, achievable batch size High — unlocks batching Batching / scheduling strategy Algorithmic Concurrent sequences, expert co-location High — raises occupancy Attention kernel fusion Micro-level Cost of a fixed attention op Low once tuned Occupancy / register tuning Micro-level SM utilisation of a fixed kernel Low once tuned Precision (FP8/FP4 GEMM) Both Compute + memory per op and batch headroom Medium — bounded by accuracy The reading is simple. Micro-level levers have a floor: once a FlashAttention-style kernel is fused and occupancy is high, you are near the hardware limit for that operation and marginal gains collapse. Algorithmic levers change what the operation is. Precision sits across both because moving a GEMM to FP8 both cuts per-op cost (micro) and frees KV-cache and weight memory to enlarge the batch (algorithmic) — which is why we treat it as a trade-off bounded by accuracy rather than a free win. How does batching interact with expert activation and occupancy? Batching in a dense model is close to linear: more concurrent sequences, more parallel work, higher occupancy, up to memory limits. In an MoE model the interaction is coupled, and the coupling is the part teams miss. Consider a worked example with explicit assumptions. Suppose a serving instance holds 32 concurrent sequences, the model has 256 experts, and each token activates 8. Across a decode step you are dispatching a few hundred token-expert assignments. If those assignments spread evenly, every expert shard does comparable work and occupancy is high. If the current batch’s prompts happen to concentrate on a subset of experts — a realistic pattern for domain-clustered traffic — a minority of shards carry most of the load and the rest idle waiting on the all-to-all barrier. The instance’s measured throughput drops even though nothing about the kernels changed. The practical consequence: batching strategy for MoE is not just “how many sequences fit in memory.” It is also “does this batch produce a routing distribution that keeps the expert shards balanced.” Larger batches statistically smooth the routing distribution, which is one reason MoE serving rewards aggressive continuous batching — but only up to the KV-cache memory ceiling, which is exactly what MLA and paged KV layout push outward. The KV-cache footprint and the batch size are the two dials, and they are linked. Runtimes like SGLang and vLLM expose these as scheduling and cache-management policies rather than kernel flags, which is a tell that the leverage is algorithmic; the profiling angle on this is covered in what profiling reveals about real DeepSeek inference bottlenecks. How do I tell a DeepSeek stack has hit its kernel-tuning ceiling? There is a recognisable signature. Use it as a diagnostic before committing another sprint to micro-optimisation. The hot kernels are already fused and tuned. Attention runs a FlashAttention-class kernel, GEMMs use the vendor library (cuBLAS/CUTLASS) or a tuned TensorRT-LLM path, and the profiler shows high SM occupancy on those kernels in isolation. Marginal kernel gains have collapsed. Each new tuning pass buys single-digit-percent improvements or less, and the wins do not survive to end-to-end throughput. End-to-end GPU utilisation is still low. Aggregate SM utilisation across the instance sits well below what the isolated kernels achieve — a gap that points at scheduling, not compute. Utilisation is bursty and expert-correlated. Idle periods line up with all-to-all barriers or with batches that route unevenly, not with compute-bound stretches. Cost-per-million-tokens is flat despite kernel work. The metric that pays the bill has not moved even as the microbenchmarks improved. When most of these hold, the ceiling is architectural. The remaining speedup is in routing balance, KV-cache layout, batching policy, and precision — not in the kernels. This is the same classification logic behind DeepSeek-R1 inference GPU utilisation in practice, where the reasoning-token workload stresses the KV cache harder than a chat workload does. What structural changes lower cost-per-million-tokens without degrading quality? The levers that move the bill are the ones that change work per token or GPUs kept busy, applied in roughly descending order of typical impact and applied only after the kernel ceiling is confirmed: Fix routing balance first. Expert-parallel placement and dispatch batching that keep shards evenly loaded recover the utilisation the sparse design promised. This is where underused GPUs most often hide. Shrink the KV cache. MLA is built in, but paged KV management and eviction policy determine how much of that headroom actually converts to a larger serving batch. Tune batching to the routing distribution, not just to memory. Continuous batching that smooths expert load raises occupancy without touching a kernel. Apply precision deliberately. FP8 and FP4 GEMM paths cut both compute and memory, enlarging the batch — bounded by accuracy validation on your workload, which makes this a first-class trade-off rather than a default. Each of these can raise GPU occupancy enough to change the serving-instance count needed for a target QPS, which is the outcome that shows up on the invoice. The one guardrail: every structural change must be validated against output quality, because unlike kernel tuning — which is quality-neutral by construction — routing, KV, and precision changes can move accuracy if applied without measurement. FAQ How does DeepSeek inference actually work? DeepSeek activates only a small subset of expert sub-networks per token via a learned router (mixture-of-experts) and compresses its key/value cache with multi-head latent attention. In practice this means per-token compute is set by the router’s active-parameter count and per-token memory by the compressed KV cache — not by the model’s nominal size. Those two properties, not kernel efficiency, dominate the serving cost. Why does DeepSeek’s mixture-of-experts routing change the GPU cost profile compared with a dense model? A dense model runs every token through the same weights, giving uniform per-token cost. MoE activates only a few experts per token, cutting active parameters per token by a large multiple versus a dense model of equivalent quality — but it introduces load imbalance across the GPUs that hold the experts. That imbalance shows up as low aggregate utilisation that kernel tuning cannot fix, because the cause is routing distribution, not kernel speed. Which parts of DeepSeek inference are algorithmic levers versus kernel-level tuning? Algorithmic levers are MoE routing and expert placement, KV-cache layout (MLA and paging), and batching/scheduling strategy — they change the shape of the work. Kernel-level levers are attention fusion and occupancy tuning, which make a fixed operation faster. Precision (FP8/FP4) straddles both. Algorithmic levers have a high ceiling; micro-level levers flatten out once the kernels are tuned. How does batching strategy interact with expert activation and GPU occupancy during DeepSeek serving? In an MoE model, batching is coupled to routing: the number of concurrent sequences sets memory pressure, but the batch’s routing distribution sets whether expert shards are evenly loaded. Uneven routing idles shards behind the all-to-all barrier and drops measured throughput without any kernel change. Larger batches statistically smooth routing, so MoE serving rewards continuous batching up to the KV-cache memory ceiling. When has a DeepSeek serving stack hit its kernel-tuning ceiling, and how do I tell? The signature is: hot kernels already fused and tuned, marginal kernel gains collapsed to single digits, end-to-end GPU utilisation still low, idle periods correlated with all-to-all barriers or uneven routing, and cost-per-million-tokens flat despite kernel work. When most of these hold, the remaining speedup is architectural — routing balance, KV layout, batching, precision — not kernel-level. What structural changes typically lower cost-per-million-tokens for DeepSeek inference without degrading quality? Fix routing balance and expert placement first, then improve paged KV-cache management to enlarge the batch, then tune batching to smooth the routing distribution, and apply FP8/FP4 precision where accuracy allows. Each raises occupancy enough to potentially reduce the serving-instance count for a target QPS. Unlike kernel tuning, these changes can move accuracy, so each must be validated against output quality. When a DeepSeek stack stalls after the kernels are already tight, the question is not “which kernel is next” but “is the workload’s shape still leaving GPUs idle at load?” That is the algorithmic-versus-micro-level classification the GPU performance audit applies deliberately — and for a mixture-of-experts serving stack, the entries that matter most are the ones the profiler never labels: routing balance, KV-cache layout, and the batching policy that ties them together.
