A GPU serving DeepSeek-R1 can read 92% busy on nvidia-smi and still waste most of the money you spent on it. That is not a paradox — it is the expected behaviour of a reasoning model whose inference is dominated by autoregressive decode, a phase that is memory-bandwidth bound rather than compute bound. The utilisation number you are watching measures whether the streaming multiprocessors are occupied, not whether they are doing useful arithmetic. For R1, those two things routinely diverge. That divergence is where R1 serving budgets leak. Teams provision capacity by watching a “busy” percentage, see it pinned near 100%, conclude they are fully loaded, and rent more GPUs to hit their throughput target. In practice the existing GPUs were never compute-limited — they were stalling on KV-cache reads, and adding hardware bought them more of the same stall. The right question is not how many GPUs you need. It is how much of the GPU you already own is doing useful work per request. Why is DeepSeek-R1 decode memory-bandwidth bound rather than compute bound? DeepSeek-R1 is a reasoning model. Before it produces a final answer it generates a long chain of intermediate “thinking” tokens — often many times the length of the visible response. Every one of those tokens is produced by an autoregressive decode step: the model reads the entire key/value cache for the sequence so far, computes attention and the feed-forward pass for exactly one new token, appends it, and repeats. The arithmetic intensity of that step is low. You move a large amount of data — the model weights for the layer plus the growing KV-cache — to produce a single token’s worth of compute. On modern accelerators the ratio of available FLOPs to available memory bandwidth is heavily skewed toward FLOPs, so a workload that reads a lot and computes little sits idle on the compute side while the memory subsystem is the actual bottleneck. This is the same structural reason single-token decode behaves differently from a big matrix multiply, and it is why what a 128GB GPU means in practice comes down to bandwidth rather than raw capacity. Prefill — the initial pass over the prompt — is the opposite. It processes many tokens in parallel, has high arithmetic intensity, and can genuinely saturate the compute cores. So R1 inference is really two workloads glued together: a compute-bound prefill and a bandwidth-bound decode, with decode dominating total time because reasoning traces are long. Treating the request as one homogeneous “inference” job hides the fact that most of the wall-clock time is spent in the bandwidth-limited phase. This is exactly the structural split that separating prefill and decode addresses at the serving-architecture level. Why can a GPU show high utilisation while still wasting capacity? The metric most teams call “GPU utilisation” — the number nvidia-smi reports, exposed as DCGM_FI_DEV_GPU_UTIL — is the fraction of time at least one kernel was resident on the device. It says nothing about how much of the compute capability that kernel used. A memory-bound decode kernel keeps the device “busy” for the entire duration it spends waiting on HBM reads; the SMs are occupied, the counter reads high, and the tensor cores are largely idle. To see the gap you have to look at a different set of counters: SM occupancy vs. achieved FLOP rate. High occupancy with a low achieved-to-peak FLOP ratio is the signature of a bandwidth-bound kernel. Memory bandwidth utilisation (DCGM_FI_PROF_DRAM_ACTIVE or Nsight’s DRAM throughput). During R1 decode this typically saturates well before FLOP utilisation does. Tensor-core active percentage. Often strikingly low during decode even when the coarse “GPU util” number is pinned high. The practical consequence is that the headline utilisation number is not a proxy for value. In configurations we have profiled, R1 decode sits near memory-bandwidth saturation while achieved compute utilisation stays a fraction of peak (observed pattern across GPU performance work, not a published benchmark). You are paying for FLOPs the workload never touches. This is the same reading discipline we apply across the three pillars of observability for GPU utilisation: the coarse gauge tells you something is running, not whether it is running well. Utilisation signals: what each one actually tells you Signal What it measures What it means for R1 decode GPU-busy % (GPU_UTIL) A kernel was resident Near 100% even when compute is idle — not a value signal SM occupancy Warps scheduled per SM Can be high while cores stall on memory Achieved FLOP rate vs peak Useful arithmetic done Usually low during decode — the real efficiency read DRAM active % Memory subsystem busy Saturates first; the true bottleneck for decode Tensor-core active % Matrix-unit utilisation Low during single-token decode The rule of thumb: if DRAM active is high and achieved FLOP rate is low, you are decode-bound, and buying more compute will not help. How do batch size and KV-cache management change effective throughput? Because decode is bandwidth-bound, the lever that moves throughput most is not FLOPs — it is amortising memory traffic across more work. Batching does exactly that. When you decode several sequences together, the model weights are read once from HBM and reused across the whole batch, so the per-token bandwidth cost drops sharply. This is why effective throughput per GPU for R1 can rise by a meaningful multiple purely from correct batch sizing, with no new hardware (observed pattern; the exact multiple depends on model size, sequence length, and available memory). The ceiling on batching is the KV-cache. Each active sequence holds a KV-cache proportional to its length, and R1’s long reasoning traces make those caches large. Batch too aggressively and you run out of HBM for the cache; batch too conservatively and you leave bandwidth amortisation on the table. The serving stack’s memory management is what decides where that ceiling sits: Paged KV-cache (the vLLM PagedAttention approach) stops fragmentation from wasting cache memory, letting you hold more concurrent sequences. Quantising the KV-cache to FP8 or lower roughly halves the bandwidth per read and frees capacity for larger batches, at a controllable accuracy cost. The same logic that makes 4-bit floating point cut inference cost on the weights applies to the cache. Prefix reuse via a radix cache avoids recomputing and re-storing KV for shared prompt prefixes, which matters when many R1 requests share a system prompt. Quantising the model weights themselves helps twice: it cuts the bandwidth cost of the per-layer weight read and frees HBM for a larger batch. Because decode is bandwidth-bound, precision reduction often buys more real throughput on R1 than it would on a compute-bound workload — the saving lands directly on the binding constraint. Frameworks like SGLang, vLLM, and TensorRT-LLM expose these knobs; the question is whether you have profiled which one is binding before you turn them. How do I measure cost per useful output token instead of cost per GPU-hour? The unit that matters for an R1 serving budget is cost per useful output token, not cost per purchased GPU-hour. GPU-hours are what you pay; output tokens are what you sell. The ratio between them is set by effective throughput at your target latency, and that is precisely the number the busy-percentage metric hides. A worked example, with the assumptions stated explicitly: Assumption: you rent a GPU at a fixed hourly rate (call it R), and you have a target time-to-first-token and inter-token latency your product requires. Measure: sustained output tokens per second at that latency target, under realistic concurrency — not a single-stream burst. Compute: cost per token = R / (3600 × tokens-per-second). Only the tokens delivered inside your latency SLA count as useful. Two GPUs with identical GPU-busy readings can differ by a large factor on this number, because one is running a well-batched, quantised-cache configuration and the other is decoding one sequence at a time. The busy percentage is identical; the cost per useful token is not. If you are modelling this before deployment, a token calculator that estimates inference cost gives you the token-volume side of the equation to pair with your measured throughput. Should I add GPUs, or profile the decode phase of what I have first? Profile first — almost always. Adding GPUs to a decode-bound deployment scales the bottleneck linearly with the bill: each new GPU stalls on memory exactly as the existing ones do, so you pay N times for a per-unit efficiency you never fixed. The comparison that matters is serving R1 at your latency target on optimised existing GPUs versus renting additional capacity to reach the same throughput. Use this order: Confirm you are decode-bound. Check DRAM-active vs achieved-FLOP during the decode phase. If DRAM is saturated and FLOP utilisation is low, more compute will not help. Fix batching. Raise batch size until either the KV-cache fills HBM or latency breaches your SLA. This is usually the single largest gain. Reclaim cache memory. Paged KV-cache, KV-cache quantisation, prefix reuse — each frees room for larger batches. Quantise weights if accuracy tolerance allows, cutting weight-read bandwidth and freeing HBM together. Only then consider scaling out, and scale by measured cost per useful token, not GPU count. Realistic gains from steps 2–4 on existing hardware are frequently a meaningful multiple of baseline throughput (observed pattern across GPU optimisation engagements; not a benchmarked rate — the ceiling depends on your model variant and sequence-length distribution). What profiling reveals about real R1 bottlenecks is the same story SGLang and DeepSeek serving profiling surfaces: the constraint is rarely where the busy gauge points. For the algorithmic side of why DeepSeek’s design drives cost the way it does, how DeepSeek inference works walks through the choices behind the numbers, and the hardware-specific view sits in DeepSeek on H100 inference cost. FAQ What matters most about DeepSeek-R1 inference in practice? R1 is a reasoning model that generates a long chain of intermediate tokens before its final answer. Each token comes from an autoregressive decode step that reads the full KV-cache and computes one token’s worth of work. In practice, most wall-clock time is spent in this decode phase, so decode behaviour — not prompt processing — dominates your serving cost. Why is DeepSeek-R1 decode memory-bandwidth bound rather than compute bound, and why does that matter for GPU cost? Single-token decode moves a lot of data (weights plus a growing KV-cache) to produce very little arithmetic, so the memory subsystem saturates while the compute cores idle. Because you pay for a GPU’s full compute capability regardless, a bandwidth-bound workload means you are billed for FLOPs the workload never uses. Correcting the bottleneck lands directly on cost. Why can a GPU show high utilisation while still wasting capacity during R1 inference? The GPU-busy percentage only reports that a kernel was resident, not that it used the compute cores. A memory-bound decode kernel keeps the device “busy” while waiting on HBM reads, pinning the gauge near 100% with tensor cores mostly idle. You need achieved-FLOP-versus-peak and DRAM-active counters to see the real efficiency. How do batch size and KV-cache management change effective throughput for R1 serving? Batching amortises the per-layer weight read across many sequences, cutting per-token bandwidth cost and often raising throughput per GPU by a meaningful multiple. The KV-cache size caps how far you can batch, so paged caches, KV-cache quantisation, and prefix reuse all matter because they free HBM for larger batches. How do I measure cost per useful output token for DeepSeek-R1 rather than cost per GPU-hour? Measure sustained output tokens per second at your target latency under realistic concurrency, then divide the hourly GPU rate by tokens delivered inside your SLA. Two GPUs with identical busy readings can differ by a large factor on this number because one is well-batched and one is not. GPU-hours are what you pay; output tokens are what you sell. Should I add GPUs to hit R1 throughput targets, or first profile the decode phase of what I have? Profile first. Adding GPUs to a decode-bound deployment scales the bottleneck linearly with the bill, since each new GPU stalls on memory the same way. Confirm you are decode-bound, fix batching, reclaim cache memory, quantise weights, and only then scale out — measured by cost per useful token, not GPU count. What realistic throughput gains come from optimising R1 inference on existing GPUs versus renting more capacity? Batching and cache/weight quantisation on existing hardware frequently deliver a meaningful multiple of baseline throughput (observed across optimisation engagements; not a benchmarked rate). The exact ceiling depends on your model variant and sequence-length distribution, but the comparison — optimise what you own versus rent to hit the same number — usually favours profiling first. A GPU Performance Audit makes this concrete: it isolates the decode-phase memory-bandwidth bottleneck that GPU-busy percentages hide and quantifies the wasted capacity per R1 request, so the “add hardware or fix the configuration” decision is made on measured cost per useful token rather than on a gauge that was never measuring value.
