“How many H100s do we need to serve DeepSeek?” is the wrong first question. It treats inference as a hardware sizing exercise: pick the GPU, load the weights, measure tokens per second, divide by the target throughput. That framing quietly buries the more expensive decision inside a spreadsheet cell — because DeepSeek’s inference performance on an NVIDIA H100 is not a property of the model alone. It is a property of the model plus the CUDA-tuned kernels, FP8 execution paths, and KV-cache memory layouts that the published serving stack assumes. The moment you commit to an H100-class deployment on those numbers, you have also committed to a portability cost that never shows up on the invoice — until someone tries to move the workload off CUDA. That is the real subject here. The tokens-per-second figure is easy. The dependency structure underneath it is what determines whether your cost-per-inference number is defensible or fictional. What does “DeepSeek on H100” actually mean in practice? When people say “DeepSeek runs well on H100,” they are compressing several things into one sentence. There is the model architecture — a large Mixture-of-Experts (MoE) transformer where only a subset of expert parameters activate per token. There is the numerical format — much of DeepSeek’s efficiency story rests on FP8 execution, which the H100’s Hopper Tensor Cores support natively. And there is the serving software — the attention kernels, the expert-routing dispatch, the KV-cache management — most of which was written and tuned against CUDA, cuDNN, and NVIDIA-specific memory primitives. The performance you measure is the composite of all three. Swap any one and the number moves. This is the same warning the broader GPU engineering practice keeps returning to: the executor is the hardware and the software stack together, never the silicon in isolation. A benchmark of “DeepSeek on H100” is really a benchmark of “DeepSeek’s CUDA-tuned FP8 serving path on Hopper,” and the two are not interchangeable claims even though teams routinely treat them as one. The reason this matters for cost is that the efficient path and the portable path are usually different paths. The configuration that gets you the best tokens/sec per GPU is frequently the one most entangled with NVIDIA-specific code — and that entanglement is the hidden line item. What throughput and cost can DeepSeek realistically hit on an H100? Any concrete number here has to be bounded, because the honest answer is “it depends on the workload shape.” Sustained throughput for a large MoE model on an H100 depends on prompt-to-generation ratio, batch size, sequence length, and how well the router keeps expert utilisation balanced. A prefill-heavy workload (long prompts, short completions) stresses different resources than a decode-heavy one (short prompts, long generations). Quoting a single tokens/sec figure without stating the request profile is how people end up with cost models that fall apart in production [observed-pattern; not a benchmarked rate]. The economics collapse to three anchors worth tracking: Metric What it captures Why it can mislead alone Tokens/sec per GPU (sustained) Real serving throughput under your request mix Peak-burst figures overstate it 2–4× under realistic concurrency GPU-hours per million tokens Direct input to cost-per-inference Ignores idle capacity from poor batching or router imbalance Cost-per-inference with portability note The defensible business number Meaningless unless the CUDA/FP8 assumptions are stated The third row is the point of the whole exercise. A cost-per-inference figure is only decision-grade when it carries its assumptions: which precision, which serving stack, which hardware family, and what it would cost to reproduce elsewhere. Our own view, echoing what we cover in how DeepSeek inference works and the algorithmic choices that drive GPU cost, is that the algorithmic structure of the model sets the ceiling and the software stack determines how close you get to it — the H100 spec sheet sets neither. For the request-profile-specific breakdown of a reasoning model under load, the companion analysis of DeepSeek-R1 inference GPU utilisation and cost in practice walks through where utilisation actually leaks, which this hub deliberately leaves to that spoke. How much of the performance depends on CUDA-specific kernels? This is the question that separates a sizing exercise from a strategy decision. DeepSeek’s efficient serving relies on several NVIDIA-specific ingredients: FP8 Tensor Core execution. Hopper’s FP8 support is what makes the low-precision path fast. On hardware without equivalent FP8 acceleration, the same numerical scheme either runs slower or falls back to a wider format, changing both throughput and accuracy. Attention kernels tuned for the CUDA memory model. FlashAttention-style kernels are written against CUDA’s shared-memory and warp-level primitives. They do not transpile cleanly. MoE dispatch and communication. Routing tokens to experts and gathering results leans on NCCL and CUDA-aware collectives when experts are sharded across GPUs. KV-cache layout. The cache is laid out to match the kernels’ access patterns; a different backend wants a different layout. None of this is a criticism of DeepSeek or of NVIDIA. It is an accurate description of where the performance comes from. But it means the answer to “how much depends on CUDA?” is: enough that a naive port will not reproduce the numbers. In configurations we have profiled, the gap between the CUDA-tuned path and a first-pass non-CUDA reimplementation is large enough to change the hardware decision entirely [observed-pattern; not a published benchmark]. That gap is the migration cost, front-loaded into your architecture whether you priced it or not. What portability cost do you accept by standardising on H100? Standardising on H100 is a reasonable decision. It is only a defensible decision when you can state what you are giving up. The cost is not “H100 is expensive” — it is that the optimisation ceiling you can reach is bounded by a software stack you do not fully control the portability of. Concretely, a few things become true the day you commit: Your fastest configuration is your least portable one. The kernels that hit peak FP8 throughput are the kernels hardest to move. Your cost-per-inference figure is only valid on the CUDA path. Re-derive it before assuming it transfers. Migration is a rewrite, not a recompile. Attention kernels, MoE dispatch, and cache layout all need reimplementation for a different backend. Teams that want to keep the option open early usually invest in an abstraction boundary — the trade-offs of which we discuss in what a hardware-agnostic GPU compute solution really means. And for the mechanics of the move itself, porting a GPU inference path off the CUDA lock-in with HIP vs CUDA is the closest concrete walkthrough. The point is not to avoid H100 — it is to price the lock-in at commit time instead of discovering it at migration time. How does MoE routing affect GPU memory and utilisation on H100? Mixture-of-Experts changes the utilisation story in a way that dense models do not. Because only a fraction of experts activate per token, the compute per token is lower than the parameter count suggests — but the memory footprint is not, because all experts must be resident (or fetched) to serve any token. That decoupling is where H100 memory capacity and HBM bandwidth become the binding constraint rather than raw FLOPs. Two failure signatures show up in practice. The first is router imbalance: if certain experts receive disproportionate traffic, some GPUs sit hot while others idle, and your effective tokens/sec per GPU drops well below the theoretical figure [observed-pattern]. The second is cross-GPU communication overhead when experts are sharded — the all-to-all dispatch can dominate latency if the interconnect is not sized for it. Both are utilisation problems that a peak-FLOPs spec sheet will never surface, which is exactly why sustained measurement under a realistic request mix beats headline numbers. The serving-stack view of this appears in what profiling reveals about real DeepSeek inference bottlenecks under SGLang. Would DeepSeek run competitively on AMD or Intel accelerators? It can — with work. AMD’s Instinct line and Intel’s accelerators have their own low-precision support and their own serving stacks (ROCm/HIP on AMD, oneAPI/SYCL on Intel). The architecture of DeepSeek is not intrinsically NVIDIA-only. What is NVIDIA-specific is the tuned implementation you are benchmarking. To run competitively elsewhere you would need to reimplement or retune the FP8 kernels for the target’s numerical units, port the attention kernels off CUDA primitives, adapt MoE dispatch to the target’s collective library, and re-lay-out the KV-cache. Whether that pays off is a workload-and-scale question, not a religious one. At small scale the engineering cost may exceed the hardware savings; at large sustained scale the calculus flips. The honest framing is that “competitive” is achievable but is a project, and the size of that project is the number your H100 decision should have on the table. FAQ What does working with DeepSeek on H100 involve in practice? DeepSeek on H100 is the composite of the model architecture (a large Mixture-of-Experts transformer), its numerical format (FP8, natively accelerated by Hopper Tensor Cores), and a CUDA-tuned serving stack of attention kernels, expert routing, and KV-cache management. The measured performance belongs to all three together, not to the H100 alone. In practice it means a “DeepSeek on H100” benchmark is really a benchmark of a specific CUDA-tuned FP8 serving path on Hopper. What inference throughput and cost per million tokens can DeepSeek realistically achieve on an H100? There is no single honest figure — it depends on the request profile (prompt-to-generation ratio, batch size, sequence length) and how well the MoE router balances expert load. The metrics worth tracking are sustained tokens/sec per GPU, GPU-hours per million tokens, and a cost-per-inference number that states its precision and stack assumptions. Peak-burst figures typically overstate sustained throughput by a wide margin under realistic concurrency, so any quoted number without its request profile should be treated as fictional. How much of DeepSeek’s H100 performance depends on CUDA-specific kernels and FP8 paths? Enough that a naive port will not reproduce the numbers. The efficient path relies on Hopper FP8 Tensor Core execution, FlashAttention-style kernels written against CUDA memory primitives, NCCL-based MoE dispatch, and a cache layout matched to those kernels. In configurations we have profiled, the gap between the CUDA-tuned path and a first-pass non-CUDA reimplementation is large enough to change the hardware decision entirely. What portability cost do I accept by standardising DeepSeek inference on NVIDIA H100 hardware? Your fastest configuration becomes your least portable one, your cost-per-inference figure is valid only on the CUDA path, and any future migration is a rewrite of kernels, dispatch, and cache layout rather than a recompile. The cost is not that H100 is expensive; it is that your optimisation ceiling is bounded by a software stack whose portability you do not fully control. The defensible move is to price this lock-in at commit time, not discover it at migration time. How does DeepSeek’s Mixture-of-Experts routing affect GPU memory and utilisation on H100? MoE lowers compute per token because only a subset of experts activate, but all experts must remain resident, so memory capacity and HBM bandwidth become the binding constraint rather than raw FLOPs. Two failure signatures dominate: router imbalance leaves some GPUs hot while others idle, dropping effective tokens/sec, and sharded-expert dispatch can let all-to-all communication overhead dominate latency. Both are utilisation problems a peak-FLOPs spec sheet never surfaces. Would DeepSeek run competitively on AMD or Intel accelerators, and what would need to be rewritten? Yes, with engineering work — the architecture is not intrinsically NVIDIA-only, but the tuned implementation is. You would need to retune FP8 kernels for the target’s numerical units, port attention kernels off CUDA primitives, adapt MoE dispatch to the target’s collective library (ROCm/HIP on AMD, oneAPI/SYCL on Intel), and re-lay-out the KV-cache. At small scale the engineering cost may exceed the hardware savings; at large sustained scale the calculus can flip. How should I fold DeepSeek-on-H100 economics into a GPU API and hardware decision rather than defaulting to CUDA? Treat the cost-per-inference figure as valid only with its precision, stack, and hardware assumptions attached, and put the estimated migration cost on the same table as the hardware cost. Decide whether to invest in an abstraction boundary early based on your expected scale, not on which path is fastest today. The goal is not to avoid H100 but to make the CUDA commitment a priced choice rather than a default. The open question this leaves on the table The tempting version of this decision measures tokens per second and stops. The version that survives contact with a migration attempt measures tokens per second and names the CUDA/FP8 assumptions baked into that number and estimates what reproducing it elsewhere would cost. A GPU performance audit treats DeepSeek-on-H100 as exactly this kind of workload — profiling whether the current API and hardware pairing is structurally optimal or simply a defaulted CUDA choice that nobody has priced. So the closing question is not “how many H100s.” It is: for your workload shape and your scale, is the precision-and-accuracy trade-off of the FP8 CUDA path worth the portability you are trading away — and have you written that trade down as a number, or is it still hiding in a spreadsheet cell?
