“DeepSeek on H100” sounds like a hardware-fit question — will the model load, will it run. In a prototype that has to reach production, it is really a serving decision, and the two are not the same problem. The model fitting in H100 memory tells you almost nothing about whether it will hold a latency target once real traffic arrives.

The pattern we see is consistent. A team runs a DeepSeek variant once in a notebook, watches it respond in a second or two for a single prompt, confirms the weights sit comfortably inside 80 GB of HBM, and declares the hardware question settled. Then production traffic shows up, concurrency climbs, and the numbers that actually govern the budget — tokens per second per GPU, p95 latency, cost per million tokens — turn out to be nothing like the demo. That divergence is the entire subject of this article.

What does “DeepSeek on H100” actually mean in practice?

The phrase collapses three separate questions into one. First, does a given DeepSeek model fit in the memory of a single H100? Second, how fast does it serve under the concurrency your application will produce? Third, what does that serving profile cost per token at your committed latency? Only the first is a hardware-fit question, and it is the least interesting one.

The reason the single-request demo misleads is that a language model on a GPU is throughput-bound, not latency-bound, once you have more than one user. A single request underutilizes the H100 badly — the tensor cores sit mostly idle while the model streams tokens one at a time, memory-bandwidth-limited during decode. Batching multiple concurrent requests is what pushes utilization up and drives the cost per token down. But batching also raises per-request latency, because each request now waits for the others in its batch. The H100 is one variable in a serving budget where batch size, quantization, KV-cache pressure, and concurrency all move the real numbers. Empirical execution under representative load — not a notebook impression — is the only reference standard that predicts production behavior.

This is fundamentally a GPU inference optimization problem, and the levers are the same ones that govern any transformer serving stack. Our broader treatment of how DeepSeek inference behaves and what it costs in production works through the cost mechanics in more depth; this article stays on the specific question of what the H100 pairing changes.

Which DeepSeek variants fit on a single H100?

DeepSeek ships in a wide capacity range, and “fits” depends heavily on precision and the context length you intend to support. The H100 SXM carries 80 GB of HBM3 per NVIDIA’s published specifications (the 94 GB H100 NVL variant exists as well). Weights are only part of the budget — the KV cache for in-flight requests grows with batch size and sequence length, and at long context under high concurrency it can rival or exceed the weight footprint.

A rough way to reason about it: model weights in FP16 consume roughly two bytes per parameter, so a dense model in the tens-of-billions range leaves useful headroom on one H100, while the largest DeepSeek variants — the MoE models with hundreds of billions of total parameters — do not fit on a single GPU at FP16 regardless of how much KV-cache room you want, and require multi-GPU tensor or expert parallelism over NVLink. Quantization to INT8 or FP8 roughly halves the weight footprint and is often what moves a borderline model onto one card with enough KV-cache headroom to matter.

DeepSeek serving question	Single H100 (80 GB)	When you need multiple GPUs
Dense model, tens of billions of params, FP16	Fits, with moderate KV-cache headroom	Only if you want very high concurrency or very long context
Same model, INT8/FP8	Fits comfortably, more KV-cache room	Rarely — quantization usually keeps it single-card
MoE / hundreds of billions total params	Does not fit at FP16	Tensor + expert parallelism over NVLink
Long-context (100k+ tokens) at high concurrency	KV cache dominates; may not fit	Shard KV cache or add GPUs

The table is a planning heuristic, not a benchmark — the exact boundary depends on the specific variant, the serving framework’s memory management, and your target context length. Treat it as a starting point for a sizing test, not a substitute for one. Our guide to what a 32B model is and when its capacity fits your use case covers the capacity-versus-fit trade-off for the mid-range case in more detail.

What throughput and p95 latency can you realistically expect?

This is where notebook impressions and production reality diverge most sharply. A single-user demo tells you the time-to-first-token and the per-token decode speed with the GPU almost entirely to itself. Under concurrent load, both numbers change, and they change in opposite directions from what a naive reading expects.

As concurrency rises, aggregate tokens per second per H100 climbs — the GPU is doing more useful work per unit time — while per-request latency also climbs, because requests share batch slots and compete for KV-cache memory. The operationally relevant measure is sustained tokens per second at a committed p95 latency, not the peak throughput you can hit by batching aggressively and ignoring tail latency. A serving configuration tuned for maximum throughput will quietly blow past your p95 target; one tuned for single-request latency will waste most of the H100 and wreck your cost per token.

The honest answer to “what can I expect” is: you cannot know without measuring your workload, because it depends on prompt length distribution, output length distribution, and the concurrency shape of your traffic. What you can commit to is a measurement protocol. Load the DeepSeek variant on the H100 in your chosen serving framework — vLLM, TensorRT-LLM, or an SGLang-based stack — replay a representative request trace at increasing concurrency, and record the throughput-versus-p95 curve. The knee of that curve is your operating point. Reading benchmark curves this way is exactly what our walkthrough of the DeepSeek R1 benchmark for real-time GenAI latency is built to support.

How do quantization and batching change memory and cost per token?

Quantization and batching are the two largest levers on the DeepSeek/H100 cost equation, and they interact.

Quantization to FP8 — which the H100’s Transformer Engine supports natively in hardware — or to INT8 reduces the weight footprint and the memory-bandwidth cost of moving weights during decode. Because decode is bandwidth-bound, lower-precision weights typically raise decode throughput as well as freeing HBM for a larger KV cache. The trade-off is accuracy: FP8 and well-calibrated INT8 usually hold quality on many tasks, but this is a task-specific question that has to be validated, not assumed. Precision is a first-class trade-off, not a free win — our overview of model optimization for edge inference covering distillation and quantisation treats the accuracy side of that decision, and the same discipline applies on datacenter H100s.

Batching is the throughput lever. Continuous batching (also called in-flight batching), as implemented in vLLM and TensorRT-LLM, lets new requests join a batch as slots free up rather than waiting for a full batch to complete. This is the single biggest difference between a serving stack that gets useful cost per token out of an H100 and one that does not. Prefix caching — reusing the KV cache for shared prompt prefixes, as covered in our explainer on radix cache and prefix reuse for production LLM serving — cuts redundant prefill work when many requests share system prompts or few-shot examples, which is common in production agents.

The combined effect: a DeepSeek variant that costs one figure per million tokens at batch-of-one, single-precision serving can cost dramatically less at FP8 with continuous batching, at the price of some added tail latency and a precision validation step. The numbers are yours to measure; the mechanism is general.

How do you turn a notebook benchmark into production serving targets?

Three numbers convert a demo into a defensible decision. Name them before promotion, not after the incident.

1. Tokens per second per H100 at a committed p95 latency. Not peak throughput. The sustained figure at the tail-latency you promise users. This is your capacity-planning unit.
2. Cost per one million tokens under realistic concurrency. Derived from GPU-hour cost divided by sustained throughput at your operating point. A prototype that meets latency for one user can exceed the GPU-hour budget by a large multiple once concurrent traffic arrives, because single-user serving wastes most of the card.
3. Memory headroom for your target context length. The KV cache your longest supported context needs at your peak concurrency, checked against what remains after weights load at your chosen precision.

If you cannot state these three, you have a notebook impression, not a serving plan. The A3 production-readiness assessment is designed to surface exactly this gap — the serving-latency and cost divergence between the demo and the committed target — before a team commits infrastructure to it.

FAQ

What matters most about deepseek h100 in practice?

Running DeepSeek on an H100 means loading a DeepSeek model’s weights into the GPU’s HBM and serving inference requests through a framework like vLLM or TensorRT-LLM. In practice it is a serving decision, not just a hardware-fit check: whether the model loads is the easy part, while sustained throughput, p95 latency, and cost per token under concurrent load are what actually determine production viability. The single-request notebook demo does not predict any of those production numbers.

Which DeepSeek model variants fit on a single H100, and when do you need multiple GPUs?

A dense DeepSeek model in the tens-of-billions-of-parameters range fits on one 80 GB H100 at FP16 with useful KV-cache headroom, and quantization to INT8 or FP8 gives more room. The largest MoE variants, with hundreds of billions of total parameters, do not fit at FP16 on a single card and require tensor or expert parallelism over NVLink. Long-context serving at high concurrency can also push you to multiple GPUs because the KV cache grows with sequence length and batch size.

What throughput and p95 latency can you realistically expect from DeepSeek on H100 under concurrent load?

You cannot know without measuring your own workload, because it depends on prompt-length distribution, output-length distribution, and concurrency shape. As concurrency rises, aggregate tokens per second per H100 increases while per-request latency also increases, so the relevant figure is sustained throughput at a committed p95 latency rather than peak throughput. Replay a representative request trace at increasing concurrency and read the knee of the throughput-versus-p95 curve as your operating point.

How do quantization and batching change DeepSeek’s memory footprint and cost per token on H100?

FP8 or INT8 quantization roughly halves the weight footprint and, because decode is memory-bandwidth-bound, typically raises decode throughput and frees HBM for a larger KV cache — at a task-specific accuracy cost that must be validated. Continuous batching lets new requests join a batch as slots free up, which is the largest lever on cost per token, and prefix caching cuts redundant prefill work when requests share prompt prefixes. Together they can cut cost per million tokens substantially, at the price of some added tail latency.

How do you translate a DeepSeek/H100 notebook benchmark into production serving targets?

Convert the demo into three named numbers before promotion: sustained tokens per second per H100 at a committed p95 latency, cost per one million tokens under realistic concurrency, and the memory headroom your target context length needs at peak concurrency. Derive cost per token from GPU-hour cost divided by sustained throughput at your operating point, not from the single-user demo. If you cannot state all three, you have a notebook impression rather than a serving plan.

When is DeepSeek on H100 the right serving choice versus a hosted API or a smaller model?

Self-serving DeepSeek on H100 tends to win when your token volume is high enough that the GPU-hour cost beats per-token API pricing, when data residency or governance rules out a hosted API, or when you need control over latency and model version. A hosted API is often better at low or spiky volume where you would leave the card idle, and a smaller model can be the right answer when it meets quality at a fraction of the serving cost. The decision hinges on your measured throughput and cost per token, not on the model’s benchmark scores alone.

Where this leaves a team is with a single, testable question that the notebook cannot answer: at the concurrency and context length your product actually generates, what is the sustained tokens-per-second-per-H100 at your committed p95, and does the resulting cost per token fit the budget? Answer that with a load test on your own generative AI traffic before you commit the hardware — the failure mode we keep seeing is a prototype that was validated against a single user and priced against a fantasy, and the A3 production-readiness assessment exists to catch that serving-latency and cost gap before it becomes a production incident.