Search “DeepSeek H100” and you get a hardware answer: which GPU, how many, how many tokens per second. That framing is where the money leaks. In a running inference pipeline the H100 is one stage among several — tokenisation, request batching, KV-cache management, output detokenisation and post-processing — and several of those stages sit on the CPU. Provisioning fleet capacity from an isolated tokens-per-second number on the H100 tells you almost nothing about what a request actually costs to serve.

The naive move is to benchmark the model on a single H100, read a throughput figure, and divide expected traffic by that figure to size the fleet. The number is real, but it measures the GPU running flat out with nothing waiting on it. Production traffic is not that. Prompts arrive at irregular intervals, get tokenised on the host, get copied over PCIe into device memory, wait in a batch, and the generated tokens make the return trip to be decoded back into text. The stage boundaries around the H100 — host-device copies and CPU-side stalls — frequently cost more than the acceleration saves on the stages that genuinely belong on the GPU.

Which parts of a DeepSeek inference run actually execute on the H100?

It helps to be concrete about where each stage physically runs, because the mental model of “the model runs on the GPU” hides the handoffs that dominate cost.

Tokenisation — turning the incoming prompt string into token IDs — is a CPU operation in almost every serving stack. Batching, the scheduler that groups requests so the GPU processes several at once, is CPU-side orchestration. The prefill and decode passes of the transformer — the matrix multiplications, the attention over the KV-cache — are the GPU’s job, and this is where the H100’s HBM bandwidth and tensor cores earn their keep. KV-cache management straddles the boundary: the cache lives in device memory, but the eviction and allocation logic is scheduled from the host. Detokenisation and any output post-processing (safety filtering, streaming assembly, format conversion) land back on the CPU.

So a single request crosses the CPU/GPU boundary at least twice, and a poorly configured server crosses it far more often. Each crossing is a host-device copy over PCIe, and each copy is time the H100 spends either stalled or servicing a copy engine instead of computing. This is the mechanism the throughput-in-isolation benchmark cannot see, because in that benchmark the copies were amortised away by an unrealistically full pipeline.

How host-device copies show up as acceleration discord

When you profile a real DeepSeek deployment — Nsight Systems or a Torch profiler trace is enough to start — the discord shows up as gaps in the GPU timeline. The H100 finishes a decode step and then sits idle for a few hundred microseconds waiting for the next batch to be tokenised and copied in. Multiply that gap across every step of every request and the GPU’s duty cycle — the fraction of wall-clock time it spends actually computing — collapses well below what the tokens-per-second figure implied.

A common pattern in serving stacks that were assembled quickly: the KV-cache tensors for a growing conversation get copied host-to-device on every turn instead of being pinned and reused, and prompt tensors get re-copied because the batching layer rebuilt them. The acceleration you paid for is real; the discord around it is what turns a fast GPU into an expensive idle one. Redundant copies and CPU-side batching stalls are the two failure modes we see most often when auditing LLM inference, and both are invisible to a benchmark that never left the device.

This is the same structural problem that shows up whenever a GPU sits waiting on a CPU-bound neighbour. In a video-analytics context the culprit is the decode path or the encoder; the way CPU encoding fits into a GPU analytics pipeline is governed by exactly the same handoff economics. In an edge context, the way edge accelerators handle the CPU-accelerator handoff is the miniature version of the H100 story. The silicon changes; the boundary tax does not.

Why a low H100 duty cycle is a routing signal, not a buy signal

Here is the reading that costs the most money. A team measures fleet H100 utilisation, sees it running at, say, 40 percent, concludes the GPUs are the bottleneck under load, and provisions more of them. But a duty cycle that low usually means the opposite: the H100s are starved, waiting on CPU-side work and copies, so adding more of them just adds more idle silicon. You have bought capacity to make the same discord happen in parallel.

Read honestly, a low duty cycle is a routing signal. It tells you that some stages currently attributed to the GPU are actually spending their time on the boundary, and that the fix is to profile the pipeline and re-place work — not to grow the fleet. The divergence between the naive and expert reading is the profile: once you can see where copies and stalls land, you keep on the H100 only the stages whose GPU economics survive the handoff and return the rest to CPU. That is an observed-pattern from GPU inference audits, not a benchmarked constant — the exact numbers depend on model size, sequence length, and batch policy — but the direction is consistent.

The measurable outcome you are steering toward is tokens-per-H100-hour holding steady or rising while the fleet H100 count stays flat. Cost-per-inference-hour against value delivered is the honest denominator; H100 count is not. If you want the framing for the fleet-level version of this argument, our GPU acceleration practice treats delivered throughput per GPU-hour, rather than peak throughput, as the number worth defending.

Decision rubric: keep on H100, or fall back to CPU?

Once the profile is in hand, each stage gets a routing decision. The table below is the rubric we apply — it is self-contained, but the inputs come from the trace, not from a spec sheet.

Stage	Where it usually runs	Keep on H100 if…	Fall back to CPU if…
Tokenisation	CPU	Rarely worth GPU offload	Almost always — it is cheap, and offloading adds a copy
Request batching	CPU (orchestration)	N/A — it is scheduling, not compute	Always; tune batch size/timeout to keep the GPU fed
Prefill (prompt encode)	H100	Sequence is long enough to amortise the copy	Very short prompts where copy dominates compute
Decode (token generation)	H100	The core GPU workload — keep it	Never; this is what the H100 exists for
KV-cache management	Boundary	Cache pinned in device memory, reused across turns	Logic re-copies tensors each turn — fix the reuse, do not move the cache
Detokenisation / post-proc	CPU	Rarely	Almost always; keep the GPU decoding, not formatting

The pattern the table encodes: the GPU should own the dense, high-arithmetic-intensity stages (prefill, decode) and as little else as possible, while the CPU owns the string handling and orchestration. The failure is not “wrong hardware” — it is stages placed on the wrong side of a boundary they cross too often. Frameworks like vLLM and TensorRT-LLM already push hard on this with continuous batching and paged KV-cache precisely because the boundary tax is the thing worth engineering against.

How a GPU Performance Audit reveals the mispriced stages

Profiling DeepSeek-on-H100 as a pipeline rather than a model is the substance of a GPU Performance Audit scoped to LLM inference. The audit walks the stage boundaries, names where prompt and tensor copies and CPU batching stalls erode the GPU economics the throughput number promised, and quantifies the recoverable duty cycle. The output is not “your GPU is slow” — it is “these three stages are being charged to the H100 while they spend their time on the boundary, and here is what re-placing them recovers.”

The stage-boundary stall is a specific, recognisable instance of the broader GPU underutilisation pattern: silicon sitting idle because a neighbouring stage cannot feed it fast enough. It is also, viewed from the other axis, a latency phenomenon — the handoff between CPU tokenisation and H100 compute is a source of tail latency that no amount of extra GPU capacity will remove. Both readings point at the same profile and the same fix.

FAQ

How does deepseek h100 actually work?

“DeepSeek on H100” means running the DeepSeek model’s inference passes on an NVIDIA H100 GPU. In practice it is a multi-stage pipeline, not a single GPU job: tokenisation, batching, and detokenisation run on the CPU, while the prefill and decode passes run on the H100. The practical meaning is that you cannot size a fleet from a GPU-only throughput number, because CPU stages and the copies between them shape the real cost.

Which parts of a DeepSeek inference run actually execute on the H100 versus the CPU?

The transformer prefill and decode passes — the matrix multiplications and attention over the KV-cache — run on the H100. Tokenisation, request batching orchestration, detokenisation, and output post-processing run on the CPU. KV-cache management straddles the boundary: the cache lives in device memory but is allocated and evicted by host-side logic.

How do host-device copies for prompts, tensors, and KV-cache show up as acceleration discord on H100?

They show up as gaps in the GPU timeline when you profile with a tool like Nsight Systems or a Torch profiler — the H100 finishes a step and idles while the next batch is tokenised and copied in over PCIe. Redundant copies (re-copying prompt tensors, or moving KV-cache tensors every conversation turn instead of reusing them) multiply these gaps across every request, collapsing the duty cycle below what the tokens-per-second figure implied.

Why can a low H100 duty cycle be a routing signal rather than a reason to add more GPUs?

A low duty cycle usually means the H100s are starved — waiting on CPU-side work and copies — rather than saturated. Adding more GPUs in that state just adds idle silicon and reproduces the same discord in parallel. Read honestly, it signals that work should be re-placed across the CPU/GPU boundary, which is a profiling-and-routing fix, not a provisioning one.

How does a GPU Performance Audit reveal which DeepSeek inference stages are mispriced across the CPU/GPU boundary?

The audit profiles the pipeline as stages rather than as a single model, walking each stage boundary to locate where prompt/tensor copies and CPU batching stalls erode the GPU economics the throughput number promised. It quantifies the recoverable duty cycle and identifies the specific stages charged to the H100 that actually spend their time on the boundary.

Which inference stages should stay on the H100 and which should fall back to CPU once the discord is profiled?

Prefill and decode should stay on the H100 — they are the dense, high-arithmetic-intensity work the GPU exists for. Tokenisation, batching orchestration, and detokenisation should stay on the CPU, since offloading them adds copies without meaningful acceleration. KV-cache should stay pinned in device memory and be reused across turns rather than re-copied.

How does resolving stage-boundary discord change cost-per-inference-hour and tokens-per-H100-hour?

Eliminating redundant host-device copies and CPU-side batching stalls recovers H100 duty cycle, so the GPU spends more wall-clock time computing. The measurable outcome is tokens-per-H100-hour holding steady or rising while the fleet H100 count stays flat, which lowers cost-per-inference-hour against value delivered without buying more capacity.

The next time a dashboard shows H100 utilisation sitting well below the throughput you benchmarked, resist the instinct to read it as a shortfall to be provisioned away. Profile the boundary first: a GPU Performance Audit scoped to LLM inference will usually find that the fleet you already own has duty cycle to recover, and that the stages costing you money are the ones sitting on the wrong side of a copy.