A benchmark slide says an H100 pushes some large number of tokens per second on a given model. Someone sizes the GenAI cluster off that figure, the product ships, and the first time a user types quickly the streaming response stutters. The number on the slide was real. It just answered a question nobody in the product was asking. That gap — between a published NVIDIA HPC or MLPerf inference number and what a real-time streaming pipeline actually delivers — is where most GPU sizing decisions for interactive GenAI go wrong. The benchmark reports aggregate throughput under conditions engineered to maximise it. Your text-to-speech (TTS) stream or streaming LLM chat is engineered around the opposite constraint: the latency before the first token comes back. These are not the same measurement, and reading one as if it were the other is the single most common way we see hardware over- or under-provisioned for interactive workloads. What NVIDIA HPC benchmarks actually measure NVIDIA’s HPC and inference benchmark suites, along with the industry-standard MLPerf Inference results NVIDIA submits to, report a specific set of figures under controlled conditions: peak throughput (often tokens per second for LLMs), samples per second, and aggregate latency percentiles for a defined scenario. MLPerf’s “Offline” scenario, for instance, feeds the system all queries at once and measures how fast it can drain the queue — a throughput number, deliberately. The “Server” scenario adds a query-arrival distribution and a latency constraint, which is closer to production but still runs at the concurrency and constraint MLPerf defines, not the one your product runs at. The headline number that ends up on a slide is usually the throughput figure, because it is the largest and the most flattering. Per NVIDIA’s published MLPerf submissions, these are legitimate benchmark-class results — reproducible, auditable, run under a declared methodology. The problem is never that the number is false. The problem is that a throughput number is produced by maximising batch size, and a real-time GenAI product cannot run at that batch size without breaking its own latency budget. That single sentence is the whole article, so it is worth stating plainly: the batch size that produces the best benchmark tokens/sec is usually larger than the batch size your streaming UX can tolerate. Everything below unpacks why. Throughput versus the latency that governs a streaming UX Interactive GenAI has two latency numbers that a throughput benchmark does not surface directly: Time to first token (TTFT) — how long after the request arrives before the first token (or first audio chunk, for TTS) streams back. This is dominated by the prefill phase, where the model processes the entire prompt before generating anything. Inter-token latency (ITL), also called time-per-output-token — the gap between successive streamed tokens during the decode phase. This governs whether the response feels fluid or stutters. A user perceives a chat or voice assistant as responsive based almost entirely on TTFT and a steady ITL. A throughput benchmark, by contrast, rewards a system for keeping the GPU saturated — which it does by batching many requests together so no compute cycle is idle. Large batches raise aggregate tokens/sec and simultaneously raise TTFT for every request in the batch, because each request waits for the batch to fill and for the shared prefill to complete. Here is the divergence in a compact form. Benchmark throughput vs real-time latency: what each optimises for Dimension Peak throughput benchmark Real-time streaming GenAI Optimised for Total tokens/sec across all requests First-token latency for each request Batch size As large as memory allows Capped to protect TTFT Governing metric Aggregate tokens/sec p95 TTFT, then steady ITL Concurrency model Queue drained at full load Bursty, user-paced arrivals Sequence length Fixed, often favourable Variable, prompt-dependent Failure that hurts users (not measured) First token arrives too late The two columns describe the same GPU running the same model. What changes is the operating point, and the operating point is chosen by batch size and concurrency — neither of which the headline number tells you. Why the batch size hides in the tokens/sec figure Modern LLM inference servers — vLLM, TensorRT-LLM, NVIDIA Triton with the TensorRT-LLM backend — use continuous (in-flight) batching to keep the GPU busy: they interleave the prefill of new requests with the decode of in-progress ones. Continuous batching is what lets a benchmark report a high aggregate number, and it is also genuinely useful in production. But its throughput scales with how many requests are in flight, and every request in flight competes for the same HBM bandwidth and the same attention-kernel compute during prefill. When a benchmark maximises tokens/sec, it is running at high concurrency with large effective batches. Push a request into that server and its first token waits behind the prefill of everyone else’s prompts. In configurations we’ve worked with, holding batch size low enough to keep p95 TTFT under a sub-second interactive budget can cut the achievable aggregate tokens/sec to a fraction of the benchmark headline — a real trade-off, not a tuning defect (observed across TechnoLynx engagements; not a published benchmark rate). The GPU is not slower. It is being asked to protect a different metric. This is why a tokens/sec figure alone cannot size hardware for interactive GenAI. You have to know the batch size and concurrency it was measured at, and then ask whether your product can run at that operating point. Usually it cannot, which means the effective throughput you will see is lower — and the GPU count you need is higher — than the slide implies. The mechanics of closing that gap on the GPU itself, from kernel fusion to scheduling, are a separate discipline; our [latency optimisation methodology for GPU inference](GPU engineering) covers how to recover throughput without sacrificing the first-token budget. How do you map an MLPerf number to your own latency budget? The translation is mechanical once you stop treating the benchmark as a product spec. Read the benchmark’s declared conditions, then re-derive the number at your operating point. Checklist: from a published benchmark to a sizing decision Find the operating point. What batch size, concurrency, and sequence length produced the number? MLPerf submissions declare the scenario and the latency constraint; NVIDIA’s TensorRT-LLM perf pages declare batch size and input/output lengths. If the source does not declare these, treat the number as unusable for sizing. State your own target. Interactive chat and TTS typically need a p95 TTFT budget — often sub-second for text, tighter for the first audio chunk in a voice stream. Write it down as a number before you look at any hardware. Match sequence lengths. Benchmarks often use short, fixed prompts. If your real prompts (RAG context, system prompts, conversation history) are 5–10× longer, prefill cost — and therefore TTFT — rises accordingly. Estimate your real distribution; a token size calculator for latency and cost budgets helps turn prompt shapes into token counts you can reason about. Re-run at your batch cap. Take the maximum batch size that keeps p95 TTFT under budget and measure tokens/sec there, not at the benchmark’s batch size. This is the number that sizes your fleet. Size against concurrency, not peak. Your GPU count follows from concurrent-users × per-request cost at your batch cap, with headroom for arrival bursts — not from dividing target daily tokens by the peak figure. The discipline is identical to reading any latency-sensitive inference benchmark. If you are working with reasoning-heavy models where decode length varies wildly per request, the same reasoning applied to a specific model appears in our walkthrough of the DeepSeek R1 benchmark for real-time GenAI latency, and the client-side view of what these suites capture is in MLPerf Client: what the benchmark measures and how to read it. Which metrics matter for TTS and streaming LLM workloads? For TTS and streaming chat, rank the metrics by what the user feels: First-token / first-chunk latency (TTFT) is the primary constraint. For voice, the first audio chunk budget is often tighter than a text first-token budget because silence at the start of speech is more noticeable than a pause in a text stream. Inter-token latency (ITL) must stay below the rate at which the user consumes output — reading speed for text, real-time playback for audio. A model that clears ITL comfortably has headroom; one that only just clears it will stutter under load. Sustained throughput at your batch cap matters for cost and fleet sizing, but only after the two latency metrics are satisfied. It is a second-order concern, not the headline. Peak offline tokens/sec — the number most likely to be on the slide — ranks last for interactive workloads. It tells you the ceiling of the hardware, not the floor your users will experience. This ordering is what separates the naive read of a benchmark from an expert one, and it is why generative-AI feasibility work has to start from the generative AI engineering practice that owns the latency budget, not from a hardware datasheet. Validating a hardware choice against a latency budget The measurable outcome of reading these benchmarks correctly is a hardware decision you can defend: a GPU choice validated against realistic concurrency and p95 first-token latency, rather than a demo-condition tokens/sec figure. That protects against both failure directions we see in practice — over-provisioning on throughput numbers that never materialise at low batch sizes, and under-provisioning that blows the first-token budget the first time real users arrive. The way to earn that confidence is to reproduce the benchmark’s operating point on your own model and hardware, then re-measure at your batch cap and your sequence-length distribution. If you cannot reproduce the vendor’s declared conditions, you cannot trust the extrapolation to yours. FAQ What does working with nvidia hpc benchmarks involve in practice? NVIDIA’s HPC and inference benchmarks — including the MLPerf Inference results NVIDIA submits — report peak throughput, samples or tokens per second, and aggregate latency percentiles under a declared scenario and operating point. In practice they tell you the ceiling of the hardware under conditions engineered to maximise a chosen metric, usually throughput. They are legitimate, reproducible results, but they only map to your product once you re-derive the number at your own batch size, concurrency, and sequence length. What is the difference between benchmarked peak throughput and the first-token / inter-token latency that governs a real-time GenAI UX? Peak throughput is aggregate tokens per second across all in-flight requests, maximised by large batches. Time to first token (TTFT) is how long a single request waits before the first token streams back; inter-token latency (ITL) is the gap between streamed tokens. A user perceives responsiveness almost entirely through TTFT and steady ITL, so a system can post a huge throughput number while delivering a first-token experience that feels slow. Why does the batch size used in NVIDIA HPC benchmarks make published tokens/sec figures misleading for low-latency streaming? Throughput scales with batch size and concurrency, so benchmarks maximise both to produce the largest number. A real-time product must cap batch size to protect first-token latency, because every request in a large batch waits behind the shared prefill. The published tokens/sec is therefore measured at an operating point your streaming UX cannot run at, so your achievable throughput is lower and your GPU count higher than the headline implies. How do you map an MLPerf or NVIDIA inference benchmark to your own concurrency, sequence length, and latency budget? Find the operating point the number was measured at — batch size, concurrency, sequence length, latency constraint — and treat it as unusable if the source does not declare it. Then state your own p95 TTFT budget, match your real sequence-length distribution, re-measure tokens/sec at the maximum batch size that keeps TTFT under budget, and size your fleet against concurrency at that batch cap rather than against the peak figure. Which benchmark metrics matter most for low-latency TTS and streaming LLM workloads? First-token (or first-audio-chunk) latency is the primary constraint, especially for voice where opening silence is noticeable. Inter-token latency must stay below the user’s consumption rate — reading speed or real-time playback — to avoid stutter. Sustained throughput at your batch cap matters for cost and fleet sizing, but only after both latency metrics are satisfied; peak offline tokens/sec ranks last for interactive workloads. How do you validate a GPU hardware choice against a real-time latency budget instead of a peak-throughput number? Reproduce the benchmark’s declared operating point on your own model and hardware, then re-measure at your batch cap and your real sequence-length distribution to get the throughput you will actually see. Size against concurrent users and p95 first-token latency with headroom for arrival bursts. If you cannot reproduce the vendor’s conditions, you cannot trust the extrapolation, and the sizing decision is not yet defensible. Read a benchmark right and the question stops being “how fast is this GPU?” It becomes “at the batch size my first-token budget allows, how many of these do I need?” — which is the only question a real-time GenAI sizing decision was ever asking.
