In one room, the throughput team celebrates. In the next room, the serving team is debugging latency spikes.
They’re looking at the same system, the same GPU, the same deployment. The throughput number went up after a batching change. Tokens per second are higher than they’ve ever been. Meanwhile, the p99 response time has doubled, the user-facing SLO is being violated, and the serving team is trying to figure out what went wrong.
Nothing went wrong. The system was optimized for one objective, and it degraded on another. This is not a bug — it’s the central tension in inference system design, and it shows up every time someone treats “performance” as a one-dimensional concept.
Throughput and latency are different axes, and in most practical AI systems they compete for the same resources. Optimizing for one without tracking the other is how you end up with a system that looks great by one measure and fails by another.
Throughput and latency answer different questions
Throughput measures volume over time: how many tokens, images, or inference steps the system can complete in a sustained window — exactly the planning lens in steady-state performance, cost, and capacity planning. It answers the capacity question — “how much work can this system handle?”
Latency measures time per unit of work: how long a single request takes from the caller’s perspective. It answers the responsiveness question — “how long does someone wait?”
These sound like they should be correlated — more throughput should mean faster individual requests, right? In practice, the relationship is often inverse once you push past the system’s natural operating point, because the mechanisms that increase throughput frequently increase per-request latency as a side effect.
Throughput vs. latency: two objectives that compete
| Dimension | Throughput optimization | Latency optimization |
|---|---|---|
| Primary metric | Total work per unit time (tokens/s, images/s) | Time per request (p50, p99, TTFT) |
| Batching strategy | Large batches to amortize overhead | Small batches to minimize queueing delay |
| GPU utilization pattern | High, sustained — GPU stays busy | Variable — GPU may idle between small batches |
| Who cares | Batch processing, offline inference, cost planning | Interactive serving, real-time APIs, user-facing SLAs |
| Risk of over-optimizing | Individual request latency spikes | Under-utilized hardware, higher cost per request |
Evidence class for this table: observed pattern across inference systems we have looked at, not a benchmarked ranking of configurations.
Batch size is where the trade-off becomes concrete
Batching is the clearest lever for this tension, and on GPU-based inference systems it’s usually the primary one.
Larger batches improve throughput because they amortize fixed overhead: kernel launch cost, memory allocation, scheduling, and framework-level bookkeeping. A batch of 32 requests pays these overheads once rather than 32 times, and the GPU’s parallel architecture means the incremental cost of additional items within a batch is often much less than the cost of processing them individually. So total tokens per second goes up.
But from an individual request’s perspective, larger batches mean waiting. Each request must wait until enough peers have arrived to form a batch (queueing delay), and then the entire batch must complete before any individual result is returned (processing delay). Both of these push per-request latency upward.
This isn’t a tuning problem with a universally correct answer. It’s a design trade-off, and the correct operating point depends entirely on what the system is optimizing for. A throughput-maximizing batch configuration looks very different from a latency-minimizing one, and a system optimized for one will underperform on the other — by design, not by defect.
We find that the teams who get into trouble are not the ones who make this trade-off explicitly. It’s the ones who make it accidentally — by optimizing for whatever the dashboard emphasizes without realizing they’ve shifted the system into a regime that violates a different objective.
Averages hide what users actually experience
In latency-sensitive systems, average latency is one of the least useful statistics you can report, and it’s often the one that gets the most attention.
The problem is that the average is dominated by the easy cases. If 95% of requests complete in 40ms and 5% take 500ms, the average looks like ~63ms — a number that describes neither the experience of the majority (40ms) nor the experience of the tail (500ms). The system looks “fine” by the average while 1 in 20 users has a genuinely degraded experience.
This is why percentile metrics — p50, p95, p99, sometimes p999 — matter so much for serving workloads. They tell you what happens at different points in the distribution, including the tail, which is usually where operational pain concentrates: timeout-triggering latency, retries, and cascading failures in downstream systems.
Contention, queueing bursts, GC pauses, CUDA context switching, and intermittent memory pressure all show up in the tail long before they affect the mean. A system can have a stable average while the tail quietly grows worse — especially under increased load, which is, of course, exactly when it matters most.
Report percentiles, not a single number
The practical rule is to report a percentile ladder rather than one summary statistic. p50 tells you the typical experience, p90 and p95 tell you where the distribution starts to fan out, and p99 (sometimes p999) tells you what your worst-served users actually live with. The gap between p50 and p99 is the signal: a wide gap means the tail is heavy and load-sensitive, even if the median looks healthy. Reporting only the average collapses that ladder into a number that hides exactly the behaviour a latency-sensitive system is judged on.
How do model latency and system latency differ?
Another common confusion is treating “model latency” and “system latency” as interchangeable.
Model latency covers the forward pass execution on the device — the time from input tensors on the GPU to output tensors on the GPU. System latency includes everything else: request parsing, tokenization, batching policy decisions, queueing, memory management, output detokenization, and transport back to the caller.
In a well-optimized serving system, model latency may account for only a fraction of end-to-end time. The rest is system overhead — not overhead in the pejorative sense, but the real mechanical cost of operating a service. If you measure only model latency (because it’s what profiling tools show most clearly), you may conclude the GPU is fast while the user is waiting on something the GPU has nothing to do with.
Collapsing these into one number creates an illusion of clarity while hiding the actual lever you’d need to pull. And when someone says “the GPU is fast but the service is slow,” the explanation is almost always in this gap.
For LLM serving, this split has its own vocabulary, and each term exposes a different face of the throughput-latency trade-off. TTFT (time-to-first-token) measures how long the caller waits before the first token appears — dominated by prefill and by queueing delay, so it gets worse as you batch harder for throughput. TPOT (time-per-output-token) and ITL (inter-token latency) measure the steady drip of generation once decoding starts; they govern how fast the response streams and degrade as more sequences share a decode batch. A throughput-tuned configuration can raise aggregate tokens-per-second while pushing TTFT and ITL the wrong way for an interactive user, which is the same competition described above, just measured per-token.
What a saturated-throughput benchmark can and cannot answer
It is worth being blunt about where a benchmark sits on this axis, because the trade-off decides what any given measurement is able to answer. A LynxBenchAI run is a throughput instrument by construction: the batch size is raised until throughput stops improving inside a defined noise band, and completed iterations are counted inside a continuous timed window after a discarded warm-up. Model architecture and numerical precision are held constant within a release; only the batch dimension adapts, which is why the batch axis is the one under discussion here.
That design has a consequence a reader should not have to guess at. A saturated-throughput score does not report latency percentiles, tail behaviour, TTFT, or inter-token latency, and no amount of re-reading it will produce those figures — the run deliberately sits at the throughput-maximizing end of the very trade-off this article describes. Getting a tail-latency answer requires a second, differently-shaped measurement on the same hardware: a fixed arrival-rate or fixed-concurrency test at a batch policy you would actually serve with, recording the full distribution rather than aggregate completions. The throughput run tells you where your device saturates; the latency run tells you what that saturation point costs the caller.
The three category scores — Training, Inference, Compute — also stay separate rather than being collapsed into one figure, because a device strong at one class of work is often weak at another, and collapsing them would hide exactly the trade-off under discussion. Choosing the wrong optimization target is a workload question, and workload questions are settled by executing the workload rather than by reading a datasheet. That is reproducible in 15–30 minutes on your own hardware with pip install lynxbench-ai (Personal Edition, the free non-commercial edition that has shipped).
Declaring the objective is not optional
The practical consequence of this tension is that every performance claim about an inference system needs to state what it’s optimizing for. A throughput number without a latency constraint is incomplete. A latency number without a throughput context is incomplete. A benchmark that just reports “tokens per second” without specifying the batch configuration, the concurrency model, and whether latency was constrained to anything is reporting a number you can cite but not one you can safely plan against.
As we discussed in the context of peak vs. steady-state behavior, the temporal regime of measurement matters too — a system’s throughput-latency trade-off can itself shift as the system transitions from peak to steady-state operation.
The organizations that navigate this well are the ones that declare their objective up front: “we optimize for p99 latency under this concurrency level” or “we optimize for sustained throughput with latency bounded below X.” That declaration constrains the design space and makes performance results interpretable. Without it, you’re optimizing a number without knowing whether the thing the number represents is the thing your users actually care about.
Declaring and defending that objective is the opening move of performance tuning for AI inference, where the throughput-versus-latency choice gets made against a real workload.
So the question to put to your own stack is not which metric is better, but which measurement you are missing: if the only number you hold is a saturated-throughput figure, what would you have to run — and at what batch policy — before you could defend a latency promise to a caller?
Frequently Asked Questions
Why do throughput and latency compete for the same resources in an AI inference system?
The mechanisms that raise throughput — larger batches, sustained GPU occupancy, amortised overhead — are the same mechanisms that add queueing and processing delay to any single request. Once you push past the system’s natural operating point, the relationship between the two often inverts: total tokens-per-second rises while per-request time gets worse. They share the GPU, the scheduler, and the memory subsystem, so a configuration tuned for one regime is structurally suboptimal for the other.
How does batch size reshape both throughput and latency together?
Batch size is the clearest lever for this trade-off on GPU-based inference. Larger batches amortise kernel launches, allocation, and framework bookkeeping across many requests, which pushes throughput up. But every request now waits for peers to arrive (queueing delay) and for the whole batch to finish (processing delay), which pushes latency up. There is no universally correct batch size — only a batch size that is correct for a declared objective.
What is the difference between model latency and end-to-end system latency in an inference benchmark?
Model latency is the forward pass on the device — input tensors on the GPU to output tensors on the GPU. System latency includes everything around it: request parsing, tokenization, batching policy decisions, queueing, memory management, detokenization, and transport back to the caller. In a well-optimised serving system, model latency is often only a fraction of end-to-end time, which is why measuring only the GPU pass can make the hardware look fast while users still wait.
When is throughput the right optimisation target, when is latency, and what does choosing wrong actually cost in an inference system?
Throughput is the right target for batch processing, offline inference, and cost planning; latency is the right target for interactive serving, real-time APIs, and any user-facing SLA. Choosing wrong costs you a system that looks correct by one measure and fails by another, usually surfacing in production rather than in the benchmark. Teams typically pick wrong by accident — optimising whatever the dashboard emphasises — and discover the mismatch when an SLO breaks or capacity planning misses.
What do TTFT (time-to-first-token), TPOT (time-per-output-token), and ITL (inter-token latency) each measure, and which throughput-vs-latency trade-off does each expose in LLM inference?
TTFT is how long the caller waits before the first token appears; it is dominated by prefill and queueing, so it worsens precisely as you batch harder for throughput. TPOT and ITL measure the steady cadence of generation once decoding begins, and they degrade as more sequences share a decode batch. Together they decompose perceived latency into a startup cost and a streaming cost — each a different face of the same competition between aggregate tokens-per-second and per-user responsiveness.
How should latency be reported across percentiles (p50, p90, p95, p99) so the tail behaviour of a latency-sensitive AI system is visible rather than hidden by an average?
Report a ladder — p50, p90, p95, p99, sometimes p999 — rather than a single summary number. p50 captures the typical experience, p90 and p95 show where the distribution begins to fan out, and p99 captures what the worst-served users actually live with. The gap between p50 and p99 is the real signal: a wide gap means the tail is heavy and load-sensitive even when the median looks healthy, which is exactly the behaviour an average obscures.
If a LynxBenchAI run reports a saturated-throughput number, what additional measurement would a reader need to run on the same hardware to get a tail-latency answer, and why can the throughput run not be re-read to supply it?
The throughput run raises batch size to saturation and counts completed iterations inside a timed window, so it records aggregate completions rather than a per-request distribution — there are no percentiles inside it to recover. A tail-latency answer needs a separate run at a fixed arrival rate or fixed concurrency, using the batch policy you would actually serve with, capturing the full per-request distribution. The two measurements sit at opposite ends of the trade-off, which is why one cannot be re-read as the other.
