Throughput in AI inference is the amount of useful work a serving system delivers per unit of time: for large language models it is usually reported as tokens per second across all concurrent requests, and for other models as requests, queries or images per second. A throughput figure only describes a real deployment when it comes with the latency limit it was achieved under, because the same hardware can post a very different number once requests have to finish in time.
The rest of this page covers the definition, the units, why the latency bound belongs inside the definition rather than in a footnote, and what to settle about your own workload before treating any throughput figure as your target.
Throughput in AI inference: the units and the formula
The general definition is a rate: tasks completed per unit of time. For batched inference it can be written as Throughput = B / L, where B is the batch size and L is the average latency of processing that batch. The formula shows that throughput depends on both the batch size and the latency of each batch.
LLM serving needs a more specific unit. MLCommons chose tokens per second as the throughput metric for its Llama 2 70B benchmark because two LLM queries can carry very different amounts of work depending on their input and output token lengths, which makes queries per second a poor measure on its own. Tokens per second is often split again into prefill (prompt processing) and decode (generation), since the two phases hit different bottlenecks. Prefill is compute-bound, while decode processes one token per request at a time and is memory-bound.
There is also the question of whose throughput is being counted. NVIDIA’s benchmarking documentation separates total tokens per second per system, counted across all simultaneous requests, from tokens per second per user, which is one request’s output length divided by its end-to-end latency. As concurrency rises, the system figure goes up while the per-user figure goes down. Both are “throughput”, so compare figures only when they count the same thing.
If you are coming at this from the processor side rather than the serving side, our piece on processor throughput in AI inference covers what the hardware-level figure does and does not translate into.
Throughput forms at a glance
| Form | What it counts | Typical unit | What it leaves out |
|---|---|---|---|
| Total tokens per second (per system) | Output tokens across all concurrent requests over a measurement window | tokens/s | How fast any single user is served; rises with concurrency until the GPU saturates |
| Tokens per second per user | One request’s output length divided by its end-to-end latency | tokens/s | Aggregate capacity; it falls as concurrency rises |
| Requests per second | Completed requests per measurement interval | req/s | Work per request, which depends on the prompt and output length mix and on batching and admission policy |
| Batch throughput (B / L) | Batch size divided by average batch latency | images/s, items/s | The latency each individual item experienced |
| Capacity (latency-bounded throughput) | Maximum load sustained while meeting a latency target | queries/s | Nothing essential, provided the target and percentile are stated |
| Goodput | Requests per second completed within the predefined SLOs | req/s | By construction never exceeds raw throughput |
The last two rows are the forms that build the latency constraint in.
Why is a tokens-per-second figure meaningless without a latency target?
A throughput figure without a latency target says little about a latency-constrained deployment. MLPerf Inference formalised this by treating latency-bounded throughput as the measure of data-centre inference performance, on the grounds that a system can deliver excellent throughput yet perform poorly once latency constraints apply.
The measured gap varies widely by system. In the MLPerf Inference paper, moving from the offline scenario (no latency constraint) to the server scenario (random Poisson arrivals and a latency bound) cost some systems as little as 3% of their throughput and others around 50%. For the translation model the server-scenario reduction across five systems was 39–55%, while for ResNet-50 v1.5 it ranged from 3% to 35%. The authors attribute the loss to suboptimal batching under the latency constraint, and conclude that a comparison with unconstrained latency has little bearing on a latency-constrained scenario.
An AI inference accelerator may stand out in one scenario while underperforming in another. Because the size of the loss differs by system and by model, a device that wins the offline comparison is not guaranteed to win the bounded one.
LLM serving research also measures throughput under latency targets. Sarathi-Serve defines its throughput metric, capacity, as the maximum queries per second a system can sustain while meeting latency targets, and reports that Orca and vLLM violate the P99 time-between-tokens SLO before reaching their maximum serviceable throughput. Under a strict 100 ms P99 time-between-tokens SLO on Mistral-7B, Sarathi-Serve reported 3.5× the capacity of vLLM.
Goodput pushes the idea one step further. It counts requests per second completed while meeting predefined SLOs on time to first token, end-to-end latency and timeouts, so it excludes errors, timeouts and SLO-violating responses and can never exceed raw throughput. The DistServe authors define per-GPU goodput as the maximum request rate served while meeting an SLO attainment goal such as 90%, and note that higher per-GPU goodput translates directly into lower cost per query.
When someone quotes a throughput figure, ask which latency target it was held to and which workload it was measured on.
Batching is the mechanism that links the two
Batching groups requests so they are processed together, which exploits accelerator parallelism and raises throughput and utilisation. The cost is higher latency. Databricks reported that on one NVIDIA A100, maximising throughput with a batch size of 64 raised throughput by 14× while increasing request latency by 4×. Past a certain batch size the system crosses into the compute-bound regime, where every doubling of batch size adds latency without adding throughput.
How batch size reshapes throughput and latency together, and how that changes which one you should optimise, is a full topic of its own; our article on choosing the wrong optimisation target between throughput and latency works through the decision and its costs. What matters for the definition is that a throughput figure belongs with the batch size it was measured at.
The latency terms a throughput figure is bounded by
LLM serving uses a small set of latency metrics, each separating a different kind of slowness. Time to first token (TTFT) runs from request acceptance to the first output token, and includes queueing and prefill. Time per output token (TPOT), also called inter-token latency (ITL), is the average gap between consecutive output tokens after the first. End-to-end latency spans the whole request, and a low TTFT does not imply a low end-to-end time, because slow generation can still dominate. The task force that chose tokens per second, for example, also set latency constraints of TTFT at most 2 seconds and TPOT at most 200 milliseconds. How these metrics trade off against throughput in a live system is covered in the throughput versus latency article.
A saturation run measures throughput well but not tail latency, which needs a measurement of its own. The Lancet authors put it plainly: throughput is easy to measure, while tail latency is harder to capture in a statistically meaningful way because it depends on factors beyond the workload itself.
Which throughput figure fits your workload?
The right throughput figure depends on who consumes the output and when. ClickHouse’s guidance is that TTFT matters most for anything a human reads as it streams, while agents, pipelines and batch jobs act only on the finished completion, so throughput and end-to-end latency win there. The Splitwise authors make the same split: for batch tasks such as summarisation throughput matters more than TTFT or time between tokens, whereas conversational APIs need tight SLOs on both. Within interactive use the emphasis still varies: DistServe notes chatbots prioritise low TTFT, while document summarisation emphasises low TPOT.
Before choosing a target, characterise the workload. Databricks recommends spending time on expected input and output lengths before anchoring on a latency target. NVIDIA’s guidance makes the mechanism explicit: longer inputs raise TTFT through the prefill stage, and longer outputs raise ITL through the generation stage, so the distribution of both should be understood before optimising hardware utilisation. Even the batching sweet spot has to be measured; a study of CNN inference across mobile and desktop hardware found the optimal batch size difficult to estimate without actual measurements, because adding one image to a batch sometimes improved throughput cheaply and sometimes worsened both throughput and latency. The same study recorded 850 images per second on a GTX 1080 against up to 4,000 on a Titan V at a 100 ms latency limit, but also very high latency variation on the Titan V, which it flagged as a problem for systems with latency requirements.
Our position is that choosing throughput as the optimisation target should come from profiling your workload. The same holds for benchmarks: a useful GPU benchmark maps to your model mix, prompt distribution, batching rules and latency targets rather than producing the highest number on a chart. Published hardware results deserve the same reading, and LynxBenchAI is the TechnoLynx benchmark for AI hardware. For the gap between rated compute and delivered throughput, our explainer on why identical TOPS deliver different throughput covers the stack effects. Once the workload is characterised, sizing a fleet against a bounded throughput figure is the next step, covered in production capacity planning for AI inference fleets.
A chat product, a nightly document job and an agent loop on the same model can each justify a different latency target, and therefore a different throughput figure from the same hardware.
