A headline tokens/sec number is not a verdict on which model or serving stack you should deploy. It is one figure taken at a specific batch size, sequence length, precision, and concurrency — a snapshot of conditions that may share nothing with your prompt distribution or interactivity target. Read it as a promise about production behaviour under your load, and you will discover the gap the first time a real user waits on a first token. We see this pattern regularly in procurement conversations. A committee lands on a vendor page, reads “up to 2,400 tokens/sec,” and treats it as settled. The number is real. It is also almost never the number that governs the deployment being planned, because it was measured under a configuration chosen to make it large. How does an LLM inference benchmark actually work? An LLM inference benchmark drives a model-and-serving-stack combination with a fixed set of requests and measures how fast tokens come out. That sounds simple, but every word of the setup matters. The benchmark fixes an input sequence length (how long the prompts are), an output sequence length (how many tokens are generated), a batch size (how many requests are processed together), a numerical precision (FP16, INT8, FP8, or something lower), and a concurrency level (how many requests are in flight at once). It then reports timing. The important thing to internalise is that a benchmark is a measurement of a tuple, not of a model. The same Llama-family model on the same GPU will produce wildly different tokens/sec depending on whether it ran at batch size 1 with a 128-token prompt or at batch size 64 with a 2,048-token prompt. The model did not change. The conditions did. This is the same reasoning we lay out for hardware-level numbers in reading MLPerf and hardware inference benchmarks honestly for deployment — the benchmark measures the executor, and the executor includes the software stack, the compiler path (TensorRT-LLM, vLLM, or a raw PyTorch loop with torch.compile), and the batching policy, not just the silicon. So the first honest reading of any inference benchmark is: what tuple was this? If the page does not tell you the batch size, sequence lengths, precision, and concurrency, the tokens/sec figure is not decision-grade. It is marketing. What do TTFT, TPOT, and tokens/sec actually measure, and how do they trade off? There are three metrics that carry almost all the operational meaning, and collapsing them into one number is where naive reading goes wrong. Time-to-first-token (TTFT) is the latency from request arrival to the first output token. It is dominated by the prefill phase — the model processing the entire input prompt before it can emit anything. TTFT scales with input length: a 4,000-token prompt has a much longer TTFT than a 200-token prompt on the same stack. For anything interactive — a chat interface, a copilot — TTFT is what the user experiences as “responsiveness.” Time-per-output-token (TPOT), sometimes called inter-token latency, is the average time between subsequent tokens during the decode phase. It governs how fast the response streams once it has started. A model that starts quickly but decodes slowly will feel sluggish on long answers. Throughput (tokens/sec) is the aggregate token production rate across all concurrent requests. This is the metric that most directly maps to serving cost, because it tells you how many tokens one GPU can produce per second, and therefore how many GPUs you need for a given token volume. Here is the trade-off that a single tokens/sec figure hides: throughput and latency pull in opposite directions. You raise throughput by increasing batch size — packing more requests together so the GPU’s compute stays saturated. But larger batches increase queueing and per-request latency. So a benchmark run at batch size 128 will report a gorgeous tokens/sec number and a TTFT that no interactive user would tolerate. The number is honest for a batch-throughput deployment and misleading for an interactive one. Quick-answer: which metric governs which deployment? Deployment shape Governing metric Why Interactive chat / copilot TTFT, then TPOT User perceives responsiveness and stream speed; batch size stays low Batch document processing Aggregate tokens/sec Nobody waits per-request; cost per token dominates RAG with long context TTFT (prefill-heavy) Long input prompts make prefill the bottleneck Short-answer classification TPOT is near-irrelevant Output is a few tokens; TTFT and throughput dominate This is observed-pattern framing drawn from serving-stack deployments; the exact crossover points depend on your model size and hardware, but the direction of the trade-off is structural, not situational. Which config parameters must you check before trusting a tokens/sec number? Before a headline throughput figure earns a place in an evaluation, four parameters have to be visible and have to match — or be adjustable toward — your deployment. Batch size. A tokens/sec number at batch 64 does not transfer to a batch-1 interactive service. Ask what batch the figure was measured at, and whether the serving stack does continuous batching (as vLLM does) or static batching. Input / output sequence length. Prefill cost scales with input length; total throughput depends on the input:output ratio. A benchmark using 128-in / 128-out tokens will look very different from your 3,000-in / 300-out RAG workload. Precision. FP8 or INT8 quantisation can roughly double throughput versus FP16 on hardware that supports it, but precision is a first-class accuracy trade-off, not a free lunch. A tokens/sec number at a precision you cannot ship at is not your number. We treat this in detail in what extreme quantisation means for your procurement evaluation. Concurrency. Throughput under one request tells you nothing about behaviour under fifty simultaneous ones. Saturation, queueing, and p95 latency only appear under realistic concurrency. If any of these four is unstated, the honest move is to treat the number as an upper bound on a favourable configuration, not as a prediction for yours. That distinction — leaderboard number versus your number — is the whole subject of why the leaderboard number isn’t your number, and it is worth internalising before any procurement decision. How do you convert benchmark figures into per-request cost and p95 latency? This is where an inference benchmark stops being trivia and becomes evidence. The conversion is arithmetic, but it has to be grounded in your prompt distribution. Worked example (explicit assumptions) Assume a benchmark reports, for a given model-and-stack tuple at FP8: TTFT ≈ 180 ms at a 2,000-token input, batch 8 TPOT ≈ 22 ms per output token Aggregate throughput ≈ 3,600 tokens/sec at batch 32 Suppose your real workload averages 2,000-token inputs and 250-token outputs, and you care about interactive latency. Per-request latency ≈ TTFT + (output tokens × TPOT) = 180 ms + (250 × 22 ms) = 180 + 5,500 = ≈ 5.7 s end-to-end. If that exceeds your interactivity target, no throughput number saves you — you need a smaller model, more aggressive quantisation, or a change to output length. Per-request cost derives from throughput: at 3,600 tokens/sec and a 250-token average output, one GPU serves ≈ 14 completions/sec in batch mode. If a GPU-hour costs $C, per-completion cost ≈ C / (14 × 3,600). Plug in your cloud rate to get a real per-decision number. p95 latency cannot be read from a mean-based benchmark at all. You have to measure it under your concurrency, because the tail is where queueing lives. These are market-direction illustrative figures, chosen to show the arithmetic — not a benchmark of any specific product. The point is that TTFT and TPOT drive latency, throughput drives cost, and both must be recomputed against your input/output mix. A single tokens/sec figure supports neither calculation on its own. Why does interactive-vs-batch change which number matters? Because the two deployment shapes optimise for opposite ends of the throughput-latency curve, and a benchmark can only sit at one point on that curve at a time. An interactive assistant lives at low batch size, where per-request latency is short but aggregate throughput — and therefore cost efficiency — is poor. A nightly batch pipeline lives at high batch size, where cost per token is excellent but any individual request waits a long time. The same model on the same GPU is a good deployment for one and a bad deployment for the other, and the tokens/sec headline was measured at whichever point flattered the number. This is why the interactive-vs-batch distinction is not a footnote. It determines which of the three metrics is even relevant. Reading throughput for an interactive service, or TTFT for a batch pipeline, is reading the wrong axis of the same measurement. Where do LLM inference benchmarks belong in an evaluation pack — and where do they mislead? Inference benchmarks belong in the operational-feasibility and cost-per-decision sections of a procurement-grade evaluation. They tell you whether a model-and-serving-stack option can plausibly hit your latency target and what it will cost per request under your load. They do not belong in the task-accuracy or failure-mode sections — a fast model that gives wrong answers is not a good procurement outcome, and throughput says nothing about correctness. Used well, inference benchmarks bound the plausible latency-and-throughput envelope so a committee can discount options that cannot physically meet the requirement. Used naively, they get mistaken for a promise about production behaviour. The way we structure this in governance and trust evidence work is to keep operational metrics and accuracy metrics in separate sections that never substitute for each other. A serving-cost conversation is a reasonable entry point, but it feeds one part of the pack, not the verdict. For the vertical view of how these numbers land against real serving workloads, the applied procurement-eval methodology in how leaderboard scores do and don’t tell you what you need is the companion piece. How should model and serving-stack options be tabulated for a committee? Give the committee a like-for-like table where every column is fixed to your conditions, not the vendor’s. The value of the table is that it forces every option onto the same config. Option Precision TTFT @ your input len TPOT Throughput @ your batch Per-request cost Meets p95 target? Model A / vLLM FP8 measure measure measure derive yes/no Model A / TensorRT-LLM FP8 measure measure measure derive yes/no Model B / vLLM FP16 measure measure measure derive yes/no Two rules make this table honest. First, every row is measured (or re-measured) at your input/output lengths and concurrency, never copied from a vendor page at their config. Second, the “meets p95 target” column comes from a concurrency run, not from a mean — the tail is where deployments actually fail. This is the same discipline we apply when reading MLPerf results as procurement evidence: the leaderboard tuple is a starting hypothesis, not the answer. FAQ How does an LLM inference benchmark work? It drives a model-and-serving-stack combination with a fixed set of requests at a chosen batch size, sequence length, precision, and concurrency, then measures how fast tokens come out. In practice it measures a tuple, not a model — the same model produces very different numbers under different conditions. If the config is unstated, the number is not decision-grade. What do TTFT, TPOT, and tokens/sec actually measure, and how do they trade off? Time-to-first-token is latency to the first output token (dominated by prefill and input length); time-per-output-token is the streaming rate during decode; tokens/sec is aggregate throughput. Throughput and latency pull in opposite directions: you raise throughput with larger batches, which increases per-request latency. A single tokens/sec figure hides that trade-off. Which config parameters must you check before trusting a headline tokens/sec number? Batch size, input/output sequence length, precision, and concurrency. A number at batch 64 does not transfer to a batch-1 interactive service; a 128-token config does not predict a 3,000-token RAG workload; a precision you cannot ship at is not your number; and single-request throughput says nothing about behaviour under concurrency. If any of the four is unstated, treat the figure as an upper bound on a favourable config. How do you convert benchmark tokens/sec and latency figures into per-request cost and p95 latency? Per-request latency ≈ TTFT + (output tokens × TPOT), recomputed at your input/output lengths. Per-request cost derives from aggregate throughput and your GPU-hour rate. p95 latency cannot be read from a mean-based benchmark at all — you must measure it under your own concurrency, because the tail is where queueing lives. Why does the interactive-latency vs batch-throughput distinction change which number matters? The two shapes optimise for opposite ends of the throughput-latency curve. Interactive services live at low batch size where latency is short but cost efficiency is poor; batch pipelines live at high batch size where cost is excellent but individual requests wait. A benchmark sits at one point on that curve, so reading throughput for an interactive service — or TTFT for a batch pipeline — is reading the wrong axis. Where do LLM inference benchmarks belong in a procurement-grade evaluation pack, and where do they mislead? They belong in the operational-feasibility and cost-per-decision sections — bounding the plausible latency and cost envelope. They do not belong in the task-accuracy or failure-mode sections; throughput says nothing about correctness. Used naively they get mistaken for a promise about production behaviour under your load. How should model and serving-stack options be tabulated for a committee? Build a like-for-like table where every column is fixed to your input/output lengths, batch, precision, and concurrency, with each row measured (not copied from a vendor page) and a “meets p95 target” column drawn from a concurrency run rather than a mean. That forces every option onto the same config and lets the committee discount headline numbers whose setup does not match the deployment. Before you let any tokens/sec figure into an evaluation, ask the one question that separates evidence from marketing: at what input length, output length, batch size, precision, and concurrency was this measured — and does that tuple resemble the load you are about to put in front of real users?
