Someone on the platform team finds a Llama benchmark, reads “3,200 tokens per second on an H100,” and pastes it into the capacity model. That number then quietly becomes the basis for a GPU purchase order and a cost-per-request estimate the finance team will hold you to. The problem is not that the benchmark is wrong — it is that it answered a different question than the one you are asking. A Llama benchmark is a controlled measurement bounded by its batching policy, sequence length, quantisation, and runtime. It tells you what a model can do under one fixed condition. It does not tell you what your traffic mix, concurrency, and cache behaviour will produce. Treat the headline as your number and you will provision the wrong GPU or budget the wrong cost-per-request. The fix is not a better benchmark — it is reading the one you have correctly, then mapping it onto your own serving path. What does a Llama benchmark actually measure? When a repository or a blog post publishes “tokens per second for Llama 3 70B,” they ran a specific script under a specific configuration and reported the result. The configuration is the whole story. Every serious LLM benchmark holds a set of variables fixed, and the value you see is only valid inside that box. The variables that matter most: Batch size. A single request measured alone is decode-bound and leaves the GPU mostly idle between tokens. A benchmark run at batch size 32 packs the matrix multiplies and reports far higher aggregate throughput. These are not the same number, and a benchmark rarely leads with which one it used. Sequence length. Prefill (processing the prompt) and decode (generating the output) have completely different performance profiles. A benchmark using 128-token prompts and 128-token outputs will look nothing like your workload if you feed 8k-token documents into a summariser. Quantisation. FP16, INT8, FP8, and 4-bit GPTQ or AWQ weights all move different amounts of data across HBM. A quantised benchmark can post double the tokens per second of an FP16 run on identical hardware — and a small, often unreported, accuracy cost. Runtime and kernel stack. The same model on the same GPU behaves differently under vLLM, TensorRT-LLM, and SGLang. Continuous batching, paged attention, and kernel fusion are runtime properties, not model properties. A number without a named runtime is barely a number at all. The first honest question to ask of any Llama benchmark is: which of these did they fix, and at what value? If the answer is not in the report, the number is decorative. Why is my production tokens-per-second lower than the benchmark? This is the most common gap we see, and it is almost never a bug. It is the difference between a saturated batch and real traffic. A benchmark maximises throughput by keeping the GPU busy — it feeds a large, uniform batch so the accelerator never waits. Your production traffic does the opposite. Requests arrive at irregular intervals, with wildly different prompt lengths, and users expect a response before the batch is full. A serving stack that waits to assemble a benchmark-sized batch would blow your p95 latency, so it doesn’t wait. It runs smaller, less efficient batches, and your effective throughput drops accordingly. Two structural effects, invisible to both the benchmark and to an APM dashboard, drive most of the divergence: Batching policy under real arrival patterns. Continuous batching (as implemented in vLLM and SGLang) lets new requests join an in-flight batch between decode steps, which recovers a lot of efficiency. But how much it recovers depends entirely on your concurrency and how bursty your traffic is. The benchmark assumed a full queue; you have a queue that empties and refills. The GPU profiling work that resolves kernel-time and batching detail is what makes this observable — a headline tokens-per-second figure hides it entirely. KV-cache behaviour. The key-value cache grows with sequence length and consumes HBM that would otherwise hold more concurrent requests. A benchmark with short sequences barely touches this ceiling. If your prompts are long, or you run large context windows, the cache limits how many requests fit on the card at once — which caps your real batch size below whatever the benchmark used. The trade-offs here are worth understanding on their own; we cover them in what LLM context windows cost at inference. Neither of these shows up on a Prometheus panel labelled “GPU utilisation.” Utilisation can read 90% while your cost-per-request is twice what the benchmark implied, because the GPU is busy doing small, inefficient batches. Reading a Llama benchmark: a diagnostic checklist Before you let any published Llama number into a capacity or cost model, run it through this. Each row is a condition the benchmark fixed that your deployment may not match. Condition What to check in the benchmark Why it moves your number Batch size Fixed batch, or throughput at saturation? Saturated throughput overstates single-request latency and understates real cost per token Sequence length Prompt vs output token counts stated? Long prompts shift work to prefill and shrink your usable batch via KV-cache pressure Quantisation FP16 / FP8 / INT8 / 4-bit, and accuracy delta? Quantised throughput is not comparable to FP16 throughput on the same card Runtime vLLM, TensorRT-LLM, SGLang, llama.cpp? Continuous batching and paged attention are runtime features, not model features Metric Aggregate tokens/sec, or per-request TTFT + inter-token? Aggregate hides the latency your users actually feel Hardware Exact GPU, count, and interconnect (NVLink/PCIe)? Multi-GPU tensor parallelism changes the scaling story entirely If a benchmark leaves three or more of these unstated, do not put its number in your model. Find one that reports its conditions, or measure your own. How do I translate benchmark throughput into cost-per-request? This is the step that turns a lab number into something the business can plan against. The mechanics are straightforward once you stop treating tokens per second as a single scalar. Start with a worked example, with the assumptions stated explicitly: Assumptions (illustrative): A benchmark reports 3,200 tokens/sec aggregate for Llama 3 70B on a single H100, batch size 32, 512-token prompts, 256-token outputs, FP8, under vLLM. Your traffic averages 1,800-token prompts, 400-token outputs, and effective concurrency of 8 during business hours. Your prompts are ~3.5× longer than the benchmark’s, which pushes far more work into prefill and enlarges each request’s KV-cache footprint. Your effective concurrency of 8 is a quarter of the benchmark’s batch of 32, so continuous batching cannot pack the GPU as tightly. In configurations like this we typically see effective throughput land well below the headline — often a third to a half of it once longer sequences and lower concurrency are accounted for (observed pattern across our inference engagements; not a benchmarked rate). Take, say, 1,200 tokens/sec effective. From there the arithmetic is honest: at a known GPU hourly price, effective tokens/sec gives you a cost per 1,000 tokens, and your average output length gives you a cost per request. The unit-economics framework that converts throughput into a cost-per-request KPI is what makes that mapping defensible rather than a guess — a raw benchmark number cannot express it. The point is that the correction factor — not the headline — is the thing you provision against. Three claims worth extracting from all of this: Aggregate tokens-per-second at benchmark saturation systematically overstates the throughput a latency-bounded production service will achieve, because production cannot run saturated batches. (observed-pattern) KV-cache growth with sequence length caps the concurrent batch a GPU can hold, so long-prompt workloads see a lower effective throughput than short-prompt benchmarks predict on identical hardware. (observed-pattern) GPU utilisation percentage does not track cost-per-request; a card can read high utilisation while running small, inefficient batches at poor cost efficiency. (observed-pattern) When should you stop trusting the benchmark and profile your own path? A benchmark is a fine starting hypothesis. It stops being trustworthy the moment a real decision — a purchase, a budget, an SLO commitment — rests on the assumption that its conditions match yours. The tell is when you cannot answer, from your own data, what your effective batch size and prompt distribution actually are. At that point the useful move is to profile the serving path you will actually run: measure time-to-first-token and inter-token latency under your real concurrency, watch the KV-cache pressure, and see where kernel time goes. This is exactly the work an [AI Inference Cost Audit](Inference Cost-Cut Pack) does — it profiles your serving path so a benchmark’s lab number translates into a cost-per-request you can actually stand behind, expressed in the p95-latency, cost-per-request, and GPU-utilisation terms the business already tracks. The benchmark gives you the ceiling; the audit gives you the number. If you are choosing a runtime rather than a GPU, the same discipline applies at the software layer — comparing how a llama.cpp benchmark reads across CPU and GPU inference is a separate exercise from comparing hardware, and conflating the two is a common source of provisioning error. FAQ How should you think about llama benchmark in practice? A Llama benchmark runs a fixed script that generates tokens under one specific configuration — a set batch size, sequence length, quantisation format, and runtime — and reports the resulting throughput, usually as tokens per second. In practice it means “this model can produce this many tokens per second under exactly these conditions.” It is a controlled ceiling measurement, not a prediction of what your deployment will achieve. What conditions does a Llama benchmark hold fixed, and why does that matter for my deployment? The main fixed conditions are batch size, prompt and output sequence length, quantisation (FP16, FP8, INT8, 4-bit), and the serving runtime (vLLM, TensorRT-LLM, SGLang, llama.cpp). They matter because each one changes the number materially — a saturated batch, short prompts, and aggressive quantisation all inflate throughput relative to a latency-bounded production service with long prompts and modest concurrency. If a benchmark leaves these unstated, its number cannot be trusted in a capacity model. Why does the tokens-per-second I see in production differ from the published Llama benchmark number? Benchmarks maximise throughput with large, uniform batches that keep the GPU saturated; production traffic arrives irregularly with varied prompt lengths and latency limits, so it runs smaller, less efficient batches. KV-cache growth with sequence length also caps how many requests fit on the card, further reducing effective throughput. Both effects are invisible to APM dashboards, which is why utilisation can read high while real cost-per-request is far worse than the benchmark implied. How do I translate a Llama benchmark’s throughput into cost-per-request and p95 latency for my own traffic? Correct the benchmark’s aggregate tokens/sec down to an effective rate that reflects your real concurrency and prompt distribution, then combine that with GPU hourly cost and your average output length to derive cost per request. p95 latency comes from measuring time-to-first-token and inter-token latency under your actual arrival pattern, not from the benchmark. The correction factor — not the headline number — is what you provision against. When should you stop trusting a benchmark and profile your own serving path with an inference cost audit? Stop trusting it the moment a real decision — a GPU purchase, a budget, an SLO — depends on the assumption that the benchmark’s conditions match yours, and you cannot confirm your own effective batch size and prompt distribution. At that point, profiling your serving path with an AI Inference Cost Audit converts the lab number into a defensible cost-per-request in the p95-latency and utilisation terms the business tracks. What benchmark conditions should I match to make a fair GPU or model comparison for Llama inference? Fix the same batch size, sequence length, quantisation format, and runtime across every candidate, and report per-request latency alongside aggregate throughput. Comparing a quantised run on one card to an FP16 run on another, or a saturated batch to a single-request measurement, produces a meaningless ranking. A fair comparison holds every serving-boundary variable constant except the one thing you are actually choosing between. How do batching and cache behaviour change my real Llama inference performance? Batching policy determines how efficiently the GPU is packed: continuous batching recovers efficiency by letting new requests join in-flight batches, but only as much as your concurrency and burstiness allow. KV-cache behaviour ties up HBM proportional to sequence length, so long prompts shrink the concurrent batch a card can hold and lower effective throughput. Neither appears on a standard utilisation panel, which is why both are common sources of the benchmark-to-production gap. The next time a Llama benchmark lands in a capacity discussion, the right first move is not to argue about the number — it is to ask which serving-boundary conditions produced it, and whether any of them survive contact with your traffic. That question, not the headline, is what decides whether you buy the right GPU.
