A model feels slow in production. Someone runs llama-bench, reads one tokens-per-second figure, and declares a verdict — the model is too big, the hardware is too weak, time to swap. That single number is the problem. It folds prompt processing and token generation into one average, hides the quantisation level, and says nothing about whether threads or GPU offload were even configured. Read blind, it invites a model swap you may not need. Read against the bottleneck an audit named, it tells you whether a runtime port, a requantisation, or a hardware move is the cheaper fix.

llama.cpp ships with llama-bench, a purpose-built harness that runs a model under controlled conditions and reports throughput. It is genuinely useful — far more so than timing a chat session with a stopwatch — but only if you read it as a decomposed profile rather than a single verdict. The distinction between the two readings is the whole argument of this article.

What does llama-bench actually measure?

llama-bench reports two throughput figures, and treating them as one is the first mistake. Prompt-eval throughput (pp) measures how fast the model ingests the input prompt; token-generation throughput (tg) measures how fast it emits new tokens. These are different computational regimes with different bottlenecks, and a port that fixes one may do nothing for the other.

Prompt evaluation is compute-bound and parallel. The whole prompt is processed in large matrix multiplications, so it scales well with FLOPS and with batch width — a GPU or a wide SIMD path helps a lot here. Token generation is the opposite: it is memory-bandwidth-bound and sequential. Each new token requires reading the full set of model weights from memory before a single token comes out, so generation throughput tracks how fast you can stream weights, not how many FLOPS the device claims. This is why a machine that rips through a 2,000-token prompt can still feel sluggish when it starts typing the answer.

The practical consequence: if your users complain about time to first token, you are looking at prompt-eval throughput, and compute or context length is the lever. If they complain that responses stream slowly, you are looking at generation throughput, and memory bandwidth or quantisation is the lever. We see teams optimise the wrong half of this all the time — buying a faster GPU to fix a generation bottleneck that a lower quant would have solved for free. When memory bandwidth turns out to be the wall, the reasoning in our note on when memory bandwidth is the bottleneck in AI inference applies directly.

How do quantisation levels change the numbers?

Quantisation is the lever llama-bench makes most visible, because you can rerun the same model at Q4, Q5, and Q8 and watch generation throughput move. Lower-bit quants store weights in fewer bytes, so there are fewer bytes to stream per token — and since generation is bandwidth-bound, generation throughput usually rises as you drop from Q8 to Q5 to Q4. The trade-off is accuracy: aggressive quantisation degrades output quality, and how much depends on the model and the task.

The mistake is reading the throughput gain without measuring the quality cost. A Q4 build that generates 40% more tokens per second (an illustrative figure — measure your own) is only a win if your evaluation set still passes. Quantisation is a first-class trade-off, not a free speed knob, and the benchmark only shows you one side of it.

Reading a quant sweep — worked example

Assume a 7B model on a single mid-range GPU with 24 GB VRAM, all layers offloaded, batch size 1, generation of 256 tokens. The numbers below are illustrative, not benchmarked — the shape is what matters, not the magnitude:

Quant	Weight size	tg tok/s (illustrative)	Fits in VRAM?	Quality delta
Q8_0	~7.5 GB	baseline	yes	reference
Q5_K_M	~4.8 GB	~+20%	yes	small drop
Q4_K_M	~4.0 GB	~+35%	yes	measurable drop

The read: if all three fit in VRAM and generation is your bottleneck, Q5_K_M is often the pragmatic middle — most of the bandwidth win, modest quality cost. If the model doesn’t fit at Q8 and spills to system RAM, the throughput cliff between fitting and not fitting dwarfs the quant-to-quant difference. That cliff is a hardware-or-quant decision the single aggregate number would have buried.

How do GPU offload, threads, and batch size affect the read?

Three configuration knobs move llama-bench numbers independently, and isolating them is the difference between a decision-grade profile and noise.

GPU offload (-ngl) controls how many transformer layers run on the GPU versus the CPU. Partial offload — some layers on GPU, the rest on CPU — creates a hybrid path where a PCIe transfer sits between the two, and that transfer can dominate. A benchmark that offloads 20 of 32 layers tells you something very different from one that offloads all 32. Whether the GPU offload figure reflects a real kernel or hardware bottleneck worth porting for is exactly the question our GPU profiling methodology is built to answer; llama-bench gives you the symptom, the profiler gives you the cause.

Thread count matters on the CPU path. More threads help prompt evaluation up to the point of memory-bandwidth saturation, after which they add contention and can slow generation. There is usually a sweet spot near the physical core count, not the logical thread count.

Batch size widens prompt evaluation. It lifts prompt-eval throughput substantially because the matrix multiplications get bigger and more efficient, but it does little for single-stream generation. If you are serving one request at a time, a batch-size benchmark flatters prompt-eval in a way that will not show up in your p95.

The discipline is to change one variable at a time and record the pair (pp, tg) for each. llama-bench accepts lists for these parameters and emits a row per combination, which is exactly the decomposition you want. A single run with defaults is a snapshot; a grid is a profile.

How do I turn the results into a port decision?

This is where the benchmark stops being a curiosity and becomes evidence. The goal is not the highest tokens-per-second — it is closing the gap between what you measure and the target the audit set: a p95 latency ceiling, or a cost-per-request budget. Those unit economics are the KPIs the tokens/sec figures must ultimately improve, and framing them is the job of the inference unit-economics framework.

Decision rubric — which lever the benchmark points to

What the benchmark shows	Named bottleneck	Cheaper fix before a model swap
Low tg, weights spill to RAM	Memory capacity / bandwidth	Requantise down (Q5/Q4) or add VRAM
Low tg, weights fit, bandwidth saturated	Memory bandwidth	Faster memory or hardware move
Low pp, high tg	Compute / context length	GPU offload, batch, or shorter prompts
Both low, CPU-only	Runtime / instruction path	Port to a GPU runtime or wider SIMD build
Numbers fine, p95 still misses	Serving-path overhead, not the model	Fix the runtime, don’t touch the model

The last row is the one that saves the most money. When the raw throughput is healthy but production latency still misses target, the model is not the problem — the serving path is. Swapping the model there is expensive motion that fixes nothing. A llama-bench grid that isolates (pp, tg) across quant and offload settings is the before-evidence a port needs; the same figures re-measured after the change are the after-evidence that the port actually recovered latency headroom without retraining or replacing the model.

This is precisely the sequence our [inference cost-cut audit](Inference Cost-Cut Pack) runs before any porting work begins: profile, name the bottleneck, then let the benchmark say whether a runtime, quantisation, or hardware change is warranted. If the audit hasn’t named the bottleneck, the benchmark number is a verdict without a defendant. You can see the broader engagement shape on our services overview.

FAQ

What does working with llama-cpp benchmark involve in practice?

llama-bench runs a model under fixed conditions and reports throughput in tokens per second. In practice it means little as a single number — its value is in running a grid across quant levels, thread counts, GPU offload, and batch size, then reading the resulting profile against a production target rather than treating any one figure as a verdict on the model or hardware.

What does llama-bench actually measure — prompt-eval throughput versus token-generation throughput — and why does the distinction matter?

It measures two separate figures: prompt-eval throughput (pp), how fast the input is ingested, and token-generation throughput (tg), how fast new tokens are emitted. The distinction matters because prompt evaluation is compute-bound and generation is memory-bandwidth-bound — they have different bottlenecks, so a fix for one may do nothing for the other. Time-to-first-token complaints map to pp; slow-streaming complaints map to tg.

How do quantisation levels (Q4, Q5, Q8) change the benchmark numbers, and what trade-off do they reveal?

Lower-bit quants store weights in fewer bytes, so fewer bytes stream per token and generation throughput typically rises from Q8 to Q5 to Q4. The trade-off is accuracy: aggressive quantisation degrades output quality by an amount that depends on the model and task. The throughput gain is only a real win if your evaluation set still passes at the lower quant.

How do GPU offload, thread count, and batch size affect the benchmark, and how do I isolate their effect?

GPU offload (-ngl) shifts layers between CPU and GPU and can introduce a dominating PCIe transfer at partial offload; thread count helps prompt-eval until memory bandwidth saturates; batch size lifts prompt-eval but barely helps single-stream generation. Isolate each by changing one variable at a time and recording the (pp, tg) pair per combination — llama-bench emits a row per parameter combination, which is the decomposition you want.

How do I turn llama.cpp benchmark results into a decision about whether to port the runtime, requantise, or move hardware?

Read the profile against the named bottleneck: weights spilling to RAM point to requantisation or more VRAM; saturated bandwidth with weights fitting points to a hardware move; low prompt-eval with healthy generation points to offload or batching; both low on CPU-only points to a runtime or SIMD port. If throughput is healthy but p95 still misses, the serving path — not the model — is the problem.

How do I compare llama.cpp benchmark numbers against a target like p95 latency or cost-per-request?

Translate throughput into the serving metric the audit baselines: p95 latency per request and throughput per dollar across quant levels and hardware targets. The benchmark’s tokens/sec become meaningful only when mapped to whether they close the gap to that target, which is the job of an inference unit-economics framework rather than the raw benchmark alone.

What benchmark mistakes make the numbers misleading enough to trigger an unnecessary model swap?

Reading a single aggregate number that folds pp and tg together; benchmarking with a batch size larger than production serves, which flatters prompt-eval; ignoring which quant level produced the figure; and reporting throughput without checking whether production latency actually misses because of the serving path rather than the model. Each of these can make the model look like the culprit when a cheaper runtime, quant, or configuration change was the real fix.

The honest closing question is not “which model is fastest” — it is whether the bottleneck the benchmark exposes is the one your audit named. A tokens-per-second figure without a bottleneck to answer to is a verdict with no defendant, and porting on it spends money to move a wall you never measured.