Run llama-bench once, read a tokens/second number, and ship it as “the model’s performance” — and you have measured almost nothing you can act on. A llama.cpp benchmark is a measurement of the whole serving path under a fixed configuration, not a property of the model. Change the quantisation level, the thread count, the batch size, the context length, or the backend, and the number moves more than swapping the model itself would. Any one of those five levers can double or halve the figure you just recorded.

That is the trap most teams walk into when they type llama cpp benchmark into a search bar looking for a quick answer. The number is real, but it is not comparable to anything — not to your last run, not to a colleague’s laptop, not to the box you plan to deploy on — unless the configuration that produced it is pinned and reported alongside it. The whole value of the benchmark is reproducibility. A benchmark that tells you which lever moved the number is a tuning tool. A bare tokens/second figure is a screenshot.

What does llama-bench actually measure?

The single most common reason a llama.cpp benchmark misleads a decision is that people treat it as one number when it is two. llama-bench reports prompt-processing throughput (pp) and token-generation throughput (tg) separately, and they have different bottlenecks.

Prompt processing is the prefill phase: the model ingests your input tokens in parallel. It is compute-bound. On a GPU it saturates the matrix-multiply units; on a CPU it leans on SIMD width and core count. A large prompt processed in one batch runs at high throughput because the hardware stays busy.

Token generation is the decode phase: the model emits one token at a time, and each step must read the full weight tensors back from memory. It is memory-bandwidth-bound. This is why generation throughput on a CPU often collapses relative to prefill, and why a quantised model — smaller weights, fewer bytes to move per token — speeds up generation far more than it speeds up prefill. If you report a single “tokens/second” and do not say which phase it came from, the reader cannot tell whether you measured a compute ceiling or a bandwidth ceiling. Those point to opposite tuning moves.

We see this conflation constantly in reviews. Someone quotes 40 tokens/second, another team quotes 8, and they are both right — one measured prefill on a long prompt, the other measured decode on a short one. They were never measuring the same thing.

Which configuration variables must be pinned?

For a llama.cpp benchmark to be reproducible — and therefore actionable — five variables have to be fixed and reported. Leave any of them implicit and the number becomes anecdote.

Variable	llama-bench flag	Why it moves the number
Quantisation level	(baked into the GGUF file)	Q4 vs Q8 changes bytes-per-weight, which dominates decode throughput and memory footprint
Thread count	-t	On CPU, throughput scales with cores until memory bandwidth saturates, then flattens or regresses
Batch / ubatch size	-b, -ub	Larger prefill batches raise prompt-processing throughput; too large and latency and memory spike
Context length	-p, -n	Longer context grows the KV cache, adds attention cost, and raises the memory footprint
Backend	build flag (CPU / Metal / CUDA)	The kernels and the memory hierarchy are different; a Metal number tells you nothing about a CUDA box

The claim class matters here: everything above is an observed-pattern from serving-path reviews, not a published benchmark rate — the direction each lever moves is reliable, but the magnitude depends entirely on your hardware and model. The point is not the exact numbers; it is that a benchmark which omits any of these five columns cannot be compared to another benchmark, because you cannot tell what changed between them.

A useful discipline: capture llama-bench’s own configuration echo (it prints the build, threads, and model) into the same log as the result, and treat the pair as a single record. If you want to understand why a given thread count stops helping, our note on processor throughput in AI inference explains where the CPU ceiling actually sits.

How do quantisation levels trade throughput, memory, and accuracy?

Quantisation is usually the first lever teams reach for, and it is the one most often misread. Dropping from Q8 to Q4 roughly halves the weight footprint. Because decode is memory-bandwidth-bound, that smaller footprint tends to raise token-generation throughput meaningfully — you are moving fewer bytes per token. Prefill throughput moves less, because prefill is compute-bound rather than bandwidth-bound.

The cost is accuracy. Lower-bit quantisation introduces representation error, and how much that matters is task-dependent: a summarisation workload may be indistinguishable at Q4, while a code-generation or structured-extraction task can degrade in ways a throughput benchmark will never show you. This is the failure mode to name explicitly — optimising the benchmark number while silently regressing output quality. The benchmark measures speed and memory; it does not measure whether the answers are still correct. Any quantisation decision has to carry an accuracy check on your actual task alongside the tokens/second figure.

That is a precision-versus-outcome trade-off, and it is the same shape whether you are quantising an LLM or porting numerics on a serving path. The framing in reading llama.cpp benchmark numbers before you port an LLM treats that trade-off from the porting angle; here the point is simply that memory and accuracy sit on opposite sides of the quantisation dial.

How do CPU, Metal, and CUDA backends compare fairly?

The honest answer is that they do not compare in the abstract — only on the same workload with the same reported configuration. A CUDA build with weights and KV cache resident in HBM will generally dominate a CPU build on generation throughput, because HBM bandwidth is an order of magnitude above system DRAM and the CUDA kernels are tuned for it. Metal sits in between, with unified memory that removes the host-to-device copy but delivers less raw bandwidth than a discrete data-center GPU.

To benchmark them fairly you fix everything you can hold constant: same GGUF file (same quantisation), same prompt and generation length, same context, and then report each backend’s pp and tg separately. What you must not do is compare a CPU decode number against a CUDA prefill number and conclude the GPU is “5x faster” — you would be comparing two different phases on two different hardware paths. Attribute each number to a phase and a backend, and the comparison becomes decision-grade instead of rhetorical.

Attributing a benchmark number to a bottleneck rather than reading it in isolation is exactly what disciplined GPU profiling is for; the same profiling logic that separates a compute-bound prefill from a bandwidth-bound decode applies whether the kernel runs on CUDA, Metal, or a CPU SIMD path. When decode throughput refuses to improve no matter how you tune threads, you are almost certainly bandwidth-bound — the pattern described in memory-intensive applications in AI inference.

How do you read the numbers to pick the next tuning lever?

Once you have a reproducible baseline — prefill throughput, generation throughput, p95 latency at your target context, and memory footprint per quantisation level — the benchmark stops being a scoreboard and becomes a ranking tool. The question is no longer “how fast is it?” but “which change returns the most throughput or cost saving per unit of effort?”

Quick diagnostic: which lever to pull next

• Generation throughput is your pain, memory headroom exists → try a lower quantisation level first (biggest bandwidth win), then re-check accuracy on your real task.
• Generation throughput is your pain, quantisation already aggressive → you are bandwidth-bound; a backend switch to a higher-HBM-bandwidth GPU is the lever, not more CPU threads.
• Prefill throughput is your pain → raise batch/ubatch size and, on CPU, thread count — until throughput stops scaling, which marks the saturation point.
• p95 latency spikes at long context → the KV cache is the cost; consider context trimming or a backend with more memory bandwidth before touching quantisation.
• Memory footprint is the ceiling → quantisation and context length are your two dials; benchmark both before committing hardware.

Each of those is a hypothesis the benchmark can confirm or reject in a single measured run. That is the entire discipline: never guess which change helped when you can measure it. The baseline is what lets you tell a real gain from noise.

FAQ

How does llama cpp benchmark actually work?

A llama.cpp benchmark, typically run with llama-bench, measures how fast a given model file runs on a given machine under a specific configuration — quantisation, threads, batch size, context length, and backend. In practice it measures the whole serving path, not the model in isolation, so the number is only meaningful when reported together with the configuration that produced it.

What does llama-bench actually measure — prompt-processing throughput vs token-generation throughput — and why are they different?

llama-bench reports two separate numbers: prompt-processing (prefill) throughput and token-generation (decode) throughput. Prefill is compute-bound because it ingests input tokens in parallel; decode is memory-bandwidth-bound because each generated token requires re-reading the weights. They point to opposite tuning moves, so quoting one combined figure is the most common way a benchmark misleads a decision.

Which configuration variables must be pinned and reported for a benchmark to be reproducible?

Five: quantisation level, thread count, batch/ubatch size, context length, and backend (CPU, Metal, or CUDA). Each of these moves the result more than swapping the model would, so a benchmark that omits any of them cannot be compared to another run because you cannot tell what changed.

How do I read llama.cpp benchmark numbers to decide which tuning lever to pull next?

Establish a baseline of prefill throughput, generation throughput, p95 latency at your target context, and memory footprint per quantisation level. Then match the symptom to the lever: bandwidth-bound decode points to quantisation or a higher-bandwidth backend; compute-bound prefill points to batch and thread tuning; long-context latency points to the KV cache. Each is a hypothesis a single measured run can confirm.

How do quantisation levels (e.g. Q4 vs Q8) trade off throughput, memory, and accuracy in a llama.cpp benchmark?

Q4 roughly halves the weight footprint versus Q8, which raises memory-bandwidth-bound generation throughput and lowers memory use, while prefill throughput moves less. The cost is accuracy, which degrades in task-dependent ways a throughput benchmark cannot see — so every quantisation decision needs an accuracy check on your real workload alongside the speed number.

How do CPU, Metal, and CUDA backends compare, and how do I benchmark them fairly on the same workload?

They compare only on identical workloads with the same reported configuration — same GGUF file, prompt, generation length, and context — with each backend’s prefill and generation numbers reported separately. CUDA generally leads on decode because of high HBM bandwidth, Metal sits in between with unified memory, and CPU trails on generation; comparing a CPU prefill figure to a CUDA decode figure is a category error.

How do I turn a llama.cpp benchmark into a baseline I can validate tuning changes against in production?

Pin and record all five configuration variables, capture prefill throughput, generation throughput, p95 latency, and memory footprint as one immutable record, and re-run the identical configuration after each change. The delta against that baseline is what distinguishes a real gain from measurement noise — which is exactly the measurement an [inference cost-cut audit](Inference Cost-Cut Pack) uses to rank tuning levers before and after.

A reproducible benchmark is worth the trouble because it converts tuning from opinion into arithmetic. The audit that ranks quantisation, thread and layer-offload tuning, and backend switches by throughput-or-cost per unit of effort starts from exactly this baseline; that ranking work is what our [inference cost-cut engagements](Inference Cost-Cut Pack) and broader R&D consulting services are built around. The failure class to watch for is the one that never shows up in the tokens/second column — a quantisation or backend change that quietly moves the accuracy, not just the speed. Benchmark the speed, but validate the output separately, or you will optimise your way into a faster wrong answer.