llama.cpp Benchmark: Reading the Numbers Before You Port an LLM

llama.cpp vs Ollama benchmarks mostly compare one engine with itself: read pp and tg separately, check build and defaults, and know what the run leaves…

llama.cpp Benchmark: Reading the Numbers Before You Port an LLM
Written by TechnoLynx Published on 11 Jul 2026

In a llama.cpp vs Ollama comparison you are mostly comparing one engine with itself. Ollama, LM Studio and llamafile all serve GGUF models through the same llama.cpp core. The gaps you see come from how each one is built and what defaults it picks. So read a llama.cpp benchmark in a fixed order. Look at prompt processing (pp) and token generation (tg) separately, then check batch size, quantisation format, backend and build. After that, ask what the run never measured: accuracy on the target backend, quality at that quantisation, your real context length and concurrency. Leave those questions open and you can port a model to a new runtime or machine on numbers that do not describe your workload.

What the pp512 and tg128 Figures in llama-bench Measure

llama-bench is the benchmarking tool that ships with llama.cpp. It reports two separate speeds, both in tokens per second: prompt processing during prefill and text generation during decode. A 2026 study of llama.cpp quantisation on Llama-3.1-8B-Instruct measured CPU throughput with exactly this configuration.

The two figures are limited by different parts of the hardware, so read them separately. Prefill handles the whole prompt in parallel and is compute-bound, which rewards raw arithmetic throughput and mature kernels. Decode produces one token at a time and is mostly limited by memory bandwidth. The vLLM authors describe the decode side the same way. Each generation step depends on the previous one and runs as matrix-vector work, which leaves the GPU’s compute underused and makes the phase memory-bound.

In practice, a backend can win on one figure and lose on the other. The 2026 quantisation study found that tg128 rose substantially for lower-bit formats, while pp512 varied more irregularly between formats. Its authors conclude that both should be measured under a fixed configuration. If a report gives a single “tokens per second” number, find out which phase it describes before you compare it with anything.

Why the Same Model Benchmarks so Differently Across Machines

A llama.cpp result reflects one combination of hardware, quantisation, backend, context and batch settings. The ELIB paper (2025) makes the general point that the performance of a quantised model on edge hardware is closely tied to the quantisation method, the software configuration and the hardware specification together.

The backend alone can shift results a long way. An independent llama.cpp scoreboard from April 2026 put an RTX 5090 at roughly 14,073 pp512 and 290 tg128 on CUDA, against 10,382 and 264 on Vulkan. That makes CUDA about 36% faster at prefill and about 10% faster at decode on the same card. A separate Radeon comparison from October 2025, reported in the same article, put ROCm ahead on prefill (753 vs 624 pp512) while Vulkan led on decode (49.4 vs 47.0 tg128). On that Radeon card, the backend ranking depended on which phase you looked at.

Quantisation shifts results too. In the ELIB measurements, q4_0 gave 1.24 to 2.88 times the token throughput of q8_0, depending on the device and on whether CPU or GPU was used. Context length is a third variable, because prefill work grows with the prompt. For that reason, Kunal Ganglani’s 2026 methodology guide recommends testing at several context lengths rather than one.

Which Batch Size Should You Test in llama.cpp?

Test more than one batch size, because batch size trades throughput against latency. llama.cpp has two batch settings. -b is the logical maximum batch size, which defaults to 2048. -ub is the physical maximum batch size, which defaults to 512. A practical sweep runs llama-bench at -b 128, 512 and 1024 to see how prompt processing scales. On CPU, vary -t as well to find the best thread count.

The ELIB paper measured both sides of the trade. Increasing the batch size raised throughput by 14 times but also raised latency by 4 times. Beyond a certain size the workload becomes compute-bound, and larger batches then add latency without adding any throughput. That plateau tells you where to stop. Pick the batch size from your latency budget, not from the peak throughput row.

Batch scaling is also a weak point for llama.cpp next to other frameworks. LLM-Inference-Bench (2024) found that llama.cpp scales weakly and does not improve much at large batch sizes, because it does not make full use of the available compute.

llama.cpp vs Ollama vs LM Studio: Why the Wrapper Reports a Different Speed

Ollama runs a modified llama.cpp engine and adds its own model packaging, templates, process management and APIs. This means any speed difference is wrapper and build cost, not a difference in the engine. How big that cost is depends on hardware and build choices.

Atomic Chat ran a matched test on an RTX 5090 with the same Q4_K_M Llama 3.1 8B file, 4096 context, batch 512, all layers on GPU and flash attention enabled. llama.cpp generated 249 tokens per second against Ollama’s 108, while prompt processing was almost identical (13,561 vs 13,426 tok/s). The authors put the decode gap down to compiling llama.cpp from source for that exact GPU, while Ollama ships prebuilt binaries. They call the result one measured data point, not a universal percentage.

Other measurements find a much smaller gap. Khired reports Ollama 8–14% behind llama.cpp on an RTX 5090 with Llama 3 8B (95 vs 82 tok/s). InventiveHQ measured 10.3% on an RTX 5060 Ti and about 14% on an Apple M3 Max. Khired and Atomic Chat both tested an RTX 5090 yet report very different gaps, so no single percentage transfers to your machine.

LM Studio shows how a ranking can flip. On the RTX 5060 Ti it was within noise of raw llama.cpp (0.3% behind), but on the M3 Max it came last, 29% behind, because its bundled llama.cpp lagged upstream. Mozilla.ai reached the same conclusion from another direction. Once weights and environment were held fixed, the four servers it tested landed within a few percent of each other on prompt processing. Turning on CUDA graphs in llamafile’s build, however, improved decoding by 4.3–16.8%. Before comparing servers, find out how each binary was compiled.

Defaults also move the numbers. Ollama processes one parallel request per model by default, and its required RAM scales with OLLAMA_NUM_PARALLEL times the context length. Ollama’s documentation gives a default of 1, while Khired reports 2; check the default in your installed version. Ollama also sets the default context from available VRAM: under 24 GiB you get 4,096 tokens, whatever the model supports. llama.cpp can go wrong in the other direction. Launched without flags in the Atomic Chat test, it reserved the model’s full 131k context and 21.6 GB of VRAM, where Ollama used 9.4 GB.

For the Ollama side of this measurement, our guide to reading an Ollama benchmark covers its reporting in detail.

vLLM vs llama.cpp: When Concurrency Changes the Answer

vLLM leads once many users hit the model at the same time. llama.cpp remains viable for a single server with moderate load, especially on non-NVIDIA hardware. Khired puts llama.cpp with --parallel and --cont-batching as viable up to roughly 20 concurrent users, and places 50 or more users with strict latency targets in vLLM’s territory. Red Hat’s A100 tests show the same pattern. vLLM’s throughput kept scaling with concurrency, while Ollama plateaued at a much lower level even with OLLAMA_NUM_PARALLEL=32.

vLLM’s advantage rests on memory management. The PagedAttention paper found that earlier serving systems used only 20.4–38.2% of their KV cache memory for actual token states, and that this waste limited batch size. LLM-Inference-Bench adds that llama.cpp does not fully use Tensor Cores or tensor parallelism, and does not gain from grouped-query attention the way vLLM and TensorRT-LLM do.

A model format change comes with the move. vLLM is built around safetensors, AWQ, GPTQ and FP8 rather than GGUF. A llama.cpp benchmark therefore does not predict vLLM performance, because the weights you would run on vLLM are a different file.

What a llama.cpp Benchmark Leaves Out Before You Port

A clean pp/tg run tells you that the hardware is working correctly. It does not tell you that the ported model behaves the same, and Mozilla.ai notes that its own benchmark did not measure time to first token, load time, memory use, power or concurrent requests. Moving a model to another runtime or machine is AI infrastructure engineering work on the deployment target, not a hand-off. It is worth doing when a measurable trigger justifies it, such as a performance ceiling or a target constraint. Our Furball Factory page covers how we port and tune models for prosumer AI hardware once such a trigger holds. Check these gaps before you trust the trigger:

What the benchmark omits Why it matters before porting
Accuracy on the target backend ELIB measured perplexity almost 10 times higher with OpenCL GPU inference on two devices than with CPU, while Metal matched CPU
Quality at the chosen quantisation Formats with similar perplexity diverged on reasoning and instruction-following tasks for Llama-3.1-8B
Your context length Prefill work grows with the prompt, so a pp512 figure does not tell you time to first token on your own long prompts
Concurrency Under parallel load, vLLM’s throughput kept scaling while Ollama’s stayed flat
Fixed host overhead A near-constant per-token cost reads as a large percentage on fast models and a small one on slow ones
Model architecture effects llama.cpp did not gain from GQA where vLLM and TensorRT-LLM did

Check the backend accuracy row first. A faster number from a backend that degrades output is a regression. ELIB’s authors say GPU acceleration needs to be tested rigorously on each platform and framework combination. For the wider question of whether a benchmark matches production load, our article on LLM inference benchmarks and how to read them for deployment goes further.

The Controls to Log with Every llama.cpp Result

A llama.cpp number only means something when it is reported with the conditions that produced it. The 2026 methodology guide lists the exact model file, quantisation, backend version, context size, thread and GPU-layer settings, and whether a laptop was on battery. Any one of these can move the result more than the choice of model. The same guide asks you to separate warm and cold runs, test at more than one concurrency level and report p50, p95 and p99 instead of the mean. The Atomic Chat test shows what this looks like in practice: it named the llama.cpp commit it built, used one GGUF file for both tools, ran 5 warmup and 10 measured runs and reported medians.

A minimal record for each run:

  1. llama.cpp commit and build flags (backend, CUDA graphs, shader toolchain)
  2. Model file and quantisation format
  3. pp and tg reported separately, with prompt and output lengths
  4. Context size, -b, -ub, -t and -ngl
  5. Warmup count, measured runs and the statistic reported
  6. Concurrency level and parallel slots, if served

Reading a Gap Before Acting on It

Suppose a decode gap over 2x appears between llama.cpp and a wrapper, as in the Atomic Chat test. Treat it as a question about builds and defaults until a matched test says otherwise. When the gap survives a matched test, the next question is whether accuracy and concurrency survive the move too. A benchmark cannot answer that on its own.

FAQ

Does llama.cpp Use CPU or GPU?

It uses both. llama.cpp runs F16 and quantised models on CPU and GPU. It also supports CPU+GPU hybrid inference for models larger than the available VRAM, with backends including CUDA, HIP, Metal, Vulkan and SYCL.

Is llama-server Part of llama.cpp?

Yes. llama-server is part of llama.cpp and provides its OpenAI-compatible API server.