You run ollama run llama3 --verbose, watch the tokens stream, and at the end you get a number: 42 tokens per second. It looks like a verdict. It is not. That single figure is an average across at least four distinct phases of work, each governed by a different bottleneck — and knowing which phase your number actually reflects is the difference between tuning something that moves the needle and burning a week quantizing a model when the runtime or the request glue was the real constraint. An honest Ollama benchmark is a layered measurement. Ollama wraps llama.cpp and the GGUF model format behind a friendly CLI and an HTTP server, so the number you read is a composite of model compute, runtime behaviour, and host handling. Treat it as a leaderboard entry and you will chase the wrong lever. Decompose it and you can attribute measured latency to the layer that owns it before committing any tuning effort. What does ollama run --verbose actually report? When you pass --verbose, Ollama prints a timing block after each response. The fields matter more than the headline tokens-per-second, because they already split the work for you: load duration — time to read the model weights into memory and initialise the runtime. On a cold start this dominates; once the model is resident it drops toward zero. prompt eval count / prompt eval duration — the prefill phase, where the model processes your entire input prompt in one forward pass and populates the KV cache. Reported as prompt-eval tokens per second. eval count / eval duration — the decode phase, where the model generates output tokens one at a time. This is the number people usually mean by “tokens per second.” total duration — wall-clock time for the whole request, which includes the two phases above plus scheduling and host overhead that the sub-timers do not capture. Prefill and decode are governed by different physics. Prefill is compute-bound: it runs a large matrix multiply over the whole prompt and saturates the GPU’s arithmetic units. Decode is memory-bandwidth-bound: each new token requires streaming the full set of model weights out of memory, so it is limited by how fast the device can move bytes, not how fast it can multiply them. This is why a long prompt with a short answer and a short prompt with a long answer produce very different Ollama benchmark numbers on the same hardware — they stress different layers. The same bandwidth-versus-compute split shows up at the hardware level too; we walk through it for one platform in what NVIDIA DGX Spark memory bandwidth means for inference bottlenecks. Which layer does your latency actually live in? The core skill is attribution. The measured latency of an Ollama request sits in one of three layers, and each responds to a different intervention. Model compute. If your decode token rate is close to the memory-bandwidth ceiling of your GPU and prefill saturates the arithmetic units, the model itself is the constraint. Quantizing to a smaller GGUF format (Q4_K_M rather than Q8_0, for instance) or choosing a smaller parameter count is the lever that moves the number. Nothing in the serving path will help. The llama.cpp / GGUF runtime. Ollama’s throughput depends on how llama.cpp was built and configured — the number of GPU layers offloaded (num_gpu), the batch size for prefill, the context window, and whether the build has the right backend (CUDA, Metal, ROCm, or a CPU fallback). A model that runs partly on CPU because it did not fit in VRAM will report a decode rate far below the GPU’s capability, and no amount of quantization tuning fixes a runtime that is silently offloading layers to the host. Host and request glue. The gap between total duration and the sum of the sub-timers is where scheduling, queueing, HTTP serialisation, and template rendering live. Under concurrent load this layer grows: Ollama processes requests through a queue, and a second request waits while the first decodes. A high per-request tokens-per-second figure can coexist with terrible tail latency if the glue is the bottleneck — which is the trap the next section names. Attribution rubric Symptom in the numbers Likely owning layer Lever that moves it Lever that does not Decode t/s near bandwidth ceiling, prefill saturates compute Model compute Quantize down, smaller model Serving config, more requests Decode t/s far below GPU capability, some layers on CPU Runtime (llama.cpp / GGUF) Fix num_gpu offload, rebuild backend, shrink to fit VRAM Quantizing further before the offload is fixed Per-request t/s fine, but p95 total-duration explodes under load Host / request glue Concurrency limits, batching, a purpose-built serving stack Quantization or a bigger GPU load duration dominates total on every call Cold-start / warm-up Keep model resident (keep_alive), amortise load across requests Anything model-side The rubric is not a decision tree you follow blindly — it is a way to read the timing block and form a hypothesis before you change anything. Measure first, attribute, then tune the layer that owns the number. Why can a high tokens-per-second figure still hide a latency problem? Here is the failure that catches teams moving from a laptop demo to a shared deployment. On a single-request --verbose run, Ollama reports a healthy decode rate — say, roughly 40 tokens per second on a mid-range GPU (illustrative figure; your rate depends on model, quantization, and hardware). Everyone concludes the system is fast. Then it goes behind an internal API, ten users hit it at once, and p95 latency climbs to several seconds even though the per-token rate never changed. The tokens-per-second number measured a single stream in isolation. It says nothing about queueing. Ollama’s default serving path handles a bounded number of parallel requests, and beyond that requests wait. The eval duration timer keeps reporting the same throughput per request because each request, once it starts, decodes at the same rate — but the time-to-first-token grows because requests sit in a queue. This is precisely the class of problem that separating prefill and decode across a serving stack is designed to address, a pattern we cover in SGLang PD disaggregation and separating prefill and decode for GPU throughput. The lesson is a claim worth stating plainly: single-stream tokens-per-second is a model-and-runtime measurement, not a serving-capacity measurement. To characterise capacity you need concurrent load and a tail-latency percentile, not an average from one request. This is the same discipline we apply when benchmarking agentic AI inference before porting the path — an average hides exactly the behaviour that breaks in production. How do quantization and context length change the numbers? Both are real levers, but they move different parts of the timing block, and confusing them wastes effort. Quantization shrinks the weights. A GGUF model at Q4 is roughly half the size of the same model at Q8, which means decode — the bandwidth-bound phase — streams fewer bytes per token and speeds up proportionally to the size reduction (observed pattern across local-inference setups; the exact ratio depends on the GPU’s effective bandwidth, not a benchmarked constant). It also lets a larger model fit entirely in VRAM, which can flip a runtime from a slow CPU-offload path to a fast all-GPU path — a much bigger jump than the raw byte-count reduction alone. The cost is accuracy: aggressive quantization degrades output quality, so the right level is a quality-versus-speed trade you measure, not assume. Context length changes the prefill cost and the KV-cache footprint. A longer prompt means more prefill compute up front and more memory reserved for the cache, which competes with weight storage in VRAM. Raising the context window on a model that already fills memory can push you back onto a CPU-offload path and slow decode down — the opposite of what a “bigger context is better” intuition suggests. If your prompts are short, tuning context length changes nothing measurable; if they are long, it dominates the prefill number. The practical rule: quantize to fix a decode or memory-fit problem, adjust context to fix a prefill problem, and never tune either until the --verbose block tells you which phase your latency lives in. FAQ How should you think about ollama benchmark in practice? Running a model with --verbose produces a timing block that already decomposes the request into load, prefill (prompt-eval), and decode (eval) phases. In practice the benchmark is a layered measurement of where time goes across model compute, the llama.cpp/GGUF runtime, and host handling — not a single verdict on speed. The tokens-per-second headline is an average across phases that each obey different bottlenecks. What do the prompt-eval, eval, and load-time numbers from ollama run --verbose actually measure? Load duration is the cost of reading weights into memory and initialising the runtime, which dominates on cold starts and drops toward zero once the model is resident. Prompt-eval measures prefill — processing the whole input prompt in one compute-bound forward pass. Eval measures decode — generating output tokens one at a time, a memory-bandwidth-bound phase that is what people usually mean by “tokens per second.” How do you tell whether an Ollama latency number reflects model compute, the llama.cpp runtime, or host glue? Compare the sub-timers against the total. If decode sits near your GPU’s bandwidth ceiling, the model is the constraint; if decode is far below capability and layers are running on CPU, the runtime’s offload configuration owns it; and if the gap between total duration and the summed sub-timers grows under load, host and request glue is the bottleneck. The attribution rubric in this article maps each symptom to the lever that actually moves it. How does quantization and context length affect Ollama benchmark results, and when does tuning them actually move the number? Quantization shrinks weights, speeding up the bandwidth-bound decode phase and potentially flipping a model from a slow CPU-offload path to an all-GPU path, at the cost of output quality. Context length changes prefill compute and KV-cache footprint. Quantization only helps when decode or memory-fit is the constraint; context tuning only matters when prompts are long enough to dominate prefill — so neither should be touched until the --verbose block identifies the phase. Why can a high tokens-per-second figure still hide a latency problem in the request-handling path? A single-stream tokens-per-second reading measures one request in isolation and says nothing about queueing. Under concurrent load, requests wait in Ollama’s serving queue, so time-to-first-token and p95 latency climb even though the per-request decode rate is unchanged. Characterising serving capacity requires concurrent load and a tail-latency percentile, not an average from one request. How do Ollama benchmarks relate to the inference-engine layers you would measure before deciding to port or accelerate? The load/prefill/decode/glue decomposition is the same layer split you profile before deciding whether to port a path to a purpose-built serving stack or accelerated hardware. Attributing latency to the right layer tells you whether the port would help at all — moving a glue-bound workload to a faster GPU changes nothing. It makes the profiling-baseline step concrete for a local-LLM path. What common mistakes make Ollama benchmark comparisons misleading across machines or models? Comparing a cold-start run on one machine against a warm run on another mixes load duration into the number differently. Comparing models at different quantization levels or context lengths conflates model choice with runtime configuration. And comparing single-stream averages while ignoring whether layers silently offloaded to CPU produces numbers that reflect the runtime state, not the hardware’s capability — so fix warm-up, quantization, offload, and load conditions before any cross-machine comparison. Where this leaves the port-or-tune decision The reason to decompose an Ollama benchmark carefully is that the decomposition is the profiling baseline for a bigger decision. Before a team commits to quantizing a model, swapping GPUs, or porting the serving path to a purpose-built engine, the --verbose timing block already tells them which layer owns the latency — and therefore which of those investments would move the number and which would waste a sprint. This is the same measurement discipline behind our GPU engineering practice and the profiling-first framing of the [inference cost-cut sprint](Inference Cost-Cut Pack). The failure class to guard against has a name: attributing latency to the wrong layer. A number that reflects a CPU-offloaded runtime looks like a slow model; a queueing problem looks like fine throughput. Measure the layers, attribute the number, then decide.
