Type “ollama benchmark” into a search box and most of what comes back is a single tokens/sec figure someone read off one run of one prompt. It looks decisive. It rarely is. A headline throughput number tells you almost nothing about whether a local runtime clears the workload once cold-start, context length, and concurrency come into scope — and those are exactly the conditions that decide whether a local-inference path is worth committing engineering to. The useful question is not “how many tokens per second does Ollama do on my machine?” It is “where does the time actually go, and does the profile clear my latency and unit-cost target under the load I will actually apply?” Those are different questions, and only the second one produces a defensible deployment call. This is the same divergence that governs any runtime-fit decision: a peak number is a starting hypothesis, not a verdict. What does an Ollama benchmark actually measure? Ollama wraps llama.cpp and exposes a small set of timing fields on every generation. When you run with verbose output, or read the /api/generate response metadata, you get more than one number, and the split is the whole point: load_duration — time to bring the model weights into memory and set up the runtime. On a cold start this is dominated by reading a multi-gigabyte GGUF file off disk and, if you are offloading layers, moving them onto the GPU. prompt_eval_duration and prompt_eval_count — time spent processing the input prompt (the prefill phase), and how many tokens that was. Divide them and you get prompt-eval throughput, which is compute-bound and scales with context length. eval_duration and eval_count — time spent generating output tokens (the decode phase), and how many. This is the number most people quote as “tokens/sec”, and it is typically memory-bandwidth-bound rather than compute-bound. A single headline tokens/sec figure collapses all of this into one line, usually reporting only decode throughput on a warm model with a short prompt. That is the most flattering possible measurement. It hides the cost of loading the model, understates the cost of a long prompt, and says nothing about what happens when two requests arrive at once. The reason to keep the phases separate is that they respond to different levers. If your workload sends long prompts and generates short answers — a classification or extraction task — prefill dominates and decode throughput barely matters. If it sends short prompts and streams long answers — a chat or drafting task — decode dominates. Reporting one blended number optimizes for neither. Which numbers matter, and how they relate Here is a compact map of what to record and what each field tells you about a deployment decision. Treat the throughput figures as observed-pattern on your own hardware, not portable benchmarks — the point of running them is that they are yours. Metric Where it comes from What it decides Cold-start latency load_duration on first request Whether the first response after a scale-up or restart clears your SLA Prompt eval throughput prompt_eval_count / prompt_eval_duration Whether long-context requests stay inside the latency budget Decode throughput (tokens/sec) eval_count / eval_duration Interactive feel; time-to-full-response for streamed output Time-to-first-token load + prompt eval (warm: prompt eval only) Perceived responsiveness for chat-style workloads Resident memory footprint OS/GPU memory after load, per quantization Whether the model co-resides with everything else on the node The relationship that trips teams up: decode throughput and time-to-first-token move independently. A model can stream tokens quickly once it starts but take a long time to start, because prefill on a 4,000-token prompt is a separate cost from generating the reply. If you only benchmark short prompts, you never see the prefill wall, and it lands on you in production when a user pastes a document. How do quantization and model size change the figures? Quantization is the single largest lever on both throughput and footprint, and Ollama exposes it directly through the model tag (q4_K_M, q5_K_M, q8_0, and so on). Lower-bit quantization shrinks the weights, which reduces the memory bandwidth needed per generated token — and since decode is bandwidth-bound, that usually raises tokens/sec while cutting resident memory. The trade is output quality, which no timing field will show you. If you want the mechanics of why 4-bit changes the arithmetic and where it costs accuracy, 4-bit floating-point (FP4) and when it cuts inference cost covers the precision side of the trade in detail. Model size interacts with your hardware in a way a benchmark surfaces immediately: if the quantized model fits entirely in GPU memory, you get GPU-speed decode; if it spills to system RAM and CPU, throughput collapses by an order of magnitude because the layers offloaded to CPU run at a fraction of the bandwidth. Ollama tells you how many layers it offloaded — watch that number. A benchmark that shows healthy tokens/sec on a model that only just fit is fragile: add a longer context, and the growing KV cache can push you over the line into partial-CPU execution mid-benchmark. The relationship between capacity and the real bottleneck is worth understanding before you size hardware, and what a 128GB GPU means in practice walks through why capacity is not the same as usable bandwidth. Why peak tokens/sec understates cold-start The most common benchmarking mistake is measuring the second request. You run once to “warm up”, discard it, then time a clean run — and report a number that assumes the model is already resident. In a long-running server with one always-loaded model, that assumption is fair. In almost every other deployment it is not. If Ollama unloads idle models to reclaim memory (the default keeps a model resident for a few minutes, then evicts it), every request after an idle gap pays load_duration again. In a scale-to-zero or multi-model setup, cold-start is not an edge case — it is the common path. Reading a 7-billion-parameter GGUF off an NVMe disk and offloading it to the GPU is on the order of seconds, per NVIDIA’s published sustained NVMe and PCIe transfer rates; on a slower disk or a larger model it is longer. That is a first-token latency the peak throughput number never mentions. The fix is to benchmark both states explicitly and report them separately: warm decode throughput as the steady-state ceiling, and cold-start latency as the tail you will hit on the first request after any idle period. Release-readiness for a local runtime hinges on exactly these conditions — cold-start, footprint, and concurrency behaviour — the same checks that any local runtime must pass before it ships. How context length and concurrency move the numbers A single-prompt benchmark measures a system under the least demanding condition it will ever see. Two things break the number as you approach real load: Context length. As the prompt and generated history grow, the KV cache grows with them, consuming memory and adding per-token attention cost. Decode throughput degrades as context extends, and prefill time on a long prompt can dwarf the generation time entirely. A benchmark run at 256 tokens of context tells you nothing about behaviour at 8,000. Concurrency. Ollama can serve parallel requests, but throughput per request does not stay flat as you add them — the requests contend for the same memory bandwidth and compute, and the KV cache for every in-flight sequence has to co-reside in memory. There is a point where adding a request lowers everyone’s tokens/sec and raises everyone’s latency. Finding that saturation point is the whole reason to benchmark under load rather than one prompt at a time. If you want the general principle behind measuring where a serving path saturates, SGLang PD disaggregation and separating prefill from decode shows how larger serving stacks tackle the same contention problem Ollama hits at smaller scale. The practical benchmark, then, is a sweep, not a point: vary context length across the range your workload spans, and vary concurrency from one up to the point where the numbers stop improving. What you are looking for is the corner where the runtime still clears your target, and how much headroom sits between that corner and your expected load. Reading the numbers against a target: a runtime-fit checklist A benchmark only becomes a decision when you hold it against a stated target. Before you commit to a local Ollama path, walk this list — each item is a way the flattering headline number can hide a disqualifying condition. State the target first. Latency budget (time-to-first-token and full-response), throughput floor, and unit-cost ceiling. Without these, no benchmark number is pass or fail — it is just a number. Measure cold-start separately from warm decode. If your deployment ever idles or scales to zero, the cold-start latency is the one that binds. Does it clear the first-token budget? Sweep context length across your real range. Does prefill on your longest expected prompt stay inside the latency budget? Sweep concurrency to saturation. At your expected concurrent load, does per-request throughput still clear the floor, and how much headroom remains? Record footprint per quantization level. Does the model co-reside with everything else on the node without spilling layers to CPU under a full KV cache? Confirm no silent CPU offload. Check the offloaded-layer count at your worst-case context length, not just at load. Translate to unit cost. Tokens/sec at the saturating concurrency, against the node’s hourly cost, gives cost-per-token. Compare that to a native or GPU-offloaded alternative before you standardize. If the runtime clears all seven against the target, a local Ollama path is a defensible call. If it clears the warm decode number but fails cold-start or saturates below your concurrency, you have learned something a single tokens/sec figure would have hidden until production — that the workload needs a native runtime, a GPU-offloaded serving stack, or a smaller quantized model. The measurement did its job by ruling out a bad commitment early. FAQ How does ollama-benchmark work? Running Ollama with verbose output or reading the /api/generate response returns timing fields that split total time into model load, prompt evaluation (prefill), and token generation (decode). In practice, benchmarking means recording those phases separately across your real context lengths and concurrency levels, not reading a single tokens/sec line off one warm run. Which metrics does an Ollama benchmark actually report — tokens/sec, prompt eval time, model load, and how do they relate? It reports load_duration (bringing weights into memory), prompt_eval_duration/prompt_eval_count (prefill), and eval_duration/eval_count (decode, the usual tokens/sec figure). They relate through the request shape: prefill dominates long-prompt/short-answer workloads, decode dominates short-prompt/long-answer ones, and time-to-first-token combines load and prefill — so a blended number optimizes for neither. How do quantization level and model size change the throughput and footprint figures you measure? Lower-bit quantization shrinks the weights, cutting the memory bandwidth per generated token, which usually raises decode throughput and lowers resident memory — at a cost to output quality no timing field shows. Model size interacts with your hardware directly: if the quantized model fits in GPU memory you get GPU-speed decode, but if it spills to CPU, throughput can collapse by an order of magnitude. How does cold-start and model-load time affect the benchmark, and why does peak tokens/sec understate it? Peak tokens/sec is typically measured on an already-resident model, so it excludes load_duration entirely. If Ollama evicts idle models or you scale to zero, every request after an idle gap pays the load cost again — reading a multi-gigabyte GGUF off disk and offloading it to GPU is on the order of seconds — making cold-start the binding first-token latency the peak number never mentions. How do context length and concurrency shift the numbers away from a single-prompt benchmark? As context grows, the KV cache grows, degrading decode throughput and inflating prefill time, so a 256-token benchmark says nothing about behaviour at 8,000 tokens. Concurrent requests contend for the same memory bandwidth and compute, so past a saturation point each added request lowers everyone’s throughput and raises latency — which only a load sweep, not a single prompt, will reveal. How do we read Ollama benchmark figures against a latency and unit-cost target to reach a runtime-fit verdict? State the target first — latency budget, throughput floor, unit-cost ceiling — then check the benchmark against it: cold-start against first-token budget, longest-context prefill against latency, saturating concurrency against the throughput floor, and tokens/sec-at-saturation against node cost for cost-per-token. The runtime fits only if it clears all of these, not just the warm decode number. When does a benchmarked local runtime clear the target versus needing a native or GPU-offloaded path? It clears the target when warm decode, cold-start, context-length prefill, and concurrency saturation all stay inside the stated budgets with headroom, and the model fits in memory without CPU spill. It needs a native or GPU-offloaded path when the warm number passes but cold-start blows the first-token budget, concurrency saturates below expected load, or a full KV cache forces layers onto the CPU. If you are weighing a local Ollama path against a native or GPU-offloaded runtime, the benchmark is only the input — the decision rests on how the profiled figures map to your latency and unit-cost target. That mapping is the runtime-fit step inside our GPU inference practice, and the profiling baseline that says whether a runtime truly clears the target is the same one described in the Inference Cost-Cut Pack. The failure to guard against is standardizing on a runtime that clears the headline number and misses the target once real load is applied — which is precisely the gap a phase-attributed benchmark is built to catch.
