Run one prompt through Ollama, read the tokens-per-second number off the terminal, and you have a figure. What you do not have is a capacity model. The single-shot number tells you how fast one request completes on an idle GPU — which is almost never the condition your infrastructure actually runs under. Yet that headline number is the one teams quote when they argue for buying a workstation or renting cloud GPUs, and it is the wrong number for that argument. An Ollama benchmark becomes useful only when you stop treating it as a leaderboard score and start treating it as workload profiling. The distinction matters because the metric that decides whether cloud rental or on-premise hardware is cheaper over 12–36 months is not peak throughput. It is the shape of your load: how sustained throughput holds up as concurrency rises, where latency degrades, and how much GPU memory each model and quantisation level actually consumes. Miss that shape and you mis-size infrastructure on both sides — over-provisioning cloud spend or buying a machine that saturates the moment real traffic arrives. Why a single tokens/sec number tells you almost nothing The naive reading assumes a linear world: if one request runs at, say, 60 tokens/sec, then the machine “does 60 tokens/sec” and you can plan around that. Decode-phase LLM inference does not behave that way. The generation loop is memory-bandwidth bound for most models at low batch sizes. A single request underutilises the GPU’s compute units — the bottleneck is streaming model weights from HBM for each token. Add concurrent requests and Ollama (via its llama.cpp backend) can batch them, so aggregate tokens/sec climbs well above the single-request figure until the GPU saturates. The per-request rate falls at the same time, because each user now shares compute and memory bandwidth. One number cannot capture two curves moving in opposite directions. This is the same trap that catches people reading raw utilisation percentages. As we cover in reading GPU utilisation through the three pillars of observability, a high utilisation figure during a single-stream benchmark can coexist with a machine that collapses under ten concurrent sessions — the metric was never measuring headroom. The correct frame for an Ollama benchmark is a small experiment matrix, not a scalar. Which Ollama benchmark metrics actually matter for a cost decision Four metrics carry the cost decision. Everything else is diagnostic detail. Metric What it tells you Why the cost model needs it Tokens/sec at target concurrency Sustained aggregate throughput under realistic load Sets how many requests a single GPU serves — the denominator of cost-per-request p95 latency Tail response time, not the average Determines whether you can pack more load onto one GPU or must add capacity for SLA headroom GPU memory footprint per model/quant Whether the model fits, and how much room is left for KV cache Decides which GPU class you need and how many models co-reside per card Effective utilisation Compute/bandwidth actually consumed vs available Reveals whether on-premise hardware would sit idle (favouring cloud) or run hot (favouring purchase) The single-shot tokens/sec figure people quote is a degenerate case of the first row measured at concurrency = 1 — the least representative point on the curve. In configurations we have profiled, aggregate throughput at moderate concurrency can be several times the single-request rate before saturation, while p95 latency rises non-linearly near the saturation knee (observed pattern across GPU inference engagements; not a published benchmark). That knee is the operationally relevant point, because it defines how much real work one GPU absorbs before the tail latency breaks your service level. How do I benchmark Ollama across concurrency levels? Sweep concurrency deliberately rather than running one prompt. The goal is to trace the two curves — aggregate throughput and per-request latency — as parallel load increases, and to find the point where they diverge. A workable procedure: Fix the prompt/response profile first. Choose input and output token lengths that match your real traffic. A 200-token summarisation prompt and a 2,000-token generation task stress memory and compute completely differently, so a benchmark run at the wrong profile answers the wrong question. Warm the model. Discard the first request — it pays the model-load and cache-warm cost that steady-state traffic does not. Sweep concurrency in steps (1, 2, 4, 8, 16…) driving Ollama’s HTTP API with a parallel client such as oha, k6, or a short asyncio harness. Set OLLAMA_NUM_PARALLEL so the server actually batches rather than queuing. Record per level: aggregate tokens/sec, p50 and p95 latency, and GPU memory + utilisation sampled with nvidia-smi or DCGM. Find the saturation knee — the concurrency where aggregate throughput plateaus while p95 latency starts climbing sharply. That point is your per-GPU capacity ceiling. The output is a table, not a headline. Sustained throughput at your SLA-bounded concurrency — not peak burst — is the operationally relevant number for GPU-accelerated inference, and it is the figure that goes into the cost model. This mirrors the argument in reading DGX Spark benchmarks for utilisation rather than peak FLOPs: the peak spec is a ceiling you rarely touch, and sizing to it wastes money. How does quantisation change Ollama throughput and memory? Quantisation is the lever that most directly moves both throughput and the GPU class you need, so it belongs in the benchmark matrix rather than fixed at one setting. Ollama ships models in GGUF format at various quantisation levels — Q4_K_M, Q5_K_M, Q8_0, and so on. Lower-bit quantisation shrinks the weight footprint, which does two things at once: it frees HBM for a larger KV cache (more concurrent requests or longer contexts) and it reduces the bytes streamed per token, which raises decode throughput on bandwidth-bound workloads. A 4-bit weight path can roughly halve memory versus 8-bit for the same model. The related question of when 4-bit precision pays off on GPUs is worth reading alongside this — how FP4 works and when it cuts inference cost covers the accuracy trade-off that a throughput benchmark alone will not show you. The trap is benchmarking throughput at Q4 and reasoning about cost as if quality were free. Lower precision can degrade output quality on some tasks, and a benchmark measures speed, not correctness. Run the throughput sweep at each quantisation you would actually deploy, then check quality separately on your own task before letting the fastest configuration into the cost model. Speed you cannot ship is not capacity. How do I turn Ollama benchmark numbers into a cloud-vs-on-premise comparison? This is the whole point of profiling. The benchmark produces the workload metrics; the decision is a break-even calculation over the hardware’s useful life. The logic runs like this. From the concurrency sweep you know sustained tokens/sec per GPU at your SLA. From your traffic forecast you know required tokens/sec, which gives the number of GPUs you need. On the cloud side, multiply required GPU-hours by the rental rate — but weight it by effective utilisation, because paying for a GPU that sits at 20% during off-peak hours is real waste. On the on-premise side, amortise hardware capex plus power, cooling, and maintenance across 12–36 months, and divide by the sustained capacity you measured. A worked example, with the assumptions stated so you can substitute your own: Assume a workload needing sustained 3,000 tokens/sec at p95 under 4 seconds. Measured (illustrative, from a hypothetical sweep): one mid-range GPU sustains ~1,000 tokens/sec at that SLA before the knee, so you need three GPUs of steady capacity. Cloud path: three GPUs rented continuously, plus burst headroom, priced at the provider’s hourly rate — variable cost that scales cleanly and needs no upfront capital. On-premise path: a three-GPU workstation or server, capex amortised over three years plus operating cost — fixed cost that only wins if utilisation stays high enough to beat the rental equivalent. The break-even hinges on utilisation and duration. Steady, high-utilisation, multi-year workloads tend to favour ownership; bursty or short-lived ones favour rental. The measured effective-utilisation figure is what turns that intuition into a number. Turning utilisation telemetry into exactly this kind of decision is a discipline in its own right — ML observability tools for GPU cost walks through how the same signals feed procurement over time. What makes an Ollama benchmark misleading for infrastructure sizing? The recurring failure is benchmarking a condition you will never run in, then sizing for it. A few patterns account for most mistakes. Quoting the concurrency-1 number as machine capacity is the headline error — it ignores batching entirely and understates a busy server’s throughput while overstating its worst-case latency. The opposite mistake is quoting peak aggregate throughput at maximum concurrency, which ignores the p95 latency you have already blown past to reach it. Both read a single point off a curve that only makes sense as a whole. Beyond that: benchmarking on an idle machine with cold caches, so the first-token latency is inflated; using a prompt/response profile that does not match production, so the memory and compute mix is wrong; and forgetting that Ollama’s default settings may not batch — without OLLAMA_NUM_PARALLEL configured, concurrent requests queue rather than share the GPU, and your throughput ceiling looks artificially low. Each of these produces a number that is internally consistent and externally useless. Ollama benchmark output is only useful when read as input to a total-cost model, not as a leaderboard. The moment the number stops being a curve and becomes a scalar you plan around, it starts lying to you. FAQ What should you know about ollama benchmark in practice? An Ollama benchmark measures how fast the local LLM runtime generates tokens, typically reported as tokens per second. In practice it should be read as workload profiling rather than a single score: you run the model across concurrency levels and prompt/response profiles to see how sustained throughput, latency, and memory behave under realistic load — not just how one request performs on an idle GPU. Which Ollama benchmark metrics (tokens/sec, p95 latency, GPU memory, utilisation) actually matter for a cost decision? Four metrics carry the decision: sustained tokens/sec at your target concurrency (the throughput a GPU actually delivers under load), p95 latency (the tail time that governs your SLA), GPU memory footprint per model and quantisation (which decides the GPU class you need), and effective utilisation (whether hardware would sit idle or run hot). The single-shot tokens/sec figure is the least useful because it is measured at the least representative point. How do I benchmark Ollama across concurrency levels to reveal a sustained-vs-burst workload profile? Fix a realistic prompt/response profile, warm the model, then sweep concurrency in steps (1, 2, 4, 8, 16…) driving the Ollama API with a parallel client while sampling GPU memory and utilisation. Record aggregate tokens/sec and p50/p95 latency at each level, and locate the saturation knee where throughput plateaus and p95 latency starts climbing — that point is your per-GPU capacity ceiling. How does model quantisation change Ollama throughput and GPU memory footprint in benchmarks? Lower-bit quantisation (e.g. Q4 versus Q8 GGUF) shrinks the weight footprint, freeing HBM for a larger KV cache and reducing bytes streamed per token, which typically raises decode throughput on bandwidth-bound workloads. The caveat is that a benchmark measures speed, not output quality, so run the throughput sweep at each quantisation you would actually deploy and validate quality separately before trusting the fastest configuration. How do I turn Ollama benchmark numbers into a cloud-vs-on-premise total cost comparison? Use the measured sustained tokens/sec per GPU at your SLA plus your traffic forecast to derive the number of GPUs required, then compare a variable cloud rental cost (weighted by effective utilisation) against amortised on-premise capex plus power and maintenance over 12–36 months. The break-even hinges on utilisation and duration: steady high-utilisation multi-year workloads favour ownership, bursty or short-lived ones favour rental. What are the common mistakes that make an Ollama benchmark misleading for infrastructure sizing? The main errors are quoting the concurrency-1 number as machine capacity, quoting peak aggregate throughput while ignoring the p95 latency you blew past, benchmarking on a cold idle machine, using an unrealistic prompt/response profile, and forgetting to enable batching so concurrent requests queue instead of sharing the GPU. Each produces a number that is internally consistent but useless for sizing. What still needs to be measured A benchmark that answers “how fast is one prompt” answers a question your infrastructure never asks. The question it does ask — how much sustained work does one GPU absorb before the tail latency breaks, and does that favour renting or buying — is answered by a table of measurements across concurrency and quantisation, fed into a break-even model. The self-serve Ollama sweep described here captures the same workload characteristics a formal GPU performance audit measures, which is why it is a reasonable first step before committing capital. The failure class to watch for is the scalar that pretends to be a curve; when someone quotes a single tokens/sec figure to justify a hardware purchase, the number to ask for is the one at the saturation knee.
