Run the built-in throughput test once, read the tokens/sec figure, and treat it as your workload’s capacity. That is the naive reading of an ollama-benchmark run, and it is where most GPU sizing decisions go wrong before they even start. The number is real. It is also incomplete in ways that matter for money. A single headline tokens/sec figure collapses two very different phases of inference into one average, ignores how much that average shifts with prompt length and concurrency, and says nothing about why the GPU is producing that rate. The expert reading treats the benchmark as a profiling exercise: measure prefill and decode separately, sweep batch size and context length, and work out whether you are compute-bound or memory-bandwidth-bound before you decide the fix is a bigger card. Read correctly, the same benchmark tells you whether a smaller quantised model doubles your throughput for free — or whether more VRAM bandwidth is the only lever you have. How does ollama-benchmark actually work? Ollama wraps a model runtime — most commonly a llama.cpp-derived backend, and increasingly GGUF-quantised weights on a CUDA or Metal path — behind a local server. When you benchmark it, you send a prompt, the model generates a completion, and the tooling reports timing. The headline output is usually tokens per second, sometimes split into “prompt eval” and “eval” rates, sometimes not. That split is the whole story, and it is easy to miss. Inference has two phases with completely different performance characteristics. Prefill processes the entire input prompt in one shot: it is a large matrix-multiply over all input tokens at once, and it tends to saturate the GPU’s compute units. Decode generates output tokens one at a time, each step reading the full model weights and the growing KV-cache out of memory to produce a single token. Decode is dominated by memory bandwidth, not raw compute. A benchmark that averages the two — or that runs a short prompt and a long generation — will report a number that describes neither phase well. In our experience profiling inference paths for clients sizing on-prem hardware, the phase that limits a given workload depends entirely on its prompt-to-output ratio, and that ratio is a property of the application, not the GPU. A retrieval-augmented chatbot with 4,000-token prompts and 100-token answers lives in prefill. A creative-writing endpoint with short prompts and long generations lives in decode. The same GPU serves them at wildly different effective rates. What do tokens/sec, prefill, and decode numbers actually measure? Break the run into its parts and the meaning sharpens: Metric What it measures Bound by What moves it Prompt eval rate (prefill tokens/sec) Tokens ingested during input processing Compute (matmul throughput) Batch size, model size, GPU FLOPS Eval rate (decode tokens/sec) Tokens generated after prefill Memory bandwidth Model size in VRAM, quantisation, HBM/GDDR bandwidth Time to first token (TTFT) Latency from request to first output token Dominated by prefill Prompt length, prefill rate Inter-token latency Gap between successive output tokens Decode rate KV-cache size, memory bandwidth The reason this table matters for capacity planning: prefill and decode scale differently under load. Prefill benefits from batching because you can process multiple prompts’ matmuls together and keep the compute units busy — this is an observed-pattern across the inference deployments we have profiled, not a fixed multiplier. Decode benefits far less from batching on a bandwidth-bound card, because every batched sequence still needs the weights streamed from memory each step. Averaging them hides exactly the information you need to decide whether concurrency helps you. If your ollama-benchmark tooling only reports one aggregate tokens/sec, that is your first signal to instrument more carefully. The technique of separating prefill from decode is the same discipline that production serving stacks such as SGLang formalise; our note on separating prefill and decode for GPU throughput walks through why serious serving systems disaggregate the two phases entirely. How do I tell whether my workload is compute-bound or memory-bandwidth-bound? This is the single most useful thing a benchmark can tell you, and it decides which hardware lever actually helps. The diagnostic is a roofline argument you can run from benchmark output alone. Decode reads roughly the full set of model weights from memory to produce each token. So a first-order upper bound on decode rate is: decode tokens/sec ≈ memory bandwidth (bytes/sec) ÷ model size in memory (bytes/token-pass) Work an illustrative example. Suppose a 7B model in 4-bit quantisation occupies on the order of 4 GB in VRAM, and the card publishes memory bandwidth on the order of 900 GB/s (per the vendor’s published specification for a mid-range accelerator). The bandwidth ceiling on decode is roughly 900 ÷ 4 ≈ 225 tokens/sec before overheads. If your measured decode rate is close to that ceiling, you are bandwidth-bound — more compute does nothing, and only faster memory or a smaller model helps. If your measured rate sits far below it, something else is the limiter: kernel efficiency, small-batch underutilisation, or a CPU-side bottleneck in the serving loop. Run the same check for prefill against the card’s FLOPS rating rather than its bandwidth. If prefill sits near the compute ceiling and decode sits near the bandwidth ceiling, you have a clean picture of a well-tuned path — and you now know that a longer-prompt workload will be compute-limited while a longer-generation workload will be bandwidth-limited. This is why DGX Spark’s memory bandwidth figure matters more than its FLOPS for many inference bottlenecks: decode-heavy workloads never touch the compute headroom. A quick diagnostic checklist Does your benchmark report prefill and decode rates separately? If not, instrument until it does. Compute the decode bandwidth ceiling (bandwidth ÷ model-size-in-memory). Is measured decode within ~70–90% of it? If decode is near the ceiling → bandwidth-bound. Levers: quantise, smaller model, faster-memory card. If decode is far below the ceiling → investigate the serving loop, batch size, and kernel path before buying hardware. Compute prefill against FLOPS. Near ceiling → compute-bound prefill; longer prompts will hurt. Sweep at your real prompt-to-output ratio, not the benchmark default. How do batch size and context length change the numbers? Both move the numbers, and they move prefill and decode in opposite ways, which is why a single-configuration run is misleading. Increasing batch size (concurrent requests) tends to raise aggregate prefill throughput — you fill the compute units with parallel matmuls. On a compute-bound prefill phase, this can improve tokens/sec per GPU substantially. Decode throughput per request usually drops as batch grows on a bandwidth-bound card, because the shared weight streaming becomes the bottleneck; aggregate decode across the batch may still rise, but not linearly. The right metric to watch under batching is aggregate tokens/sec across all concurrent requests, not the per-request rate a single-stream benchmark reports. Increasing context length enlarges the KV-cache, which lives in VRAM and grows with every generated token. A longer context means more bytes to read per decode step and more VRAM consumed, both of which push a workload harder into the bandwidth-bound regime — and eventually into out-of-memory territory. A benchmark run at 512-token context tells you nothing reliable about behaviour at 32,000 tokens. Techniques that reuse cached prefix computation, like the radix cache approach to prefix reuse, exist precisely because context growth dominates cost in long-conversation workloads. The practical rule: sweep both axes across the range your production traffic actually spans. One number at one configuration is a data point, not a capacity model. When does quantising a model improve benchmark throughput — and when does it not? Quantisation is the free-throughput lever people reach for, and whether it works is entirely a function of which regime you are in. If your workload is bandwidth-bound — decode near the memory ceiling — quantising from 16-bit to 4-bit roughly quarters the bytes read per token, and decode throughput can rise close to proportionally, on the same card, at no extra hardware spend. This is the case where the roofline math pays off directly. If your workload is compute-bound — prefill-heavy, long prompts, short answers — quantisation helps far less, because the limiter was never memory traffic. You may even lose accuracy for negligible throughput gain. There is always an accuracy cost to weigh, and it is model- and task-dependent; the mechanism and the trade-off are covered in our explainer on 4-bit floating point and when it cuts inference cost. The benchmark’s job here is not to decide accuracy — it is to tell you whether the throughput upside even exists before you spend evaluation effort on the quality question. How do I turn benchmark output into a tokens/sec-per-dollar sizing decision? This is where the profiling pays for itself. The measurable outcome you want is not tokens/sec — it is tokens/sec per dollar, or equivalently cost per million tokens and requests served per GPU-hour. Worked example, with assumptions stated explicitly: Workload: RAG assistant, ~2,000-token prompts, ~200-token answers (prefill-heavy). Measured on Card A: prefill 3,000 tok/s, decode 120 tok/s at batch 1; aggregate decode ~400 tok/s at batch 8. Card A cost: assume an illustrative $2.00/GPU-hour on-demand. Effective serving rate at your prompt/output mix, batched: derive from the aggregate numbers, not the single-stream figure. If the aggregate throughput at your real mix yields, say, on the order of a few hundred output tokens/sec per GPU, then cost per million output tokens = ($2.00 ÷ 3600) ÷ (tokens/sec) × 1,000,000. The exact figure is yours to compute from your measured rates — the point is that the inputs to that formula come from the phase-separated, swept benchmark, not from a headline number. Now the decision becomes defensible. If you are bandwidth-bound and quantisation doubles decode throughput, tokens/sec per dollar roughly doubles on the same card — buy no hardware. If you are compute-bound and near the FLOPS ceiling, a faster card genuinely helps and the spend is justified. Either way you are sizing against a bottleneck you have located, not over-provisioning against a number whose provenance you never questioned. That reasoning transfers directly to the cost framing in our ollama benchmark walkthrough for cost decisions, which anchors the tokens/sec-per-dollar view against local-versus-cloud sizing. Why is a single headline throughput number a poor basis for capacity planning? Because it silently assumes your production workload looks like the benchmark’s default configuration, and it almost never does. A headline number bakes in one prompt length, one output length, one batch size, and one phase mix. Change any of those — and real traffic changes all of them, request to request — and the effective rate moves, often by a large factor. Planning capacity against that single figure means over-provisioning when the true bottleneck was elsewhere, or under-provisioning when concurrency behaviour was worse than the single-stream test suggested. The correct posture treats sustained rate at your real prompt-to-output distribution and concurrency as the operationally relevant measure, not the transient peak a one-shot run produces. This is the same profiling-first discipline that separates a modest GPU port from a multi-day redesign: the fix depends entirely on where the bottleneck actually lives, and a single number hides that. FAQ How should you think about ollama-benchmark in practice? Ollama wraps a model runtime (commonly a llama.cpp-derived backend on CUDA or Metal) behind a local server, and benchmarking it sends a prompt, generates a completion, and reports timing — typically tokens per second. In practice the number only means something once you split it into prefill (prompt processing, compute-bound) and decode (token generation, bandwidth-bound), because the phase that limits your workload depends on the application’s prompt-to-output ratio, not the GPU. What do tokens/sec, prefill, and decode numbers in an ollama-benchmark run actually measure? Prompt eval rate measures how fast input tokens are ingested during prefill and is bound by compute throughput; eval rate measures how fast output tokens are generated during decode and is bound by memory bandwidth. Time to first token is dominated by prefill and inter-token latency by decode. Averaging prefill and decode into one figure hides the information you need, because the two phases scale differently under batching and context growth. How do I tell from a benchmark whether my LLM workload is compute-bound or memory-bandwidth-bound? Compute the decode bandwidth ceiling as memory bandwidth divided by model size in memory, then compare your measured decode rate to it. If measured decode sits near that ceiling you are bandwidth-bound — only faster memory or a smaller model helps; if it sits far below, the limiter is the serving loop, batch size, or kernel path, not the hardware. Run the same check for prefill against the card’s FLOPS rating. How does batch size and context length affect the throughput numbers ollama-benchmark reports? Larger batch size raises aggregate prefill throughput by filling compute units, but per-request decode throughput usually drops on a bandwidth-bound card because shared weight streaming becomes the bottleneck. Longer context enlarges the KV-cache, adds bytes read per decode step, consumes more VRAM, and pushes the workload harder into the bandwidth-bound regime. A run at one batch size and context length is a single data point, so sweep both across your real traffic range. When does quantising a model improve benchmark throughput on the same GPU, and when does it not? Quantisation helps most when the workload is bandwidth-bound: moving from 16-bit to 4-bit roughly quarters the bytes read per token, and decode throughput can rise close to proportionally on the same card at no extra hardware cost. It helps little when the workload is compute-bound (prefill-heavy, long prompts, short answers), because memory traffic was never the limiter. There is an accuracy cost to weigh separately, but the benchmark’s job is to tell you whether the throughput upside even exists first. How do I turn ollama-benchmark output into a tokens/sec-per-dollar GPU sizing decision? Take the phase-separated, swept throughput measured at your real prompt-to-output mix and batching, then divide the GPU’s hourly cost by that effective rate to get cost per million tokens or requests served per GPU-hour. If you are bandwidth-bound and quantisation doubles decode throughput, tokens/sec per dollar roughly doubles on the same card, so buy no hardware; if you are compute-bound near the FLOPS ceiling, a faster card is justified. The decision is defensible because it targets a located bottleneck rather than a headline figure. Why is a single headline throughput number a poor basis for capacity planning? A headline number bakes in one prompt length, one output length, one batch size, and one phase mix, and it silently assumes production traffic matches that configuration — which it almost never does. Real traffic varies all of those request to request, so the effective rate moves, often by a large factor. Planning against the single figure leads to over-provisioning when the true bottleneck was elsewhere or under-provisioning when concurrency behaviour was worse than a single-stream test suggested. Where this leaves your sizing decision An ollama-benchmark run is the entry-level version of the profiling step that decides whether you buy hardware, quantise a model, or reconfigure the serving path. The same profiling-first discipline that our GPU performance work applies to simulation workloads applies here: locate the bottleneck before you spend. The A1 GPU audit formalises that step — separating prefill from decode, sweeping the real load envelope, and reading the roofline — so the sizing decision rests on where the constraint actually lives, not on a single number whose provenance nobody checked. The open question worth carrying forward is your traffic distribution itself: until you know the prompt-to-output ratio and concurrency your production workload will really see, even a perfectly-read benchmark is measuring the wrong point on the curve.
