A committee is looking at a slide that says a local model runs at 58 tokens/sec with a 240 ms time-to-first-token on the team’s workstation, and someone asks whether that means it is the right model to buy. It does not. The number is real, but it answers a different question than the one on the table. This is the recurring failure with Ollama benchmarks. Ollama makes it trivial to pull a model, run it on local hardware, and read off throughput, time-to-first-token, and latency-versus-model-size in a few minutes. The convenience is genuine. The trouble is what happens next: those figures get lifted out of a terminal and dropped into an approval discussion as if they settled the model choice. They didn’t. An Ollama benchmark is a serving-throughput and local-hardware-fit signal — nothing more, and nothing less. That distinction matters because it determines where the numbers belong in a procurement-grade evidence pack. Read correctly, Ollama benchmarks are excellent input to the cost-per-decision and latency-under-load sections. Read naively, they get mistaken for the whole picture and collapse the first time a reviewer asks about the buyer’s actual task, data, or risk tolerance. What do Ollama benchmarks actually measure? When you run a model through Ollama and time it, you are measuring how fast a specific model, quantised a specific way, serves tokens on a specific piece of hardware under whatever load your test harness applied. Three figures dominate the output: Tokens per second — sustained generation throughput once the model is producing output. Time-to-first-token (TTFT) — how long the prompt takes to process before the first token appears, dominated by prompt length and prefill compute. Latency versus model size — how the two figures above shift as you move between, say, an 8B and a 70B variant, or between quantisation levels. All three are properties of the serving stack: the model weights, the quantisation format, the runtime (Ollama sits on top of llama.cpp and its GGUF-format kernels), and the hardware — CPU, GPU VRAM, memory bandwidth. On an Apple M-series chip the bottleneck is usually unified-memory bandwidth; on a discrete GPU it is VRAM capacity deciding whether the model fits at all, then compute. These are benchmark-class measurements when the hardware and quantisation are named — reproducible on the same setup. What none of them touch is whether the model is correct on your task. A 70B model quantised to 4 bits might serve at 30 tokens/sec and produce confidently wrong answers on your domain; an 8B model might serve at 90 tokens/sec and be perfectly adequate, or hopeless, depending entirely on the workload. Throughput and accuracy are orthogonal axes. Ollama measures the first and is silent on the second. This is the same gap we treat carefully in Ollama benchmarking for regulated AI, where the numbers feed one section of the evidence pack and not another. Why the throughput number can’t answer the accuracy question The temptation is to treat a fast local model as a good one. That confuses two things that live in different parts of an evaluation. Serving throughput is a function of how much compute it takes to move a token through the model on your hardware. Task accuracy is a function of what the model learned, how well its training distribution matches your data, and how it behaves on the edge cases that matter to you — the long-tail inputs, the adversarial prompts, the failure modes that only show up under your risk tolerance. No amount of tokens-per-second tells you anything about those. A benchmark that runs your prompts and never checks the answers is measuring the engine, not the driving. This is exactly the trap covered in why the leaderboard number isn’t your number when you separate an inference benchmark from a workload evaluation. An Ollama run is an inference benchmark on your own metal. It tells you the local-serving story with unusual honesty — because it is your hardware, not a vendor’s lab — but it still stops exactly where task evaluation begins. When a committee asks “what about our task, our data, our risk tolerance?”, the throughput slide has nothing to say. That silence, mistaken for a green light, is the divergence point where naive readings do damage. Where Ollama numbers belong in a procurement-grade pack The fix is placement, not suppression. Ollama benchmarks are good evidence for the operational-feasibility and cost sides of a decision. They are not headline evidence for the model choice itself. Here is the division that keeps a pack defensible. Evidence-pack section Does an Ollama benchmark belong here? Why Task accuracy / quality No Throughput says nothing about correctness on your data Failure-mode / safety No Requires adversarial and edge-case task evaluation Cost-per-decision Yes Tokens/sec + hardware cost → per-decision serving cost Latency-under-load Yes TTFT and throughput under real concurrency → p95 latency Operational feasibility Yes Whether the model fits your VRAM / memory budget at all Vendor / licence risk Partly Local serving confirms on-prem viability; licence is separate The pattern is simple to state and easy to get wrong. If a number describes how well the model does the job, an Ollama benchmark does not produce it. If a number describes how expensive and how fast it is to serve the model on hardware you control, an Ollama benchmark is one of the best signals you have — and better than a vendor spec sheet, because it is measured on your setup rather than a reference rig. How to convert tokens/sec and TTFT into committee-grade figures Raw throughput does not survive a first challenge. “58 tokens/sec” invites the question “under what load?” and “at what cost?” — and if you cannot answer, the number is anecdote, not evidence. Converting the local measurement into decision-grade figures is a short, explicit calculation. Here is a worked example with the assumptions stated up front. Assumptions (illustrative, state your own): Measured sustained throughput: ~55 tokens/sec at your test concurrency Average decision consumes ~800 output tokens Hardware amortised cost: ~$0.40/hour for the workstation class under test Measured TTFT at target concurrency: ~260 ms; p95 ~ 410 ms Cost-per-decision: at ~55 tokens/sec, 800 tokens take roughly 14.5 seconds of serving time. At $0.40/hour that is about $0.0016 per decision in raw compute. That figure is only meaningful at the stated concurrency — push more parallel requests through the same hardware and both throughput-per-request and cost shift. This is an observed-pattern-class conversion: the arithmetic is exact, but the inputs are specific to your rig and load, so label it as such and never present it as a universal rate. Latency-under-load: the number that matters is not the median TTFT on an idle box, it is the p95 under the concurrency your deployment will actually see. Ollama’s single-shot benchmarks default to a quiet machine; you have to drive real parallel load to get a defensible p95. Report the concurrency level alongside the latency, always. Done this way, two local-serving options compare on a like-for-like basis: same prompt distribution, same concurrency, same hardware class, per-decision cost and p95 latency side by side. That like-for-like framing is the whole point — it is what lets a committee choose without arguing about whose test conditions were fairer. The broader procurement-evaluation methodology for AI infrastructure treats this feasibility layer as one input among several, sitting beside the accuracy and failure-mode evidence that Ollama cannot supply. For the governance framing that ties these layers together, our work on AI governance and trust sets out how the pieces assemble into a defensible whole. How should local-hardware results be represented for comparison? Represent them as measured operating points, with their conditions attached — never as bare headline numbers. A tokens/sec figure with no concurrency, no quantisation format, and no hardware label is not comparable to anything and should not enter a pack. The minimum viable representation for each option: Model + quantisation (e.g. llama3:8b-q4_K_M) — because the same model at a different quantisation is a different operating point Hardware class and the binding constraint (VRAM-bound, bandwidth-bound, compute-bound) Throughput and TTFT at a named concurrency, with p95 not just median Derived cost-per-decision with its assumptions listed An explicit note that these figures are operational-feasibility evidence, not accuracy evidence The same measurement discipline applies whether you are reading a local Ollama run or a published hardware benchmark; our take on reading MLPerf and hardware inference benchmarks honestly for deployment makes the same argument from the vendor-benchmark side. Ollama’s advantage is that the conditions are yours to control and disclose; use that advantage by disclosing them. FAQ What should you know about ollama benchmarks in practice? Ollama pulls a quantised model and runs it on your local hardware through the llama.cpp runtime, then times generation. In practice it produces tokens/sec, time-to-first-token, and how those shift with model size — all properties of the serving stack (weights, quantisation, runtime, hardware), measured on your own metal rather than a vendor’s reference rig. What do Ollama benchmarks actually measure — throughput, time-to-first-token, latency — and what do they not measure? They measure serving throughput (tokens/sec), prompt-processing latency (time-to-first-token), and how both scale with model size and quantisation on given hardware. They do not measure task accuracy, correctness on your data, or failure modes — throughput and accuracy are orthogonal axes, and Ollama is silent on the second. Where do Ollama benchmark numbers belong in a procurement-grade evaluation pack, and where do they mislead? They belong in the cost-per-decision, latency-under-load, and operational-feasibility sections. They mislead when placed in the task-accuracy or failure-mode sections, or presented as the headline verdict for a model choice — because they say nothing about whether the model is correct on your task. How do you convert raw tokens/sec and time-to-first-token into cost-per-decision and latency-under-load for a committee comparison? Combine throughput with average tokens-per-decision and amortised hardware cost to get per-decision cost, and report p95 time-to-first-token at your target concurrency rather than median on an idle machine. State every assumption — concurrency, token count, hardware class — because the arithmetic is exact but the inputs are specific to your setup. Why can’t Ollama benchmarks answer task-specific accuracy or failure-mode questions on their own? Serving throughput is a function of compute-per-token on your hardware; task accuracy is a function of what the model learned and how well it matches your data and edge cases. A benchmark that runs prompts without checking answers measures the engine, not the driving, so it cannot speak to correctness or safety. How should local-hardware benchmark results be represented so options compare on a like-for-like basis? Represent each option as a measured operating point with its conditions attached: model and quantisation, hardware class and binding constraint, throughput and p95 latency at a named concurrency, derived cost-per-decision with assumptions listed, and an explicit note that these are feasibility not accuracy figures. Same prompt distribution, same concurrency, same hardware class, side by side. The decision this evidence actually supports The convenience of Ollama is also its risk. It puts a hard, honest local-serving number in front of a decision-maker in minutes — and that number is easy to over-read. The discipline is not to distrust the figure but to know which section of the pack it belongs in, and to refuse to let it stand in for the accuracy and failure-mode evidence it was never measuring. When you next see a tokens/sec slide carried into an approval meeting, the useful question is not “is that fast enough to buy?” but “what concurrency was that measured at, and where does it sit relative to the task evidence?” — the answer tells you whether the pack is decision-grade or just decorated with numbers.
