Ollama Benchmark: Measuring Local LLM Serving Before It Hits Production

Benchmark Ollama models on your own hardware: tokens/s, TTFT, VRAM, quantisation, why single-user numbers fail under load, and when to switch engines.

Ollama Benchmark: Measuring Local LLM Serving Before It Hits Production
Written by TechnoLynx Published on 11 Jul 2026

To benchmark Ollama models honestly, measure five numbers on your own machine: generation tokens per second, time to first token (TTFT), prompt-processing rate, peak memory, and concurrent throughput. You measure them after a warm-up run, with the prompt, quantisation and context length fixed and written down. The single-user tokens-per-second figure most home rigs report is not the same as multi-user throughput. What follows is the method we would use, what quantisation and VRAM do to the results, and how to tell when Ollama has stopped being the right serving engine.

Start with the One-Command Floor, Then Treat It as a Floor

The quickest local measurement is ollama run <model> --verbose with a short prompt. The eval rate line it prints is generation speed in tokens per second, and prompt eval rate is prompt-processing speed. That single observation is a floor, not a benchmark.

The API responses carry these timings in a form you can script. Ollama’s API responses include load_duration (model load time), prompt_eval_count and prompt_eval_duration (input tokens and the time spent on uncached ones), prompt_eval_cached_count (prompt tokens served from cache), eval_count and eval_duration (output tokens and generation time), and total_duration. Generation tokens per second is eval_count divided by eval_duration once the duration is converted to seconds, and prefill rate is prompt_eval_count divided by prompt_eval_duration the same way. A large load_duration in a timed run marks a cold start, which makes that speed figure meaningless.

How Do You Benchmark an Ollama Model’s Tokens per Second and Latency on Your Own Hardware?

You turn the single observation into a repeatable harness: same prompt, same quantisation file, same context size, same temperature, all documented. The order matters, so here it is as a procedure.

  1. Pin the configuration. Record the model tag, quantisation, context window, prompt, temperature, seed where supported, Ollama version and hardware. Ollama applies its own default system prompt and sampler settings, so set these explicitly when you compare it with another runner.
  2. Warm up. The first run loads weights from disk and is not representative. Preloading with an empty request to /api/generate or /api/chat keeps the model resident.
  3. Repeat. Five iterations with the first dropped is the floor for a stable mean; a P99 latency needs at least about 100 requests to exist at all.
  4. Fix the prompt shape. A 4-token prompt and a 4,000-token prompt give very different prefill rates, so use the prompt length your workload actually sends. One reasonable single-figure baseline is generation tokens per second at concurrency 1 with a fixed 512-token prompt and 128-token output, averaged over five runs after warm-up.
  5. Set temperature to what production uses. On identical hardware, one study measured 6,767 tokens/s at temperature 0 against 5,573 tokens/s at temperature 0.7 in a 1,000-request batch, a 21% throughput difference. Benchmarks defaulting to temperature 0 fail to reflect the compute needs of non-deterministic workloads.
  6. Sample memory during the run. nvidia-smi --query-gpu=memory.used --format=csv -l 1 while the benchmark runs, keeping the peak; on Apple Silicon, powermetrics and memory_pressure give the equivalent.
  7. Sweep concurrency. Covered below, because most home rigs are tested only at concurrency 1.

For latency, measure both TTFT and time per output token (TPOT). Bench360 (2025) defines TTFT as the delay from issuing a request to receiving the first token, and TPOT as total generation time divided by the number of generated tokens. TTFT dominates perceived latency for short prompts and chat. For how streamed output keeps that first token fast in chat interfaces, see real-time streaming for generative AI applications.

If you want a lower-level cross-check of the same GGUF file, llama.cpp’s llama-bench runs prompt-processing (pp), text-generation (tg) and combined tests, repeats each five times by default, and reports mean tokens per second with standard deviation. Its timings exclude tokenisation and sampling. Our llama.cpp benchmark guide to CPU/GPU inference measurement covers that tool in depth.

What to Record, and Where It Comes From

Metric What it tells you Where to get it
Generation tokens/s Steady-state output speed eval_count / eval_duration, or eval rate in --verbose
Prefill tokens/s How fast long prompts are ingested (RAG, coding agents) prompt_eval_count / prompt_eval_duration
TTFT Perceived latency for chat Time from issuing a request to the first token
Load time Cold-start cost load_duration
Peak VRAM Headroom for context and parallel slots nvidia-smi sampled during the run
Concurrency curve Real capacity of the box Aggregate tokens/s at rising parallel load

How Quantisation Changes Ollama Models’ Speed and Quality

Quantisation shrinks the model and can speed up inference, and it can also cost accuracy, so a benchmark has to measure both halves. On size, the llama.cpp project lists Llama 3.1 8B at 32.1 GB originally and 4.9 GB at Q4_K_M, and 70B at 280.9 GB against 43.1 GB.

Speed does not simply improve as bits go down. Bench360 found that INT8 and INT4 let 2 to 4 times larger models fit a 24 GB budget and often score higher, but TPOT and energy per token rose by more than 350% in the worst case (Qwen2.5-32B at INT4), because the larger model still needs more operations per token. A quantised 32B model and a 7B model at FP16 are different products with different latency profiles, not the same model at different speeds.

On quality, the most comprehensive study to date by its own account (over 500,000 evaluations on the Llama-3.1 family) found FP8 effectively lossless, well-tuned INT8 at 1 to 3% accuracy degradation, and INT4 weight-only quantisation competitive with 8-bit. That study analysed inference performance through vLLM. For GGUF specifically, a study of llama.cpp quantisation evaluated 3 to 8-bit K-quants and legacy formats on Llama-3.1-8B-Instruct for task scores, perplexity, CPU throughput and size in one place.

Two practical points follow. Small models are hit hardest: the morphllm guide reports that models under 7B lose the most quality at 4-bit, so a larger model at Q4 usually beats a tiny one on the same hardware. And the default tag matters: the morphllm model guide says Ollama defaults to Q4_K_M, while a markaicode test says it pulled Q4_0 unless the FP16 tag was requested. We would read the quantisation off the actual tag rather than assume it. Either way, comparing a Q4_K_M run on one machine with a Q8_0 run on another compares models, not machines.

To measure the quality loss yourself, llama.cpp’s perplexity tool compares a quantised model against FP16, and its KL-divergence output shows how far the quantised token distribution has moved, with 0 meaning identical. Symmetric percentiles suggest the quantisation is mostly adding noise; a heavier negative tail means the model is actually getting worse. An importance-matrix file (--imatrix) during quantisation can reduce that loss.

How Much VRAM an Ollama Model Needs, and What Happens When It Does Not Fit

VRAM use is the model weights plus the KV cache, and the KV cache grows roughly linearly with context length. One guide’s measurements put an 8B model’s KV cache at about 0.3 GB at 2K context, about 5 GB at 32K and about 20 GB at 128K. Another puts the KV overhead at default context at roughly 2 to 6 GB on top of the download size.

Ollama picks a default context from your VRAM: 4k tokens below 24 GiB, 32k between 24 and 48 GiB. The same documentation recommends at least 64,000 tokens for agents, web search and coding tools, and warns that a larger context needs more memory. That gap is where many benchmarks quietly go wrong: a model that looked fast at the 4k default is a different memory proposition at the context a coding agent needs, so benchmark at the context you will ship.

Parallel requests multiply the memory bill. Ollama’s FAQ states that required RAM scales with OLLAMA_NUM_PARALLEL times OLLAMA_CONTEXT_LENGTH, so a 2K context with 4 parallel requests becomes an 8K allocation.

When the model does not fit, Ollama splits layers between GPU and system RAM rather than failing. ollama ps shows the result in its PROCESSOR column: 100% GPU, 100% CPU, or a split such as 48%/52% CPU/GPU. The speed penalty is large. One guide measured Qwen 3 8B falling from 40 tokens/s fully in VRAM to 8 tokens/s with 25 of its 36 layers on the GPU, and attributes the bottleneck to PCIe transfers. Those are one blog’s figures on one setup, but the direction matches Ollama’s own advice to avoid offloading to CPU for best performance. One benchmarking guide notes an exception worth testing: for models under about 3B parameters, CPU can win because PCIe overhead dominates, so benchmark both backends. For prosumer AI hardware sized to run local models like these, see Furball Factory.

Why Published Ollama Benchmarks Fail Under Concurrent Production Load

Single-stream tokens per second does not predict multi-user behaviour, and most home benchmarks are run at concurrency 1.

Ollama’s FAQ gives OLLAMA_NUM_PARALLEL a default of 1 and OLLAMA_MAX_QUEUE a default of 512, beyond which requests are rejected. Red Hat’s 2025 comparison described a default of four parallel requests and found Ollama’s throughput plateaued even after tuning, while vLLM’s scaled almost linearly up to concurrency 64, the highest level it reported. The two disagree on the default; we would trust the vendor’s current documentation for your version, and the disagreement itself is a reason to set the value explicitly rather than inherit it.

At least one model architecture ignores the setting entirely. A 2026 re-test by ComputingForGeeks found that Ollama classifies Qwen3.8 as the qwen35 architecture, refuses parallel slots for it and launches its runner with one slot regardless of the environment. Sent 32 requests on an RTX 4090, the median request waited over a minute and a half for its first token. The same GGUF served 32 parallel slots under upstream llama.cpp.

A multi-scenario evaluation published by MDPI found Ollama needed 54 to 122 seconds for the first token, hit a bottleneck near 10 concurrent users and showed timeout-based error rates of 13 to 30.06% under heavier loads, while vLLM produced first tokens in 0.5 to 3.5 seconds and stayed stable up to 100 users. The same study saw significant memory growth in both engines during long endurance runs, which is a reason to run soak tests, not just bursts.

The Benchmark Client Can Be the Bottleneck

Your load generator can inflate the result too. A 2026 study showed that single-process, asyncio-based Python benchmarking clients are throttled by the Global Interpreter Lock as request rates rise, and record their own queuing as server latency. At 1,000 queries per second, single-process tools added between 8 milliseconds and 58 seconds of artificial delay, against 0.63 milliseconds for a multi-process design. A single local Ollama box will not see those rates, but the lesson carries: if TTFT climbs, confirm the client is not saturated before blaming the server.

The same study notes that decode TPOT often stays flat on a loaded system while queue time and prefill degrade sharply, which is why it proposes normalised TPOT (end-to-end latency divided by output tokens) as an early saturation signal. In Ollama terms, a queue shows up as time to first token, not as GPU utilisation, so graph both.

The concurrency sweep that answers the capacity question is simple: test at 1, 4, 8 and 16 parallel streams, keep doubling until aggregate throughput stops rising, and read the turnover point as your capacity. That point depends on the card, the quantisation and the context length, so a figure from someone else’s rig does not transfer.

When Is Ollama Good Enough for Production Serving, and When Should You Move to vLLM, SGLang or llama.cpp?

Ollama is good enough for one person at a keyboard or a low-traffic demo. Beyond that, the measured evidence favours an engine built for continuous batching.

For a single user, Ollama with its default multi-token prediction draft was the fastest option on a 24 GB card in one test. ComputingForGeeks measured its default multi-token prediction draft lifting decode from 39.9 to 76.2 tokens/s on an RTX 4090 and from 76.8 to 129.7 on an RTX 5090. A separate single-rig test found Ollama ahead at one stream (45 against 38 tokens/s), with a 3.2-second cold start against vLLM’s 8.7, but vLLM ahead at eight concurrent requests (187 against 82 tokens/s) on Llama 3 8B at FP16. That author’s own advice is to confirm both on your hardware and current versions before committing.

Situation Engine the evidence points to Why
One user at a keyboard, local development Ollama Easiest to run; fastest single-user decode on a 24 GB card in one 2026 test
Several users on a 24 GB card llama.cpp server Matched vLLM throughput on an RTX 4090 and kept first token at 6.5 s with 32 requests
Shared endpoint with VRAM headroom vLLM 936 tokens/s across 32 callers with a 2.3 s first token on an RTX 5090
Light to moderate concurrency SGLang is worth testing Performed well at light to moderate load in Bench360 but breached the TTFT threshold earlier on an L4
Frequent cold scale-ups Ollama, or keep a vLLM instance warm Lower cold-start time in one test

Two caveats sit under that table. The llama.cpp 32-slot result leaves each slot only 1,024 tokens of context on a 24 GB card, so real prompts mean trading slots for context. And in one test’s informal spot checks, the gap between engines tends to narrow with RAG prompts of 8K tokens or more. Bench360’s broader conclusion holds: there is no single best local setup, only a measured fit to a deployment. For a wider view on LLM inference numbers that match production, our Llama benchmark guide to production-matching inference measurement goes beyond Ollama.

Comparing Ollama Models for a Coding or RAG Workload

Model lists tell you how big Ollama models are; they say very little about how well each does your task on your hardware. One curated list gives qwen3-coder:30b as 19 GB at Q4_K_M with a 256K context but notes it has no published numeric benchmark on its card, and devstral:24b as 14 GB at Q4_K_M with 46.8% on SWE-Bench Verified. That is useful for shortlisting by VRAM and not much more.

The comparison we would run uses your own prompts. Take a set of real tasks from the workload (code edits against your repositories, questions against your documents), score them with a task metric you trust, and only then measure system metrics for the survivors at the quantisation and context you would ship. Bench360 likewise combines user-defined quality metrics such as accuracy or F1 with latency, memory, cold start and concurrency in one framework.

For coding, precision matters more than the averages suggest: one guide recommends q8_0 over Q4_K_M when VRAM allows, because 4-bit errors show up as subtle off-by-one and API-misuse mistakes. For RAG, prefill dominates: long retrieved contexts make prompt-processing rate and TTFT the numbers to watch, and normalised TPOT folds that prefill cost into one figure. Report prompt_eval_cached_count alongside, since cache hits change TTFT based purely on request order. Our retrieval-augmented generation examples and guidance covers the pipeline side of that workload.

Profile the Serving Path Before Switching Engine or Model

We would profile first, because the levers that usually decide cost and latency (quantisation, context sizing, parallel slots, caching, runtime choice) sit below the model and can be tuned independently. Replacing the model is the most expensive lever and often leaves the workload economics unchanged.

The diagnostic order is short. Run ollama ps and confirm the model sits at 100% GPU. Divide your card’s published memory bandwidth by the model file size to get a ceiling for decode speed; if you land at a small fraction of it, the usual causes are partial CPU offload, the model reloading between requests, and thermal throttling. Then graph TTFT alongside GPU utilisation under real load, with DCGM and Grafana or equivalent, because queuing appears in the first and not the second. High utilisation is not proof of efficiency either: Bench360 found energy per token and GPU utilisation were not strongly correlated across engines.

Which context decides the next step comes from that profile, not from a general rule about which engine is faster. TechnoLynx audits production AI stacks to find these cost and latency improvements before recommending a model replacement, as part of our MLOps engineering work. Once the numbers are trustworthy, the next job is keeping them that way across Ollama upgrades and model updates, which is what turning local LLM numbers into regression baselines covers.

FAQ

What Is the Best Ollama Model?

No single setup is best. The right choice depends on your VRAM, the context length you need and the task you run, and Bench360 (2025) concluded that the right combination of model, quantisation and engine has to be measured against each deployment’s requirements.