A quick Ollama benchmark is ollama run <model> --verbose. After each response it prints two separate speeds: the prompt eval rate, which is how fast the prompt was processed, and the eval rate, which is how fast new tokens were generated. The eval rate is the generation speed in tokens per second. The API gives you the raw fields to compute it yourself: divide eval_count by eval_duration, and remember that Ollama reports durations in nanoseconds. Read that way, the number describes how fast one user gets output on one machine with one model build. It does not tell you how a shared service built on the same stack would behave.
Running an Ollama Benchmark from the Command Line and the API
Ollama makes measurement easy with the --verbose flag. Here is a run of gemma2:2b on a short prompt, as published in a Hexmos debugging guide:
ollama run gemma2:2b "Hello World" --verbose
total duration: 5.564087708s
load duration: 4.478411408s
prompt eval count: 11 token(s)
prompt eval duration: 391.177149ms
prompt eval rate: 28.12 tokens/s
eval count: 18 token(s)
eval duration: 610.898556ms
eval rate: 29.46 tokens/s
For scripted runs, the API’s final response carries the same timings as integers. The Ollama documentation defines them like this:
total_duration: time spent generating the whole response, in nanoseconds.load_duration: time spent loading the model, in nanoseconds.prompt_eval_count: the number of input tokens in the prompt.prompt_eval_duration: time spent evaluating uncached prompt tokens, in nanoseconds.eval_count: the number of output tokens generated.eval_duration: time spent generating those tokens, in nanoseconds.
The word “uncached” matters. If part of a repeated prompt is served from cache, dividing prompt_eval_count by prompt_eval_duration can overstate prefill speed, so we would measure prefill on prompts the model has not just seen.
Prompt Eval Rate or Eval Rate: Which Tokens-per-Second Number Matters?
The two rates measure different phases of inference, and each phase has its own bottleneck. In the prefill phase the whole input prompt is processed and the first output token is produced. In the decode phase, output tokens are generated one at a time, and each new token goes back through the model to produce the next one. Prefill runs in parallel over the prompt and is compute-bound. Its cost grows quadratically with input length. Decode is memory-bound, with intensive memory access driven by the KV cache.
In practice, prompt processing sets how long you wait for the first token, and that wait grows with prompt length. Eval rate governs how fast text streams once it starts.
Worked Example: Turning eval_count and eval_duration into Tokens per Second
The Ollama API reference shows a final response from llama3.2 with eval_count 282, eval_duration 4,535,599,000, prompt_eval_count 26, prompt_eval_duration 342,546,000, load_duration 1,334,875 and total_duration 4,883,583,458. Here is the arithmetic:
| Quantity | Working | Result |
|---|---|---|
| Decode (eval rate) | 282 tokens ÷ 4.5356 s | about 62.2 tokens/s |
| Prefill (prompt eval rate) | 26 tokens ÷ 0.3425 s | about 75.9 tokens/s |
| Model load | 1,334,875 ns | about 1.3 ms |
| Total | 4,883,583,458 ns | about 4.88 s |
The load time of about a millisecond fits a model that was already loaded. The verbose run above looks different: 4.48 s of its 5.56 s total was model loading. The Hexmos guide notes that load is usually slow only on the first run and much faster afterwards if the model stays loaded. A cold first run carries the slow model load in its timing. Discard that run, or preload the model before you start the timer. The Ollama FAQ describes preloading as sending the server an empty request to /api/generate or /api/chat.
What Counts as a Good Tokens-per-Second Figure for a Local LLM?
For interactive chat, the token rate you need follows from reading speed. The Etalon paper (Agrawal et al., 2024) works out that an average English reading speed of 250 words per minute needs a token rate of roughly 6 tokens per second. A 2025 study of LLM serving evaluation uses the same 6 tokens per second as typical reading speed. Above that floor, Inventive HQ finds a 10% speed gap invisible in interactive chat. For batch work, though, the gain is real: Inventive HQ notes that a 10% throughput loss means 10% more wall-clock time when you process thousands of prompts.
Your own figure depends on factors you can name:
- Model size against memory bandwidth. Single-user decoding reads the entire model from memory for every token, so the limit is memory bandwidth, not raw compute.
- Quantization, and what you use it for. Quantizing the same model can make it faster: on consumer Blackwell GPUs under vLLM, a 2026 study measured NVFP4 at 1.6× the throughput of BF16, with 2-4% quality loss. Using quantization to fit a larger model is a different trade. Bench360 found INT8/INT4 let 2-4× larger models fit in 24 GB, but TPOT rose by more than 350% in the worst case, Qwen2.5-32B at INT4.
- Context length. In the same Blackwell study, doubling the context length roughly halved throughput, and 32k context cost 5.5× the 8k baseline in energy per token.
- Where the layers actually live.
ollama psshows whether a model loaded as 100% GPU, 100% CPU or split across both, such as 48%/52% CPU/GPU. A model that spills into system memory gives a very different number from one that fits.
So whether 30 tokens per second is good depends on your workload. It is good if it clears reading speed for chat, or meets your batch deadline, on the model and quantization you actually intend to run.
A Fair Ollama Benchmark Run, Step by Step
These steps run in order.
- Confirm placement with
ollama psand make sure the model shows 100% GPU if that is what you mean to measure. - Preload the model or throw away the first run, so
load_durationstays out of the timing. - Fix the prompt set and the sampling settings. NVIDIA notes that sampling choices such as greedy, top_p, top_k and temperature can change generation speed, and recommends keeping them consistent within one setup.
- Repeat each measurement and report the spread as well as the mean. This is how llama-bench reports results: it averages repetitions and gives the standard deviation.
- Record the Ollama version, the context length and
OLLAMA_NUM_PARALLEL. The FAQ notes that required RAM scales withOLLAMA_NUM_PARALLEL×OLLAMA_CONTEXT_LENGTH. - If you care about more than one user, sweep concurrency from 1 to a value not much above the engine’s maximum batch size. Throughput usually saturates around that point while latency keeps rising.
Which Tools Benchmark Ollama, and What Does Each Measure?
Some read Ollama’s output, some drive the llama.cpp engine directly, and some load-test a serving endpoint.
| Tool | What it measures | Fits when |
|---|---|---|
| llm-benchmark (ollama-benchmark) | A cross-platform CLI that measures tokens-per-second throughput of local models through Ollama | You want one quick throughput figure per model on your machine |
| LLM-Bench | Token generation speed, execution time, prompt and generated token counts, memory and CPU usage, Metal acceleration status, and GPU power with --gpu-metrics |
You run Ollama on Apple Silicon, which the tool is optimised for |
| llama-bench (llama.cpp) | Prompt processing (pp), text generation (tg) and the two combined (pg), with defaults of a 512-token prompt and 128 generated tokens, repetitions and standard deviation | You want the engine’s own speed without Ollama’s layer; it excludes tokenization and sampling time |
| GenAI-Perf (NVIDIA) | Time to first token, end-to-end latency, inter-token latency, tokens per second and requests per second, while holding exactly N requests in flight | You are load-testing a serving endpoint |
| AIPerf | Client-side latency and per-token throughput under configurable request distributions and concurrency, over HTTP and gRPC | You are stress-testing a serving stack |
Two cautions apply across the table. First, NVIDIA points out that these client-side tools define, measure and calculate metrics differently, so a number from one tool is hard to compare with a number from another. Second, a runner’s self-reported rate does not count overhead outside the engine. Inventive HQ argues that wall-clock tokens per second (completion tokens divided by total elapsed time) is the fair way to compare runners, because it counts the HTTP round-trip, the prompt template and the scheduler.
Ollama vs llama.cpp, LM Studio and vLLM on the Same Hardware
For GGUF models, Ollama and LM Studio both wrap llama.cpp, so on the same GPU the tokens come out of the same llama.cpp kernels. With the model file, quant and kernels held equal, Inventive HQ traces any speed gap to three places: the llama.cpp version the wrapper bundles, its default flags (flash-attention, KV-cache type, batch size, context length, GPU layer count), and its own overhead.
Here is how big those gaps were in published measurements:
- llama.cpp against Ollama, single user. Inventive HQ measured llama.cpp ahead by 10.3% on an RTX 5060 Ti (77.0 against 69.1 tokens/s) and by about 14% on an Apple M3 Max (53.5 against 46.2 tokens/s) with a 7B Q4 model.
- LM Studio depends on the platform. In the same tests LM Studio was within noise of llama.cpp on the RTX 5060 Ti (−0.3%) but the slowest of the three on the M3 Max (−29%), which the authors attribute to its bundled llama.cpp lagging upstream.
- The bundled build can matter more than the wrapper. ComputingForGeeks found that Ollama 0.35.1 ships a llama-server built from llama.cpp 0.5.0-dev. On an L40S, that bundled binary decoded at about 20 tokens/s, while upstream v0.6.0 with identical flags, file and GPU reached 39.1. They measured no overhead from Ollama’s Go layer.
- vLLM, single stream. In the ComputingForGeeks runs on Qwen3.8 without speculative decoding, vLLM won single-stream decode on every card by a small margin.
Concurrency is where the engines separate. ComputingForGeeks found that Ollama 0.35.1 refuses parallel slots for the qwen35 architecture and starts its runtime with one slot whatever the environment says, so 32 requests on an RTX 4090 left the median request waiting over a minute and a half for its first token. Red Hat Developer (August 2025) measured vLLM throughput rising with concurrency while Ollama’s stayed flat, and found that Ollama tuned to OLLAMA_NUM_PARALLEL=32 on an A100 still could not keep up. Bench360 likewise found vLLM the most robust engine for multi-user serving.
The sources disagree on one setting. Red Hat states that Ollama handles at most four parallel requests by default, while the current Ollama FAQ gives the OLLAMA_NUM_PARALLEL default as 1. We would trust the FAQ, because it is the vendor’s documentation for its own product, and check the value on whatever version you run. Inventive HQ expects the single-user gap to narrow as Ollama tracks newer llama.cpp releases, calling it “a versioning-and-defaults gap, not an architectural one”. That makes any comparison, or any regression baseline, stale unless it states the versions.
Why a Local Ollama Benchmark Does Not Predict Production Cost or Latency
A single-stream benchmark keeps one prompt in flight and sends the next only when the answer returns, so no queue forms. Bench360’s “server” scenario instead simulates concurrent users, each sending requests independently at a set rate. A single-stream number therefore does not cover queueing.
As load rises, the per-user view and the system view move apart. NVIDIA notes that adding concurrent requests raises total system TPS while TPS per user falls as latency rises. The 2026 Blackwell study gives the conversion: 488 TPS at 64 concurrent users is about 7.6 tokens per user per second. That is barely above reading speed, even though the headline throughput looks large. Where the queue builds, it shows up as time to first token, not as GPU utilisation, which is why ComputingForGeeks recommends graphing both under real load.
Averaged metrics such as TPOT hide the generation stalls users notice. Two systems can have identical TPOT while one streams smoothly and the other produces bursts separated by long stalls. In Etalon’s measurements, Groq reached 600 tokens/s on a TPOT basis, but the tail time-between-tokens metric showed about a quarter of that throughput, which the authors attribute to jitter and stalls between decode iterations. A local eval rate is exactly this kind of average.
The conditions also differ. The Blackwell study is explicit that steady-state measurements on synthetic workloads omit the variable request patterns and cold starts of production. Request handling can dominate, too: one serving-evaluation study cites CPU overheads contributing more than 50% of latency.
Before deciding what to optimise, profile the real serving path. For the next step, see our guide to porting AI inference across runtimes and hardware.
If the open question is the hardware rather than the engine, LynxBenchAI measures throughput, latency, memory and energy across accelerators. LynxBenchAI puts laptop CPUs and data-centre GPUs on one scale. Those results describe the hardware under a controlled method, not your own workload. When the serving path is too slow or too expensive, TechnoLynx’s GPU performance engineering profiles it and tunes kernels and memory to make it faster and cheaper.
So the question to settle before trusting any tokens-per-second figure is how many users will share the machine. With one user, a carefully run Ollama benchmark is close to the answer. If you run local AI for one user on your own prosumer hardware, see Furball Factory. With more than one, measure time to first token and key latency percentiles under a concurrency sweep, which --verbose output does not report.