Run the built-in Ollama eval, note tokens-per-second on your GPU, pick the fastest model, move on. For a hardware-shopping exercise that is fine. For a regulated clinical workflow it produces a performance anecdote, not evidence — and the two look almost identical until an auditor asks the wrong question. The gap is not in the number. A benchmark that reports 42 tokens/second is neither true nor false in the abstract; it is a measurement whose meaning depends entirely on what surrounded the measurement. In a procurement conversation, that context evaporates the moment the decision is made. In a GxP-regulated deployment, that same context is the deliverable. The benchmark is not there to help you buy a GPU. It is there to prove, later, that the deployed model behaves as approved, on the hardware it was approved on, at the version that was signed off. That reframe is what separates a usable Ollama benchmark from a discarded one. Below, we walk through what “count as evidence” actually requires, why tokens-per-second is the least interesting number in the run, and how a captured benchmark converts a multi-day pre-audit re-run into a one-line lookup. How does an Ollama benchmark work, and what does it actually measure? Ollama is a local runtime for serving quantised open-weight models — Llama, Mistral, DeepSeek variants and others — behind a simple API on your own hardware. When you run its evaluation path, it loads a model at a given quantisation level, feeds prompts, and reports throughput and latency. The mechanics are unremarkable and that is the point: it is easy to get a number, which is exactly why the number so often arrives stripped of everything that would make it defensible. What the built-in view measures well is generation speed under the conditions of that specific run — this GPU, this quantisation, this prompt length, this concurrency, this driver and CUDA stack. What it does not measure, and cannot infer, is whether those conditions match the regulated task you intend to run in production. A prompt set of “write me a poem” tells you nothing about latency distribution on a 4,000-token clinical summarisation prompt, and a single mean tokens/second figure hides the tail behaviour that a p95 latency floor cares about. Teams frequently choose Ollama precisely because it keeps model weights and patient data inside the estate rather than shipping prompts to a hosted API. That data-residency advantage is real. But it only counts toward compliance if the benchmark that demonstrates the model works — and the provenance of that benchmark — land inside the HIPAA/GxP evidence pack. A local model with an unrecorded benchmark has kept the data private and lost the audit trail in the same move. Who reads the number decides what the number has to carry The divergence between the naive and the regulated approach is not technical sophistication. It is audience. A procurement engineer reads throughput once, to rank candidate models against a budget, then never looks again. The number’s job ends at the decision. A compliance auditor reads the benchmark eighteen months later as a claim: this deployed model was validated to perform within these bounds, and here is the proof it was measured under the conditions it now runs in. The number’s job has barely started. Those two readers need different things from the same run. The procurement reader tolerates missing context because the context is in their head at decision time. The audit reader has none of that context and cannot ask you to reconstruct it — reconstruction after the fact is precisely the thing an audit is designed to distrust. This is the same distinction that separates a leaderboard figure from a deployment number, which we treat at length in why the leaderboard number isn’t your number: the score is real, but portability to your regulated task is the open question. What run conditions must a benchmark capture to count as evidence? The dividing line is simple to state and easy to fail: a benchmark counts as validation evidence when a second engineer, at a different site, could reproduce it from the record alone. Everything below exists to make that reproduction possible. Minimum capture set for an audit-usable Ollama benchmark Field Why an auditor cares Failure if omitted Model identity + weight hash Proves the artefact benchmarked is the artefact deployed “Llama 3” could be any of a dozen revisions Quantisation level (e.g. Q4_K_M, Q8_0) Accuracy and latency both shift materially with quantisation Numbers unattributable to a runnable config Hardware + driver/CUDA stack Throughput and tail latency are hardware-bound Envelope can’t be tied to approved hardware Prompt set drawn from the real task Generic prompts don’t predict regulated-task behaviour Benchmark measures the wrong workload Latency distribution (p50/p95/p99) Regulated SLAs live in the tail, not the mean Single mean hides the number that fails Task-accuracy measurement + rubric Speed without accuracy is not a validated model No accuracy floor to detect drift against Run timestamp + operator + Ollama version Ties the run to a signed-off point in time Can’t place the evidence in the change record Read down that table and note how little of it Ollama’s default output volunteers. Model name it gives you; model hash, quantisation, prompt provenance and accuracy it does not — those are things you record around the run, not things the tool hands over. This is the practical work of turning a tool output into evidence, and it is a discipline, not a feature. Pinning the model hash and quantisation together matters especially, because extreme quantisation changes model behaviour in ways worth measuring; a benchmark that names the model but not its quantisation has recorded the wrong thing precisely. Which metrics matter beyond tokens-per-second? Tokens-per-second is a marketing metric. It compresses a distribution into a mean and answers a question — “how fast, on average?” — that no regulated SLA is written in terms of. Three measurements matter more for a validated workflow, and each is a distribution, not a point. Latency at the tail is first. A summarisation step that averages 800ms but hits 6 seconds at p99 will breach a clinician-facing SLA on roughly one call in a hundred, and “on average it’s fine” is not a sentence that survives an audit. Record p50, p95 and p99 from a run of representative length prompts, not a warm-up handful. Task accuracy is second, and it is the one procurement benchmarks routinely skip. A fast model that hallucinates a dosage is worse than a slow correct one. Accuracy has to be measured against a rubric tied to the regulated task — extraction correctness, clinically-relevant error rate, whatever the step demands — and captured as a distribution across the prompt set. How you choose and report those metrics is itself a defensible-choices exercise; our note on choosing what to report in a model-evaluation pack covers the trade-offs. Third is stability across repeated runs. A single benchmark is one sample. Two runs a week apart on the same config, with the same numbers within a tolerance you state in advance, is what lets you claim the measurement is a property of the system rather than of the afternoon you ran it. Quick answer — the three numbers to pin, and why: p95/p99 latency (not mean) — regulated SLAs live in the tail. Task accuracy distribution against a rubric — speed is worthless if the model is wrong. Run-to-run stability — proves the number is a system property, not a fluke. A single mean tokens/second figure carries none of these and is not, on its own, validation evidence — it is an observed-pattern performance note (not a benchmarked accuracy claim) until the accuracy rubric and distributions are attached. How do you turn a benchmark into an approved operating envelope? Here is where the captured numbers start paying rent. An approved operating envelope is a small set of bounds — say, p95 latency ≤ 2s and task-accuracy floor ≥ 0.95 on the defined rubric — fixed at validation and recorded in the pack. It converts the benchmark from a snapshot into a contract. The value shows up on change. When someone swaps the GPU, upgrades Ollama, or moves to a new quantisation to save memory, the envelope tells you exactly what to re-measure and exactly what “still valid” means. Drift becomes measurable rather than argued. Instead of a room full of people debating whether the new setup “feels the same,” you re-run the pinned prompt set and check the two numbers against the recorded floor. If they hold, the change is inside the approved envelope and the evidence is a one-line addendum. If they don’t, you have caught a regression before it reached a patient, not after. This is also what stops teams re-litigating model performance at every audit cycle. The first validation is expensive; every subsequent audit is a lookup against a fixed envelope — provided the envelope was written down as bounds, not buried as a paragraph of prose. The reliability side of this discipline, where accuracy floors feed a validation pack, is the same shape we describe for producing approval-grade evidence for a reasoning model; the evidence formats have to align so a change in one place doesn’t orphan the other. Where do the results sit in the HIPAA/GxP evidence pack? The mechanical answer: benchmark results belong in the same pack section as the rest of the validation evidence for the specific regulated step they support — not in an appendix of raw logs, and not in a separate performance report nobody links to. The tie is explicit. An Ollama benchmark is only audit-usable when its run conditions and version pins are recorded alongside the validation evidence for the step it underwrites. This is the anchoring logic behind our AI governance and trust practice: evidence is only evidence when it is filed against the claim it supports. Practically, that means each regulated step that invokes the model carries a reference to: the model hash and quantisation, the hardware and stack, the prompt set and its provenance, the measured latency and accuracy distributions, and the approved envelope derived from them. An auditor tracing “does this step use a validated model?” should reach the benchmark in one hop, and the benchmark should answer “validated under what conditions?” without a follow-up email. Self-hosting via Ollama changes the lineage story A hosted LLM API and a self-hosted Ollama deployment produce different data-handling lineages, and the difference matters to the pack. With a hosted API, prompts — which in a clinical setting contain PHI — leave the estate; your lineage story includes a third party’s data-processing agreement, their security posture, and their model-version opacity. With Ollama, the weights and the inference stay inside your boundary, which simplifies the residency argument considerably. But self-hosting shifts the burden rather than removing it. You now own the version control the hosted provider used to manage: which weights, which quantisation, which runtime, pinned and recorded. The upside is that you can pin them, exactly, forever — a hosted model can change under you between audits, whereas a hashed local weight file does not. That controllability is the compliance case for local models, and it is only realised if the pinning discipline above is actually practised. The workflow-readiness dimension of choosing local for data residency sits with the life-sciences lens; our treatment of what makes an AI workflow HIPAA- or GxP-ready is the right companion read there. FAQ What does working with an Ollama benchmark involve in practice? Ollama loads a quantised open-weight model on local hardware and reports throughput and latency for a given prompt set. In practice the raw output measures generation speed under one specific configuration; its meaning for a regulated workflow depends entirely on whether that configuration and prompt set match the deployed task and whether the run conditions were captured. What run conditions must be captured for an Ollama benchmark to count as validation evidence? At minimum: the model identity and weight hash, the quantisation level, the hardware and driver/CUDA stack, the prompt set drawn from the real task, the latency and accuracy distributions, and the run timestamp, operator and Ollama version. The test is reproducibility — a second engineer at another site should be able to recreate the run from the record alone. Which metrics beyond tokens-per-second matter, and how do you record them? Tail latency (p50/p95/p99), task accuracy measured against a task-specific rubric, and run-to-run stability all matter more than mean tokens/second for a regulated workflow. Record each as a distribution across a representative prompt set rather than a single number, because regulated SLAs and accuracy floors live in the tail and across repetitions, not in the average. How do you define an approved operating envelope so changes trigger a drift check? Fix a small set of bounds at validation — for example p95 latency ≤ 2s and a task-accuracy floor on the defined rubric — and record them in the pack as bounds, not prose. When the GPU, runtime, or quantisation changes, re-run the pinned prompt set and compare against those bounds; drift becomes a measurable pass/fail rather than a subjective debate. Where do Ollama benchmark results sit inside the HIPAA/GxP evidence pack? They belong in the same pack section as the rest of the validation evidence for the specific regulated step they support, with run conditions and version pins recorded alongside. An auditor tracing whether a step uses a validated model should reach the benchmark and its conditions in one hop. How does self-hosting via Ollama change the data-handling lineage compared with a hosted API? Self-hosting keeps prompts and weights inside the estate, simplifying the data-residency argument but shifting the version-control burden onto you. The advantage is that a hashed local weight file can be pinned exactly and does not change between audits, whereas a hosted model can — but that advantage only counts if the pinning and benchmark provenance are captured in the pack. How do you make an Ollama benchmark reproducible across sites so the evidence travels? Capture the full minimum set — model hash, quantisation, hardware and stack, prompt-set provenance, distributions, and versions — so the run is defined by the record rather than by the machine it happened on. When those conditions travel with the numbers, another site can re-run and confirm rather than re-validate from scratch, which is what lets the evidence move between audit cycles instead of being regenerated at each one. Before your next deployment sign-off, run one test: hand the benchmark to someone who wasn’t in the room and ask them to tell you which model, at which quantisation, on which hardware, was measured against which task — from the record alone. If they can’t, you’ve documented a performance anecdote, and the failure class is a validation gap that SVC-VALIDATION exists to close before an auditor finds it for you.
