A GPU benchmark measures the hardware only when everything else is held still. To get there, pin and disclose five things: the software stack and its versions, the inference backend and its defaults, the precision, the batch size and concurrency, and the clock and thermal state. Then measure under sustained load after a discarded warm-up, repeat the run, and call a gap between two GPUs real only when it is clearly larger than the spread you measured across the repeats. Each of those is a separate control rather than one setting. That is why a GPU benchmark tool that reports a single number without its configuration tells you less than it seems to.

You can see score differences between platforms inside a single published release. In the 26Q3 results, the GeForce RTX 5090 on native Linux scores 985.8 for inference and the same card model under WSL scores 891.8. The RTX 3080’s compute score runs the opposite way to its own inference score: 64.8 on native Linux against 72.5 under WSL. The card model is the same in each pair; only the platform differs. Each is a single-window 26Q3 score with no spread, so these gaps are not shown to exceed run noise or to come from the platform.

Release gating and refusal ablation borrow the same discipline but are separate problems.

How do I log prefill and decode separately in llama.cpp or Ollama?

Report prompt processing (prefill) and token generation (decode) as two throughput figures. The two phases load the GPU differently: input tokens are processed in parallel, while output tokens are generated one after another.

LynxBenchAI benchmarks AI performance on your own hardware with stock PyTorch, which gives a hardware baseline to set your prefill and decode figures against; the first step is to run it on each card from the LynxBenchAI page.

TechnoLynx can build this as a measurement protocol for you. The work starts by defining prefill and decode as separate phases and choosing the prompt-length and generation-length matrix that matches your workloads. From there it covers configuring the instrumentation, running the harness on your hardware, checking the results for confounds, and handing over the protocol so you can repeat it on future cards. The published LynxBenchAI results and Personal Edition give a hardware baseline measured with stock PyTorch, and a protocol like this one adds the llama.cpp or Ollama figures.

In llama.cpp, the llama-bench tool records the prompt, generation and depth sizes of each test (n_prompt, n_gen and n_depth). It repeats each test five times by default and reports average tokens per second with a standard deviation. Its timings leave out tokenisation and sampling. Some output formats, JSON for example, keep every individual repetition, and those per-run values are what you need for a noise estimate later.

Ollama’s API responses report prompt and output token counts and durations separately. Each response carries prompt_eval_count and prompt_eval_duration for the uncached prompt tokens, eval_count and eval_duration for the generated tokens, and prompt_eval_cached_count for prompt tokens read from cache, with every timing in nanoseconds. Dividing each count by its duration gives a prefill rate and a decode rate. The cached count matters when prompts repeat. In one published comparison where every request carried the same prompt, llama.cpp reused the cached prefix and evaluated only a few new tokens after the first request, which flattered its time to first token compared with real traffic using distinct prompts.

Which prompt and generation lengths make the split matter

One fixed length hides the split, so test a grid. LLM-Inference-Bench (2024) used input and output lengths of 128, 256, 512, 1024 and 2048 tokens with batch sizes of 1, 16, 32 and 64. That study found that throughput rises as output length falls at a fixed input, and drops as input length falls at a fixed output. Batch size moves the numbers too: on an A100 at a token length of 2048, the same study measured 26.6 times the throughput at batch 64 compared with batch 1.

Is there a standard way other reviewers report prefill against decode? We know of no single cross-reviewer convention for that split. Comparability across channels therefore comes from publishing the grid, the tool and its version alongside the numbers, not from a shared format. The MLPerf Inference rules (MLCommons) are the closest discipline to borrow: for “Available” submissions they require the software version of every component, the hardware configuration, the software stack, the containers and all settings.

What test sequence should run between card swaps on one shared bench?

The answer is a fixed sequence run the same way for every card. We would run it in this order:

1. Set the driver state the same way each time. On Linux, nvidia-smi can set persistence mode, which resets to disabled after every reboot.
2. Decide whether to lock clocks. nvidia-smi can lock GPU and memory clocks on Volta and newer cards. With auto boost on, the GPU boosts above its floor whenever power, thermal and utilisation headroom allow. Locked clocks isolate the silicon. Unlocked clocks show what the card actually holds, so record which one you chose.
3. Start telemetry before the first request. By default, nvidia-smi dmon logs power, temperature, SM and memory clocks, and utilisation once per cycle. Where the card supports it, the T.Limit reading reports the margin to maximum operating temperature, and at 0 °C or below the GPU may optimise its clock for thermal conditions.
4. Discard the warm-up. NVIDIA’s GenAI-perf uses a sliding window so that “warming up” and “cooling down” requests are left out of its metrics.
5. Leave a cooldown between tests. llama-bench has a delay setting between tests for this purpose.
6. Repeat and keep every run.

This order follows the sustained-load position LynxBenchAI holds: only performance a machine can hold under heavy, realistic load is operationally relevant. LynxBenchAI therefore discards a warm-up phase, then counts completed iterations inside a continuous timed window at a workload size already scaled until throughput stops improving.

Can telemetry track clocks over a long run? nvidia-smi’s loop mode, with the CLOCK display selected, reports clock readings every five seconds by default.

Normalising across days with different ambient temperatures is the weakest step. We have no source for a validated method, and any correction is approximate. A controlled-environment bench beats it. Our judgement is to log ambient and thermal headroom with every run, and to re-run or flag runs that start with less headroom than the others rather than scaling their scores. Short loaner windows also cap how many repeats you get, so the confidence you can claim is lower.

Should we disable TF32, fix seeds and pin cuDNN before comparing two GPUs?

Yes, when the goal is to tell how much of a gap comes from the hardware and how much from the software version. Turn the determinism controls on identically on both sides, then measure how much variance is left before you read the main difference. TechnoLynx can build that protocol for you: fixed seeds, non-deterministic kernels off, cuDNN algorithm selection pinned, backend, context length and batch specified, repeated trials to establish the noise floor, and a report with the full configuration disclosed. The first step is to agree which of two questions the comparison answers: separating hardware from software effects, or comparing GPUs for one deployment.

LynxBenchAI benchmarks AI performance on your own hardware with stock PyTorch, which gives both GPUs a common hardware baseline; the first step is to run it on each card from the LynxBenchAI page.

Deterministic configuration that isolates hardware from software version

NVIDIA’s TAO Toolkit documentation says that cuDNN convolution algorithms and fused attention backward passes are non-deterministic by default, so results can differ between runs even with a fixed seed. It also says to keep cuDNN benchmark mode off when you need reproducibility, because benchmarking auto-selects algorithms that can vary between runs. Bouthillier et al. (2021) found that their convolutional models did not reproduce until they enabled cudnn.deterministic and disabled cudnn.benchmark.

Pinning matters across architectures as well as across runs. In a 2026 study on A100, L40S and H100, kernels on TF32 tensor cores produced none of nine test cases bitwise identical across the three architectures: A100 and L40S agreed with each other and both disagreed with H100. The same study showed that a performance-first rule sizing GEMM splits by SM count picks a different split on each GPU (the three have 108, 142 and 114 SMs), so the numerical output differs on four of six shapes. Without the pin, part of the gap belongs to the library’s choices, not the card.

There are two limits to know. First, not every operation has a deterministic kernel, so the controls reduce non-determinism without removing it everywhere. Second, NVIDIA states that reproducibility holds per GPU architecture, library version and GPU count. Deterministic paths are also slower: NVIDIA notes that deterministic kernels are generally slower than auto-tuned or fused ones. Thinking Machines Lab reported about a 20% loss against cuBLAS for its batch-invariant matmul. A deterministic run is the right tool for attribution and the wrong one for quoting deployable throughput. We would publish both, clearly labelled.

How much variance comes from non-determinism alone, and when is a 30% gap real?

We know of no universal figure, so measure your own. The published figures below measure different things. A 2018 study of deterministic implementations for deep reinforcement learning found that GPU non-determinism in deep Q-learning compounded to a 12.41% relative standard deviation at the end of training. That is error compounding over thousands of training steps, not single-pass inference timing. In a bench comparison by Computing for Geeks, two RTX 4090 boxes on different driver branches decoded within three percent of each other (39.0 and 39.9 tokens per second). Sinha et al. (SC22) collected over 18,800 hours of data across more than 90% of the GPUs in five large clusters to characterise performance variation between nominally identical GPUs. Their title states the premise: not all GPUs are created equal.

For a throughput gap, the noise floor that matters is the spread of the same card on the same configuration across repeats. Run card A against itself first. A 30% gap measured against a 3% spread is a different claim from a 30% gap measured against a 20% spread.

For the decision itself, a gap in averages is not enough. Bouthillier et al. (2021) show that a z-test needs a difference of at least z·√((σA² + σB²)/k) before it controls false detections. They propose calling A better than B only when the probability that A beats B in a single run reaches 0.75, which needed 29 runs for a 5% false-negative rate in their setting. Miller (2024), writing on language-model evals, adds that scores need standard errors, and that sampling temperature should not be lowered just to reduce variance. With five repeats instead of 29, state the spread and claim less.

The 26Q3 LynxBenchAI results are a single timed window per test after a discarded warm-up, not a median of repeats, and scores are comparable only within that release. They compare devices on one suite within that release. They do not give a per-device noise band.

Fix one backend or benchmark both llama.cpp and vLLM?

Decide first whether the backend is a controlled variable or part of the question. If you want to know which GPU is faster, fix one backend at one version. If you want to know which stack to deploy, run both and report them separately, never as one blended chart. The backend is not a neutral wrapper. A 2026 study holding weights, decoding and hardware constant found up to a 16.6 percentage-point accuracy spread between backends for DeepSeek R1 Distill Qwen 7B on GSM8K. Its ablations traced Ollama’s largest gaps to its hidden BOS token and default repetition penalty of 1.1. Its authors recommend reporting the engine and its configuration as a first-class experimental parameter.

The backend version matters too. Computing for Geeks found that Ollama 0.35.1’s bundled llama.cpp build decoded at about 20 tokens per second on an L40S, against 39.1 for upstream llama.cpp with identical flags, file and GPU. A comparison that says “Ollama” without the build number is not reproducible.

Does continuous batching change which GPU looks faster? It can. vLLM and TensorRT-LLM use continuous batching to process requests that arrive at different times and have different lengths. Thinking Machines Lab explains that server load sets the batch size kernels run under, which changes each request’s result. On throughput, LLM-Inference-Bench measured LLaMA-3-70B on H100 improving 39 times from batch 1 to 64, against 3 times on A100. Because batch scaling differs this much by GPU, the gap at a single stream can differ sharply from the gap under concurrency, so the gap you report depends on the concurrency you chose.

The same applies to context length and batch for a 70B chat model: there is no universal “typical” setting. Pick single-stream low-latency or multi-user serving according to the deployment you are buying for, state it, and accept that the result applies to that model, quantisation and pattern only.

Disclosure checklist for a controlled GPU comparison

This table is the minimum configuration disclosure we would publish with any comparison.

Variable	How to pin it	What to report
Driver, CUDA, PyTorch, library versions	One container image per run set	Every version string; GPU model and CUDA driver both changed results in Bouthillier et al.
Backend and build	One engine at one build, or each reported separately	Engine, version, prefix caching, repetition penalty, template handling
Precision	Same format on both cards	Format used; for one reasoning model on AIME’24, BF16 showed up to 9% accuracy standard deviation across GPU type, count and batch, FP32 near zero
Determinism controls	Seeds fixed, cuDNN benchmark off, deterministic flags on for attribution runs	Which flags, and whether the throughput run used them
Batch, concurrency, context	Fixed grid matching the deployment	Grid values and whether continuous batching was active
Prompt and decoding	Fixed prompt set, temperature 0 for timing	Prompt lengths, output cap, cache hits
Clocks and thermals	Locked or explicitly unlocked; warm-up discarded; cooldown between tests	Clock policy and the telemetry log
Noise	Repeated runs per card	Mean, spread and number of runs

Hoefler and Belli (SC15) propose a minimal set of twelve rules for reporting parallel performance results so that they can be interpreted. MLCommons submission rules ask for a system description that combines hardware and software stack information, together with run logs. For comparisons across vendors, see our piece on building reproducible cross-vendor benchmark disclosures.

Release gates and refusal ablation need their own controls

Gating a model’s release on a harness score, or rejecting an ablated model on a harm threshold, is a different problem from a GPU comparison. The isolating discipline still applies to the score the gate reads. Backend choice alone shifted the accuracy of DeepSeek R1 Distill Qwen 7B on GSM8K by up to 16.6 points in the 2026 backend study, and in a numerical-precision study, BF16 inference varied by up to 9% in accuracy for one reasoning model across GPU type, count and batch size. A threshold set on one stack can therefore pass or fail on another stack with no change to the model.

A go/no-go gate before technicians use the assistant

The score bar itself belongs to whoever owns the operational risk. A benchmark cannot set it. What a benchmark discipline adds is a pinned harness version, a pinned inference configuration, a standard error on the score, and a re-run whenever the model, prompts or serving stack change. A limited pilot after a passing score is still sensible, because a harness samples the floor’s conditions rather than reproducing them.

Rejecting an ablated model on a harm threshold

Which harm taxonomy, which threshold and who signs off are governance decisions for the approving body. Which method computes a refusal direction for a given Llama checkpoint, and whether it carries over to another checkpoint, is model-internals work. We do not settle either here. The one point we would hold to is the same as for hardware: measure the harmful-output rate under a disclosed, fixed configuration, with its error bar, before treating a crossing of the threshold as real.

Who can set up a controlled GPU comparison for us?

TechnoLynx designs and runs controlled hardware comparisons that separate the GPU’s contribution from the stack’s. In order, that means: framing which variable you are isolating, writing a protocol that pins seeds, kernels, backend, precision, batch and clocks, running repeated trials to establish the noise floor, and reporting the result with its configuration disclosed. For a self-service baseline, LynxBenchAI benchmarks AI performance on your own hardware with stock PyTorch, from laptop CPUs to data-centre GPUs. Start from the LynxBenchAI page.

We would not treat a measurement engagement as regulatory sign-off on a release gate. That stays with your own risk owner. If you only need one quick number for a single card, llama-bench with its default repetitions and a published spread may be enough.

Before choosing any of these routes, decide which question the comparison answers: the silicon under a pinned stack, or the stack you will actually deploy. Each one is a legitimate measurement, but they need different controls and lead to different decisions.

Sources

1. LLM-Inference-Bench: Inference Benchmarking of, arxiv.org
2. llama.cpp tools/llama-bench README, github.com
3. Usage – Ollama API documentation, docs.ollama.com
4. Ollama vs vLLM vs llama.cpp: Speed Benchmarked, computingforgeeks.com
5. MLPerf Inference Rules (MLCommons inference_policies), github.com
6. nvidia-smi documentation (NVIDIA), docs.nvidia.com
7. Metrics — NVIDIA NIM LLMs Benchmarking, docs.nvidia.com
8. Deterministic Training and Reproducibility — Tao Toolkit, docs.nvidia.com
9. Accounting for Variance in Machine Learning Benchmarks (Bouthillier et al.), arxiv.org
10. Accelerating the Mitigation of LLM Inference Nondeterminism Across GPU Architectures, arxiv.org
11. Defeating Nondeterminism in LLM Inference (Thinking Machines Lab), thinkingmachines.ai
12. Deterministic Implementations for Reproducibility in Deep Reinforcement Learning, arxiv.org
13. Not All GPUs Are Created Equal: Characterizing Variability in Large-Scale, Accelerator-Rich Systems (SC22), arxiv.org
14. Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations (Miller, 2024), arxiv.org
15. The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibility, arxiv.org
16. Understanding and Mitigating Numerical Sources of ..., arxiv.org
17. Hoefler & Belli, Scientific Benchmarking of Parallel Computing Systems: Twelve Ways to Tell the Masses (SC15), evaluate.inf.usi.ch
18. MLCommons Submission Rules, github.com