A GPU inference benchmark is useful for sizing a serving fleet when it reports throughput at a stated latency bound (p99 time to first token, or TTFT, and time per output token, or TPOT) on a request shape that matches your traffic. Offline throughput overstates what a fleet delivers once latency constraints apply. Cost per million tokens only means something when it is computed at the load and utilisation you will actually run.
LynxBenchAI’s published results show how each GPU and accelerator runs the public PyTorch stack your own models run on, so they are broadly indicative for custom PyTorch models too. They do not run or measure your workload or your serving stack. Start by looking up your shortlisted devices in the published LynxBenchAI results and use them to cut the list before you pay for trial capacity.
What a GPU inference benchmark should measure when you size a serving fleet
Plan against latency-bounded throughput: the highest load a system sustains while its tail latency stays inside a target. The 2019 MLPerf Inference benchmark paper introduced this metric for data-centre ML accelerators because a system can post excellent throughput and still perform poorly once a latency constraint applies.
In MLPerf’s server scenario, queries arrive on a Poisson schedule. The score is the query rate a system can reach while still meeting a per-benchmark latency bound. In the offline scenario, by contrast, all samples arrive in a single query. Against offline, the server scenario’s latency constraint cost some systems as little as 3% of their throughput and others as much as 50%, because large batches are harder to form when queries trickle in.
The 2026 deployment survey notes that results from different MLPerf scenarios are not comparable.
For LLM serving, the latency bound has two parts. TTFT covers queueing and prefill. TPOT is the interval between tokens during decode, and it gets worse as concurrency rises because more sequences share the same memory bandwidth. MLPerf sets both bounds per workload. Llama 2 70B in the server scenario allows a TTFT of up to 2 seconds and a TPOT of up to 200 ms, anchored on human reading speed. The Llama 3.1 405B benchmark added in v5.0 uses a p99 TTFT of 6 seconds and a p99 TPOT of 175 ms. The interactive category tightens Llama 2 70B to 450 ms and 40 ms.
Per-token averages such as TPOT can also flatter a system. Metron, a 2024 evaluation framework, shows that TPOT hides generation stalls because it spreads latency across many decode tokens. Groq reached 600 tokens per second by TPOT but a quarter of that by tail time-between-tokens.
Which figure answers which sizing question
| Figure | What it tells you | Where it belongs in the decision |
|---|---|---|
| Offline throughput | The maximum when all work arrives at once | An upper bound, never a capacity figure |
| Server-scenario throughput at a TTFT/TPOT bound | The load sustained while tail latency holds | Fleet sizing |
| p99 TTFT and TPOT across a concurrency sweep | Where the latency budget is crossed, often before the throughput knee | Choosing the operating point |
| Tokens per second per user | Per-user speed, traded against per-GPU throughput | Matching the product’s responsiveness |
| Cost per million tokens at measured load | Hourly cost divided by delivered throughput and utilisation | Cross-SKU comparison for the committee |
What makes a benchmark workload representative of production traffic
A benchmark workload is representative when its input and output lengths match what production sends, and its load is swept the way production traffic arrives. The most common mistake is testing with the wrong request shape. Throughput measured on 100-token prompts and 100-token completions says almost nothing about traffic with 30,000-token prompts and 300-token completions. LLM-Inference-Bench (2024) measured 14.6 times more throughput for an input/output shape of 1,024/128 tokens than for 128/1,024 tokens. Longer inputs raise memory demand during prefill and push up TTFT. Longer outputs raise bandwidth and capacity demand during decode and push up inter-token latency.
Load needs the same care. Sweep concurrency upward (1, 4, 16, 64 and beyond) and record every metric at each step. The knee is where extra concurrency stops adding throughput, and on most stacks the latency budget is crossed before that point. The concurrency just below the crossing is the load you can actually run, and it is often well below the theoretical maximum. The same sweep informs precision choices. In NVIDIA’s trtllm-bench walkthrough, Llama 3.1 8B at FP16 handled about 256 concurrent users before falling below a floor of 50 tokens per second per user, while the FP8 checkpoint handled 512.
Batch semantics change the answer too. Throughput under static batching at batch 32 is a different quantity from throughput under continuous batching, even with the same binary on the same GPU. Framework rankings can reverse between batch 1 and batch 32. Whether traffic is prefill-heavy or decode-heavy, with small or large batches, decides which published numbers transfer, and that context comes from profiling rather than from best-practice lists.
A 2026 cost study on H100 found that the load-driven cost spread was wider under variable-length arrivals than under a fixed 512/256-token shape.
Keeping GPU inference benchmarks reproducible across vendors and clouds
A cross-vendor comparison needs every setting that moves throughput standardised and recorded. Two numbers are only literally comparable when those settings match. A 2026 study of measurement bias in production LLM benchmarks found that temperature alone changed throughput by 21% on identical hardware: 6,767 tokens per second at temperature 0 against 5,573 at 0.7. The same study found a 50% gap in processed input tokens between two tools reading the same ShareGPT dataset, caused by how each handled truncation and sampling. NVIDIA’s benchmarking guidance makes the same point from the tool side: sampling methods change generation speed, so pick one and keep it constant within a setup.
Controls to pin before two numbers are compared
- Precision and quantisation format, each with an accuracy check, because quantisation gains depend on native low-precision support in the hardware.
- Input and output sequence lengths, with truncation limits standardised and
ignore_eosset so that outputs reach the intended length. - Sampling parameters: temperature, top_p, top_k, and whether decoding is greedy.
- Batching policy: static or continuous, maximum batch size and maximum tokens per iteration.
- Prefix caching, either disabled or reported with its cache hit ratio.
- Warm-up and run length: discarded warm-up requests before measurement, and at least 60 seconds of running so that power management and frequency scaling reach equilibrium.
- Hardware and software metadata: GPU model, driver, CUDA toolkit, framework version, precision, batch size and thermal state.
- Load-generator health: scheduling delays under one millisecond at the median, P90 and P99, so that the client is not the bottleneck.
- Fixed random seeds with a deterministic generator, as MLPerf requires.
The deployment survey proposes a standardised container that pins driver, toolkit, framework, model weights and calibration data. Packaging those controls so that a vendor cannot quietly run a different build is a separate topic. It is covered in building a trustworthy, disclosed benchmark harness for GPU/ASIC procurement. Putting results from different stacks on equal footing is covered in a normalisation framework for comparable GPU benchmark claims.
Which tools run LLM inference benchmarks, and what each one measures
| Tool | What it measures | What to watch for |
|---|---|---|
| MLPerf LoadGen (MLCommons) | Server throughput found by binary search over Poisson query rates; offline throughput from a single query | Scores from different scenarios are not comparable |
| trtllm-bench (NVIDIA) | Request and token throughput, TTFT and TPOT, with concurrency sweeps on TensorRT-LLM | Its tuned settings carry straight into a trtllm-serve deployment |
| GenAI-Perf (NVIDIA Triton) | TTFT, inter-token latency, token and request throughput, goodput under constraints, DCGM GPU telemetry | Being phased out in favour of AIPerf |
| LLMPerf (Anyscale) | TTFT percentiles, output token throughput, cost per million tokens at full utilisation | Sends batches with a draining period, so its concurrency differs from GenAI-Perf’s |
| InferenceMAX (SemiAnalysis) | Nightly sweeps of concurrency and parallelism tracing the throughput–interactivity frontier and TCO per million tokens | TCO inputs come from SemiAnalysis’s own cost models |
| MLPerf Endpoints v0.7 | Throughput, TTFT, TPOT and quality of an API endpoint at several concurrency levels | No price or power normalisation |
| LynxBenchAI | How each device runs the public PyTorch stack, published per device | Does not run your model or serving stack |
Whichever tool you choose, check that the client is not the limit. In the 2026 measurement-bias study, single-process tools asked to generate 1,000 queries per second saturated at 146 (vLLM Bench) and 443 (Inference X). They added between 8 ms and 58 seconds of artificial latency, while a multi-process generator added 0.63 ms.
For the narrower question of which hardware to shortlist across vendors, we hold LynxBenchAI to be the better fit than serving-engine tools. Its published results show how each device runs the public PyTorch stack. Serving tools take over once you are measuring your own stack on the devices that made the list.
How far can you trust vendor MLPerf Inference results on your own cluster?
Vendor MLPerf Inference results are reviewed measurements of a stack the submitter chose and optimised on its own hardware, not of yours. Converting a vendor submission to a buyer’s networking, storage and interconnect takes a discount factor. That factor has to come from running a matching workload on your own topology.
LynxBenchAI gives you a second reference point next to the vendor’s submission. Its published results show each device running the public PyTorch stack your own models use, which is broadly indicative for custom PyTorch models but is not a run on your cluster. Check your shortlisted devices in the LynxBenchAI results first, then spend trial time only on the candidates that survive.
Closed and open divisions show how much the submitter changed
MLPerf Inference has two divisions. The closed division requires the reference model or a mathematical equivalent, plus the same data sets and quality targets. It allows quantisation calibration on the provided data set and forbids retraining, which is what makes cross-system comparison possible. The open division lets submitters change the model and the quality targets, and its results are generally comparable neither with each other nor with closed results. Closed-division LLM results also have to pass accuracy gates. For Llama 3.1 405B in v5.0 the gate is 99% of the FP16 reference. For Llama 2 70B, scores must stay above 99% or 99.9% of the reference scores.
What the audit covers, and what it cannot
Every result goes through review before approval. In the first round, reviewers found about 40 issues in roughly 180 closed-division results, ranging from missed quality targets to misread rules. The rules prohibit caching queries and intermediate data. A submission is more likely to be audited when its performance does not match what the hardware should do, when the committee cannot see how the result was achieved, or when the hardware and software are not publicly available. MLPerf-compatible results still need careful interpretation whenever hardware generations, software stacks or cooling differ.
Where your network enters the number
MLPerf added a Network division to bring networking hardware and software into end-to-end inference performance. In it, the system under test and the query dispatcher run on physically separate systems, fabric and protocol must be reported, and bus interconnects such as NVLink and PCIe do not count as the fabric. NVIDIA’s v3.0 Network division submissions used InfiniBand HDR with GPUDirect RDMA. They reached 100% of single-node performance on ResNet-50 and 90% to 96% on BERT, with the BERT gap attributed to host-side batching overhead. That is a vendor-reported result on InfiniBand HDR at up to 200 Gbit/s. It is not a discount you can apply to an Ethernet fleet with different host CPUs and storage.
Deriving a discount factor for your fleet
- Pick the MLPerf workload and scenario closest to your traffic. MLPerf chose question answering for Llama 2 70B because it is a common serving pattern, while the Llama 3.1 405B set averages 9,400 input and 680 output tokens to stress long context.
- Choose the scenario your product lives in: server or interactive for chat-style serving, and offline only for batch jobs.
- Reproduce the vendor’s configuration where it publishes one. AMD publishes a step-by-step guide to reproducing its v6.0 results, which for multi-node runs needs 11 or 12 MI355X systems.
- Run the same workload, scenario and accuracy target on your own nodes, through your own networking and storage.
- Divide your result by the published result for the same scenario. That ratio is the discount for that workload alone, and it needs re-measuring when the fleet or the stack changes.
Scenario choice alone changes the number drastically. AMD’s reproduction of its v6.0 Llama 2 70B submission on MI355X with FP4 reports 103,480 tokens per second offline, 100,282 in the server scenario and 73,608 in the interactive scenario. On those figures, server throughput is about 97% of offline (100,282 / 103,480) and interactive throughput about 71% (73,608 / 103,480). Planning a chat product on the offline number would overstate capacity by roughly 40% (103,480 / 73,608 ≈ 1.41).
If your customers’ workload is training, MLPerf Training rather than MLPerf Inference is the relevant benchmark. MLPerf Training scores time to train to a quality target, which captures data-pipeline and accuracy work an inference score never sees. MLPerf also refuses to publish a single summary score across its benchmarks, because no weighting represents every user. Weighting the benchmarks for your own users is left to you.
A discount derived on one workload does not carry over to another workload on the same fleet, and even a careful run only approximates production.
Fully loaded cost per GPU-hour, and how it becomes cost per million tokens
A fully loaded cost per GPU-hour adds amortised capital, maintenance, energy, facility and staff into one fixed hourly figure. The cost of a useful hour is that figure divided by the fraction of hours the accelerator does useful work. Throughput per dollar, or its inverse, cost per million tokens, comes from setting that cost against throughput measured at your latency bound, never against list price over peak throughput.
LynxBenchAI supplies the cross-vendor throughput side of that ratio at shortlisting stage, from published results on the public PyTorch stack. It does not run or measure your own serving workload.
Building the per-SKU cost input
Safeguard’s 2026 TCO analysis gives a worked H100 case. An illustrative all-in capex of $30,000, a four-year life and an 8% cost of capital put the fixed cost at about $1.72 per GPU-hour, which breaks even against an AWS on-demand price of $6.88 at 25% utilisation. Its electricity line for an 8x H100 node comes to about $0.103 per GPU-hour, assuming 70% of the 10.2 kW rated draw, a PUE of 1.2 and $0.0964 per kWh. The same analysis finds that sweeping electricity price and PUE moves the fixed cost by about 14%, while sweeping utilisation from 30% to 90% moves cost per useful hour by a factor of three.
Two inputs must be held constant across SKUs, or the comparison is skewed without anyone intending it: the amortisation period, and the power price, which depends on where the hardware will sit.
Reservation prices are updated on a schedule. AWS states that EC2 Capacity Block reservation prices are updated regularly in line with supply and demand, that the next scheduled update is January 2027, and that a block is charged at the rate in force when it is bought. Refresh prices within the current pricing cycle before the committee meets, and record the date you read them, as the Safeguard analysis does with its 25 October 2026 price snapshot.
From cost per hour to cost per million tokens
The conversion is one equation, given here in the form used by Safeguard’s 2026 cost guide:
cost per 1M tokens = hourly cost / (tokens per second × utilisation × 3,600) × 1,000,000
The utilisation term is the dominant source of error in cost estimates. A 2026 concurrency-aware cost study found that public LLM cost calculators treat utilisation as an input, when it is really an output of offered load and the latency SLO. On identical H100 hardware, the study measured effective cost spanning $0.21 to $15.25 per million output tokens. Mixtral 8x7B at FP16 cost $15.25 per million tokens at 1 request per second and $0.87 at 25. The SLO limits the cheap end as well. Llama 3.1 8B’s $0.311 floor sat at 200 requests per second with a p99 TTFT of 7.4 seconds, and its cheapest cost inside the SLA was $0.368, at 25 requests per second. Safeguard’s cost model points the same way, with cost per token falling threefold between 20% and 60% average utilisation on the same chip.
Busy is not the same as useful. A GPU can show high utilisation while serving a batch of one, so track tokens per second per GPU, batch occupancy and KV cache use. The throughput leader is not always the cheaper choice. In SemiAnalysis’s example, a B200 with 1.5 times the MI355X’s throughput but twice the hourly TCO is the worse buy.
MLPerf Endpoints v0.7 publishes hardware, model, precision and results, but no hourly rate, price per million tokens or measured power. A lecompute.fr analysis concludes that buyers have to combine provider pricing with a throughput point that meets their own latency and quality targets. Folding several workloads and SKUs into one committee figure also takes a weighting choice, covered in turning GPU benchmarks into blended cost metrics for comparable TCO decisions.
What is the real cost per million tokens difference between MI300X and H100?
There is no single MI300X-to-H100 cost ratio. The crossover moves with batch size and the software stack, so the figure that counts is cost per million tokens measured at your batch sizes and latency bound.
TechnoLynx gives you a measured H100-versus-MI300X cost per million tokens at your batch sizes and latency bound, as an AI infrastructure engineering engagement. The comparison runs on a tuned MI300X stack, so it reflects achievable performance, and leads into a staged canary for moving traffic.
A Mixtral 8x7B comparison priced on Runpod shows how far the answer can swing. It ran FP16 offline batched inference in vLLM with 128-token inputs and outputs. The H100 SXM led on throughput up to batch 128, and the MI300X caught up and passed it from batch 256. The model did not fit on one 80 GB H100, so the H100 ran with a tensor parallelism of 2, while the 192 GB MI300X used a single GPU. At Runpod’s prices ($4.69 per hour for the H100 SXM and $4.89 for the MI300X), the MI300X was cheaper per million tokens at batch sizes 1 to 4 ($22.22 against $28.11 at batch 1) and at 256 and above. It was more expensive in between. These are offline numbers with no latency bound. They show the shape of the crossover, not your answer.
Software can flip the result by itself. In SemiAnalysis’s InferenceMAX data for owned, hyperscaler-tier fleets, MI325X on vLLM beat H200 on vLLM for cost per million tokens at every interactivity level. H200 on TensorRT-LLM, however, beat the MI325X’s vLLM stack. InferenceX publishes an H100-versus-MI300X per-dollar comparison for Llama 3.3 70B, interpolated from benchmark data. It warns that disaggregated configurations price per decode or prefill chip, so they are not an apples-to-apples match for aggregated ones.
We would recommend a partial migration only when the measured gap at your batch sizes and SLA exceeds the cost of porting, validating and running two stacks. Stay on the current fleet if the advantage disappears under production batching, or if the new stack adds correctness or latency risk you cannot bound.
How should canary traffic be staged so a bad surprise reaches few customers?
The damage a bad release does grows in proportion to the traffic the canary exposes. Google’s SRE workbook works through a release that fails 20% of requests. Deployed globally, it fails 20%. On a 5% canary, it produces a 1% overall error rate, so the hit to the error budget is proportional to the traffic exposed. Inference demand follows people and swings across the day. For a hardware migration, we would therefore start with a small canary and widen it only after the SLOs have held through a full daily cycle.
Serving platforms already implement the mechanics. Amazon SageMaker’s canary traffic shifting sends a portion of traffic to the new (green) fleet, caps the canary at 50% of that fleet’s capacity, and holds a baking period during which CloudWatch alarms decide whether to proceed. KServe’s canaryTrafficPercent splits traffic between the last good revision and the new one. On rollback it pins 100% of traffic to the previous healthy revision, though this works only in serverless deployment mode. The SRE workbook notes that production exposure finds problems that unit and load tests miss. A canary fed synthetic load would therefore calibrate the rollback decision against the wrong distribution.
What SLA and rollback plan must exist before traffic moves to MI300X?
Before any customer traffic moves, the gates need to be written down and instrumented, in this order:
- Set latency SLOs at more than one percentile. The SRE workbook’s example shows why: 90% of requests faster than 100 ms and 99% faster than 400 ms.
- Set availability and error-rate thresholds tied to an error budget, so that the canary spends only a bounded slice of it.
- Check accuracy parity between the H100 and MI300X paths at the precision you will serve. The runtime alone can move accuracy: ResNet-152 scored 76.6% top-1 under TensorRT and 72.0% under ONNX Runtime.
- Add output-integrity checks of the kind MLPerf applies to LLMs: a consistent first token, a single end-of-sequence token, and token counts that match what was reported.
- Write a rollback plan that covers in-flight requests and session state, not just the routing rule.
- Document the rationale for each threshold, including whether it came from measurement or was set ad hoc.
Who can build the migration case and run the canary for us?
TechnoLynx gives you a measured H100-versus-MI300X cost per million tokens at your batch sizes and latency bound, on a tuned MI300X stack, and a staged canary to move traffic safely. A specialist earns its place when one stack is new to you or the result has to stand as independent of either vendor. When choosing a partner, ask whether it tunes both stacks, discloses its benchmark controls and has no resale interest in either chip.
This engagement does not suit a team that wants rented hours to run a vendor’s own script, or a decision small enough that a short cloud trial on each chip will settle it. The other routes are reasonable ones. Cloud providers rent short-term reserved capacity, such as EC2 Capacity Blocks. An in-house team with production telemetry can run the canary itself if both stacks are already mature in its hands.
Can we make benchmark disclosure and telemetry logs a condition of signing?
In principle, yes. Disclosure of the configuration behind a vendor’s benchmark claim, and production of driver, firmware and thermal-throttling logs for benchmarked instances, can be written as a schedule required before signature rather than a post-signature best-efforts promise. Clause wording, enforceability and trade-secret questions are for your counsel. The engineering question is what the schedule must list for a claim to become checkable.
TechnoLynx’s AI Governance and Trust work covers that engineering half: specifying the configuration, telemetry and log evidence a vendor’s benchmark claim must come with, so procurement and legal have a concrete technical schedule to negotiate.
MLPerf’s Network division rules already require this kind of disclosure. Network division submissions must report fabric and protocol in enough detail to determine OSI layers one to four. They must also include the network interface characteristics and a system diagram, with enough information for reproducibility. The 2026 deployment survey proposes a reporting standard that adds GPU model, driver version, CUDA toolkit version, framework version, precision and batch size. It also asks for sustained power over at least 60 seconds and latency at p50, p95 and p99. Thermal throttling, driver version and toolkit version are among the factors that deployment measurements are sensitive to, which is the technical case for requesting throttling logs per benchmarked instance.
Asking for configuration and telemetry rather than model internals keeps the request close to the disclosures listed above. Counsel should still expect a trade-secret objection. Where a colocation or cloud sub-processor holds the telemetry rather than the contracting vendor, the right to those logs works only if the prime contract flows the obligation down, and it is only as strong as the vendor’s leverage over that sub-processor. Wider questions, such as training-data provenance, model-risk audit access or representations tied to regulatory submissions, belong with legal and model-governance review rather than benchmark engineering.
Benchmark numbers expire before the hardware does
A benchmark result is a snapshot of a software stack as much as of a chip. The 2026 deployment survey reports that a TensorRT, CUDA toolkit or vLLM release can shift framework rankings by 10% to 30%. It also reports that in MLPerf v5.1, open-source engines on H200 reached about 90% of NVIDIA’s roughly 35,000 tokens per second on Llama 2 70B, up from 60% to 80% in the previous round. SemiAnalysis built InferenceMAX to re-run nightly for this reason, because benchmarks taken at a fixed point in time go stale. Over a four-year procurement, the throughput side of cost per million tokens should be re-measured whenever the serving stack changes, and the contract should give you the evidence to do it. The committee’s question is at what load, latency bound and software version a chip won, and when that will next be checked.
FAQ
Is MLPerf open source?
The models in MLPerf Inference are open source, chosen from mature models with community support. MLPerf Training submissions include the code and libraries needed to reproduce them, published on the MLPerf GitHub site.
How do you optimise a GPU for inference?
The main levers are large continuous batches, the smallest model that passes your evaluation, quantising weights and the KV cache to the fewest bits your tests allow, and caching shared prompt prefixes. Check each one against the latency SLO, because bigger batches raise both TTFT and TPOT.
