Justify a compute stack choice the way a reviewer will test it. Replace the vendor’s peak figure with a documented bundle of measured metrics (latency-bounded throughput, tail latency, energy per inference and cost per run), taken on workloads that resemble yours and scored with weights fixed before the first test. Then record the work so that someone with no stake in the outcome can check the result instead of trusting it. A datasheet number is not evidence of how your models will run, because the factors that dominate real AI performance (memory locality, on-chip data reuse, execution strategy and software behaviour) depend on the workload and cannot be read reliably from a specification.
The method stays the same for a single workstation GPU, a research pipeline that may later face GxP inspection, or an HPC capacity expansion. What grows with the stakes is the documentation the decision has to carry, from a short benchmark note up to qualification records and a weighted award matrix.
What should a benchmark report instead of a peak TOPS figure?
Cite four measured quantities, each tied to a stated accuracy or quality target: throughput achieved while a latency bound holds, latency at a stated percentile, energy per inference measured for the whole system, and cost per run or per million tokens. The latency and throughput measures follow the MLPerf Inference paper (2019), which built its comparisons on latency, latency-bounded throughput, throughput and inferences per query, all subject to a predefined accuracy target.
LynxBenchAI supplies the cross-device layer of that evidence. Its published results show how each device scores on training, inference and compute tests running the public PyTorch stack, so they are broadly indicative for custom PyTorch models, although they do not measure your application. When the note needs figures for your own models, TechnoLynx runs your representative workloads across the shortlisted accelerators, cross-checks the method against independent benchmark practice and delivers the metric bundle as a report you can cite. The starting point is the published LynxBenchAI results for the candidate devices.
The metric bundle and where each number comes from
| Metric | What it captures | How it is measured credibly | Why a spec sheet cannot stand in |
|---|---|---|---|
| Latency-bounded throughput | Work completed per second while a latency bound holds | MLPerf’s server scenario issues Poisson-distributed queries and reports the queries per second sustained within the quality-of-service bound | Adding the latency constraint cut throughput by as little as 3% for some systems and as much as 50% for others, relative to the unconstrained offline scenario |
| Tail latency | Response time at a stated percentile | Enough queries to bound the percentile statistically: MLPerf needs about 270K queries for a 99th-percentile guarantee on its multistream and server vision tasks | Percentile behaviour depends on batching, which varies with the use case |
| Energy per inference | Joules per sample for the whole system | Integrate measured full-system power over the active window, for at least 60 seconds | TDP and PSU ratings grossly overestimate actual draw, and ML.ENERGY (2025) found TDP-based estimates up to 4.1 times the measured energy |
| Cost per run or per million tokens | Money per unit of useful work | Derive the time to produce a million tokens from measured throughput, then multiply by the hourly price and the device count | A Hyperion Research, Intel and Ansys survey puts the initial purchase at only about half of lifetime cost |
| Quality at the tested precision | Whether the fast result is still correct | MLPerf requires almost all implementations to reach within 1% of the FP32 reference accuracy | Systems sometimes trade model quality for lower latency or cost |
Why a peak TOPS figure fails as evidence
A 2026 review of LLM accelerators puts it directly: standardised benchmarks such as MLPerf Inference exist because peak specifications do not predict delivered performance. The same review reads vendor TOPS figures as rough capability classes measured under differing conditions, not as comparable benchmarks. It also notes that vendor headline figures typically assume structured sparsity, so Blackwell’s quoted FP4 throughput is roughly double its dense figure.
Two physical mechanisms open a gap between nameplate and delivered throughput. The first is memory bandwidth. LLM decode is bandwidth-bound, so sustained bandwidth governs token rates more than peak floating-point throughput does, and even FlashAttention-2 reaches only about 35% utilisation on an H100, compared with 80 to 90% for a well-tuned matrix multiply. The second is heat. In a 2026 sustained-load study an iPhone 16 Pro lost 41.5% of its peak throughput over 20 iterations as it heated into a Hot state. Software maturity adds a third layer: the 2026 review cites independent benchmarking in which the AMD MI300X reached about 620 of its 1,307 marketed dense BF16 TFLOPS.
A single number also hides where each device is strong. In the LynxBenchAI 26Q3 results, the AMD Instinct MI300X scores 1023.3 on compute against 487.1 for the NVIDIA H100 80GB HBM3, while the H100 scores higher on inference, at 1678.1 against 1304.0. Those scores reflect one software stack at one point in time. LynxBenchAI measures across accelerators, backends and precisions, so the software stack is part of what any comparison measures. If you need to test a vendor’s number claim by claim, that step is covered in validating vendor and hardware claims before procurement.
Citing credible evidence in a research note
For a research note, cite an independent, cross-vendor benchmark report that publishes the whole bundle, not a vendor’s own chart. MLPerf Inference meets that standard on process. Its methodology was built with input from more than 30 organisations, and each submission carries a system description, benchmark code and LoadGen logs, which go to a public repository for peer review before release. The review does real work: it found about 40 issues in roughly 180 closed-division results.
Two caveats belong in the note next to the figures. Submitters may reimplement and optimise the reference models on their preferred toolchain, so the results show what a tuned stack achieves, not what yours will achieve. Coverage also has gaps. A 2026 edge study observes that MLPerf’s edge LLM scenarios measure peak performance without tracking thermal throttling under sustained load. SPEC’s power and performance methodology states the general rule: no single benchmark is a direct indicator of performance in every environment, and results support comparison, not planning, unless the workload and configuration match yours.
How should the metrics be weighted so the ranking can’t be called arbitrary?
Weight each metric by the share its workload holds in your actual use, write the formula down, and have it signed before any candidate is measured. Large public procurements already work this way. PRACE’s procurement white paper describes offerings evaluated quantitatively against pre-defined criteria, with the key criteria weighted and the final evaluation taken as the weighted sum. It recommends weighting benchmark results by their importance to the service’s workload, and designing suites flexible enough that marginal under-performance in one area can be offset in another. The NERSC-10 procurement evaluates system performance with its Workflow-SSI metric, computed from results on the NERSC-10 Benchmark Suite. Where every measured point matters equally, SPEC’s methodology sums performance across segments and divides by the summed power. That is still a weighting, but an explicit one.
No weighting is neutral. The MLPerf Inference paper makes the point with accuracy: giving up 1% accuracy for 50% lower total cost is prudent for identifying cat photos and risky for detecting pedestrians. Stating that value judgement before scoring turns a possible accusation of bias into a documented decision. The harness that produces the weighted numbers, from pilot design to fairness controls, is covered in evaluating compute candidates through benchmarks, pilots and fair comparisons.
For a capacity-expansion ranking matrix, measured sustained throughput on representative workloads should drive the ranking, with vendor peak TOPS kept only as a sanity bound. The precedent for fixing that method before vendors submit is PRACE’s insistence that benchmark conditions critical to acceptance be completely explicit, together with MLPerf’s closed division, where every submitter uses the same models and quality targets. TechnoLynx frames the workload mix and precisions with you, runs the sustained-throughput suite across the candidates and documents it for procurement, with the AI Readiness Scorecard as the decision step.
Which hidden costs belong in a total-cost comparison beyond the hardware price?
Price the whole life of the system, not the box. Support contracts, system integration, validation consulting, power and cooling, licences and staff time all belong in the comparison. A Hyperion Research, Intel and Ansys survey found that the initial purchase typically accounts for only about half of total expenses over a system’s useful life.
LynxBenchAI narrows the shortlist before any costed comparison: its published device results show how each candidate scores on training, inference and compute tests, and the first step is to check them for the devices you are pricing.
TechnoLynx benchmarks your actual workload on the shortlisted hardware, so the price-performance figures in the comparison are measured, not assumed. Before that run it agrees the workload characteristics, validation constraints and hard budget cap with you.
PRACE’s procurement guidance counts maintenance, system support, infrastructure such as cooling and uninterruptible power, and running costs (electricity in particular) alongside the system itself. The LBNL Energy Efficient HPC Working Group (2019) defines total cost of ownership over a specified period with a defined workload, and uses TCO per unit of workload performed as a selection factor. Mevasis’s HPC investment feasibility guide, a consultancy breakdown of a three-to-five-year total, puts hardware at 40 to 50%, licences at 15 to 25%, personnel at 15 to 20%, data-centre costs at 10 to 15% and maintenance contracts at 5 to 10%. Treat those shares as a checklist of categories, not a forecast for your site.
Smaller on-prem cluster or a cloud or HPC allocation?
The choice depends more on utilisation pattern and validation constraints than on list price. Lenovo’s TCO whitepaper sets out a usable method, though it is a vendor document and limits its analysis to compute hardware, power and cooling. First find the breakeven point where cumulative cloud spend matches the on-premises investment, then find the daily usage hours above which owning becomes cheaper. In its eight-GPU H100 example the threshold is about five hours a day against on-demand pricing, and discounted reserved pricing raises it.
Steady utilisation, combined with a validation regime that wants a fixed and auditable environment, points to a smaller dedicated cluster sized to the cap. Bursty use, or a pre-approved framework cloud allocation that your validation can accept, points the other way. Under a hard cap, the best-value configuration is the cheapest one that clears your acceptance thresholds on the measured bundle once integration, support and validation costs are added, so reserve headroom for those costs before you finalise the configuration. Two limits apply. A configuration that is best value for one workload type may not be for another, and a procurement framework may restrict which configurations you can actually buy. Two neighbouring questions are covered in costing and capacity-planning AI compute for finance and operations: turning query volume into a budget figure with a breakeven against self-hosting, and planning for capacity you can actually get at peak, including spot preemption.
Can published GPU reviews replace testing a card on your own workload?
No, but they do most of the shortlisting. Independent reviews are reliable enough for relative positioning. Before a non-recoverable purchase, though, either secure a return or trial window or run the shortlisted card on a scaled-down version of your own workload. Running real software workloads is the reference standard for judging an accelerator, and estimates from specifications are a fallback for when that is not possible.
LynxBenchAI sits between a review and your own test. LynxBenchAI also runs on your own hardware with stock PyTorch. That means a card in a trial window can be checked against its published device results before the window closes. It measures the public test set, not your application: it confirms that the card performs as published, not how your pipeline will run on it.
Reviews still earn their place in the shortlist. Tom’s Hardware, for example, says its Bench database tests a spread of titles representative of what players actually play and documents its benchmarking methodology. That kind of transparency is what to look for. The limit is just as clear: unless a review’s workload and configuration match yours exactly, its results can mislead as a forecast for your stack.
The physical card is a variable too. An SC22 study of five large GPU clusters found 8% average performance variation, and up to 22%, between GPUs of identical architecture and SKU, with outliers 1.5 times slower than the median. A 2026 cluster study measured up to 5% variation across nodes that were identical by specification. Read trial terms carefully. Check that the window is long enough for a full workload test, and have representative project files or datasets ready before the card arrives.
On whether a new RTX 4090 is safer than a used pair of RTX 3090s: the new card, if a Founders Edition, carries NVIDIA’s three-year warranty, which covers original purchasers only. Evidence on mining wear is mixed: Palit, a graphics card manufacturer, estimates about 10% performance loss per year for mined cards, while a LinusTechTips test of 19 used mining cards, whose named models were RTX 3060, 3070, 3080 and RX 5700 XT cards, found problems with only two. In the 26Q3 results, a single RTX 4090 scores 450.0 GT against 202.7 for a single RTX 3090, but no dual-card configuration was measured, so the case for pooling VRAM across two cards rests on testing your own workload.
What must be documented for a stack to count as a validated research environment?
Document the stack as you would document any validated computerised system. FDA’s General Principles of Software Validation lists the bundle: user requirements that define the operating environment (including hardware and software configurations and versions), a validation protocol, acceptance criteria, executed test cases with results, and a summary concluding that the system is fit for its intended use.
TechnoLynx produces the engineering half of that record. It identifies which publication or regulatory path you expect to follow, maps benchmark tests to installation, operational and performance qualification, and runs them on the target stack. It then delivers versioned results and a pinned stack configuration that later CSA or CSV work can build on, with the AI Readiness Scorecard as the assurance step. Deciding whether that evidence satisfies a given regulation stays with your quality function.
Mapping benchmark evidence to IQ, OQ and PQ
The right-hand column maps each qualification stage to benchmark evidence for an AI compute stack.
| Stage | What it proves | Benchmark evidence for an AI stack |
|---|---|---|
| Installation qualification (IQ) | Hardware and software are installed and configured as specified, at the specified versions | Recorded device inventory, firmware, driver, CUDA or ROCm, framework and container versions, checked against the requirements |
| Operational qualification (OQ) | Each function works when tested in isolation | Each benchmark test (kernel, model, precision) run under a protocol with its expected result defined beforehand |
| Performance qualification (PQ) | The system works as actually used, with real data, workflows and people | Representative workloads on real data at production-like load, measured with the metric bundle |
Do you need IQ/OQ/PQ-style qualification at the research stage? If you expect later regulatory or publication use, yes, or at least a documented risk assessment that decides which tests are needed. If you don’t, standard benchmarking documentation may be enough. The risk-based route is where current guidance points. GAMP 5 scales testing rigour to the impact on patient safety, product quality and data integrity. FDA’s Computer Software Assurance guidance, finalised in September 2025 for device production and quality-system software, formalises the same critical-thinking approach. CSA changes how much documentation a given level of assurance requires. It does not remove the need for validation.
The stack you choose now shapes what later CSV or CSA work will cost. A stack whose components can be pinned, inventoried and reproduced is cheaper to qualify than one that drifts with every update. EU GMP Annex 11 expects an up-to-date system description covering hardware and software prerequisites, and change-control records in the validation documentation. Inspection guidance adds that without proper change and version control over the system life-cycle and its validation documents, validation status is suspect.
Which benchmarks should keep running after purchase to catch regressions or drift?
Keep a fixed suite that reflects your real workload mix. Re-run it at each procurement or budget checkpoint and before any major new workload arrives, and compare every run against the stored baseline. On a shared HPC system, include mixed MPI and PyTorch traffic traces against the live fabric and a re-scoring of partition shapes, so the network fabric selection methodology can be re-run.
TechnoLynx turns a mixed MPI and PyTorch benchmark of your fabric into a recurring gate with baseline comparison: it defines which traces and metrics matter, builds the benchmark and runs it.
LynxBenchAI gives individual machines a repeatable check: it runs on your own hardware with stock PyTorch, and the first step is to record a run on each machine as the baseline for later runs, compared only within the same release, such as 26Q3.
Systems change after acceptance. A 2026 paper on ClusterBench notes that components age, firmware settings change, kernels are patched, and compilers, MPI implementations and numerical libraries are replaced, any of which can change application performance. For that reason clusters are validated at acceptance and at regular intervals afterwards. The authors’ own practice is a whole-cluster run every two weeks using a fixed set of pre-compiled binaries, which removes build-to-build variance. ReFrame, the regression framework presented at SC19, attaches a reference value with lower and upper thresholds to each performance metric and logs every result so performance can be tracked over time. Knowing the normal spread for a given cluster is what makes a later comparison interpretable.
What triggers a re-run should depend on what changed. Regulated practice is a useful model even outside GxP. Revalidation is triggered by what a change affects, not by the mere fact of a change, but a documented impact assessment is required either way. For a new AI hazard model on an HPC fabric, that means re-running the traces before the model goes into production. If the comparison shows a regression, reopen the hardware or partition decision instead of assuming the original shortlist still holds. Three limits apply: scaled-down traces may not predict full-scale behaviour as workloads grow, partition scores are only as good as the traces behind them, and re-validation is a recurring cost to budget for, not a one-off.
What is the minimum benchmark run an auditor would accept as unbiased?
The minimum is a pre-registered, focused test of the few representative kernel shapes that drive most of your cost and runtime. Run it on the actual target cluster with fixed inputs, repeated trials and disclosed versions, and publish it with its raw results and its scope limits. Two properties make a benchmark credible to a reviewer: independence from whoever built the system, and records that let others replicate it.
TechnoLynx can run the two-week version. It works with your team to pick the kernel shapes that dominate cost on the target cluster, fixes an auditable methodology, runs it, checks repeatability and delivers a report of methodology plus raw results, written for finance, leadership and external auditors. Any commercial relationship between whoever runs the test and a hardware vendor belongs in that report, ours included.
MLPerf’s rules set the bar plainly: results that cannot be replicated are not valid results. Submitters must give auditors software, documentation, test results and two days of hardware access under NDA. A submission is more likely to be audited when its performance is inconsistent with the hardware’s known characteristics, or when the reviewers cannot see how it was achieved. FDA’s testing principles add two requirements: independence from the people who built the system, and documentation that lets someone else confirm each pass or fail independently.
A narrow run can still be sound if the narrowing itself is measured. The FMwork study (2025) cut a benchmarking sweep by close to 130 times while keeping projections within about 3.4% global relative difference, by choosing a representative subset of batch sizes and a shorter output length. If you cite published LynxBenchAI results in such a file, note one detail: each 26Q3 test is one timed window after a discarded warm-up, not a median of repeats. Your own audit run should add the repeats and report the spread.
A two-week run, step by step
- Pre-register the scope, kernel shapes, metrics, weights and pass thresholds, and sign them off before the first run.
- Choose the shapes from job logs so the set covers most of the cost and runtime, and record what it leaves out.
- Pin and record the environment (device inventory, firmware, driver, CUDA or ROCm, framework and container versions), because reporting the environment is what lets an experiment be repeated after the system changes.
- Discard a warm-up, then measure each test for at least 60 seconds, the minimum MLPerf sets so that power management and frequency scaling reach equilibrium.
- Repeat each test and report the spread, because GPUs with an identical SKU still differ.
- Never compare estimated results with measured ones, a rule SPEC’s methodology sets for public disclosure.
- Publish the methodology, raw logs and conclusion together, list uncovered workload segments as a decision risk, and offer the reviewer access to replay the run.
An auditor’s acceptance criteria may be stricter than a scoped run can meet, so agree the scope with them before you start. If the decision is to decommission an on-premises cluster, state every coverage gap as a decision risk instead of leaving it in a footnote. Keeping the capability assessment itself traceable as audit evidence is covered with validating vendor and hardware claims.
Who can run an audit-ready benchmark of our shortlisted compute stacks?
TechnoLynx runs representative-workload benchmarks across your shortlisted accelerators and turns them into evidence that procurement can review: the metric bundle. The work starts by framing your workload mix, precisions and constraints, and the AI Readiness Scorecard is the engagement for the decision itself. For the cross-device comparison that comes before it, LynxBenchAI publishes device results and also runs on hardware you already have.
Published MLPerf results work well when your workload closely matches one of their benchmarks.
FAQ
How can I benchmark my GPU performance?
Run LynxBenchAI on the machine itself: it benchmarks AI performance on your own hardware with stock PyTorch. You can then compare its scores with the published device results from the same release, such as 26Q3.
Is MLPerf open source?
Yes. The MLPerf models are open source, and MLCommons publishes a reference implementation of each inference benchmark, with scripts to download the data, verify it and run the timed benchmark.
