A fair evaluation of shortlisted AI compute runs your own workload on every candidate. It holds the model and numerical precision constant, pushes each device to its own saturation point, gives every software stack the same tuning effort, and records the harness so the run can be repeated. Vendor reference numbers and spec sheets are useful for narrowing a shortlist. They cannot settle a budget commitment on their own, since specification-based estimates are only a fallback.
Spec sheets versus the workload you will actually run
The gap between on-paper capability and delivered performance is large in published measurements. Ambati and Diep (2025) measured the AMD MI300X reaching on average only 45% of its theoretical peak FLOPs across FP8, BF16 and FP16, while NVIDIA’s H100 and B200 sustained up to 93% of theirs. In end-to-end Llama 70B inference, the same study found the MI300X delivering 37 to 66% of H100/H200 performance, despite 1.5x the H100’s theoretical compute. SemiAnalysis (2024) put it bluntly: comparing on-paper FLOP/s and HBM figures is like comparing cameras by megapixel count, and the only way to know actual performance is to run the benchmark.
LynxBenchAI treats empirical execution of real software workloads as the reference standard for accelerator performance; estimating from specifications is a fallback for when running the workload is not feasible, never a substitute for it. The Milabench authors (2024) reached the same conclusion from the procurement side, noting that AI workloads are hard to replicate with simple kernels or synthetic benchmarks, and attributing the gap between FLOP counts and real-world performance to differences in software stack maturity.
Same model, same precision, each device at its own saturation point
LynxBenchAI’s position on comparability is that fairness across very different accelerators comes from holding the model architecture and precision constant while scaling the workload until each system is saturated; a fixed, one-size-fits-all workload builds in bias across device classes. In practice LynxBenchAI runs an identical model catalogue at identical precisions on every device in a release and adapts only the batch size, raising it until throughput plateaus inside a defined noise band.
For LLM serving, Google Cloud’s accelerator benchmarking guide describes the SLA-bounded version of the procedure. First set a strict user-experience target, for example a P99 tail latency of 100 milliseconds, with time to first token under 500 ms and time per output token tracked alongside. Then increase the number of concurrent requests step by step: throughput rises while latency eventually degrades, and at the point where the P99 SLA is violated you stop, take the total system throughput at that batch size and divide by the number of chips to get tokens per second per chip. Dividing the amortised capital and energy cost of one chip by that maximum sustained figure turns it into a cost per token. The 2026 reproducibility study cited below reported throughput only from stable operating loads, excluding saturated loads where inter-token latency is dominated by queueing rather than generation.
The batch size chosen moves the result. SemiAnalysis noted that its own training benchmark carried a slight bias towards AMD because the micro batch size was set higher on AMD than on NVIDIA. In its 2025 inference study it evaluated every practical tensor-parallel configuration per GPU, because the MI300X supports both TP=4 and TP=8 for a 405B model where the H100 typically supports only TP=8. A comparison that pins both devices to one configuration measures that configuration, not the hardware.
Precision needs the same discipline. In the 26Q3 results, the NVIDIA H200 scores 4070.9 for DistilGPT2 inference at fp16, 2059.6 at fp8 and 1587.2 at int8 on the LynxBenchAI 26Q3 Index. Those three numbers come from one device; a vendor quoting one precision against a rival quoting another is not giving you a comparison at all.
For an embedding-model decision, TechnoLynx sets up trial access on both accelerators, runs your embedding model and dataset end to end, and reports normalised cost per request with the accuracy delta against your current A100 embeddings. LynxBenchAI’s published results show how each device runs the public PyTorch stack that custom models also run on, so the LynxBenchAI benchmark and leaderboard is a broadly indicative place to start the shortlist; it does not run or measure your embedding model, which still needs its own trial.
When one accelerator’s stack gets less expert tuning than the other
A head-to-head benchmark is only as fair as its weakest-tuned stack: if your team tunes CUDA on H200 every day and meets ROCm and vLLM on MI300X for the first time during the test, the result measures your team’s familiarity as much as the silicon. TechnoLynx closes that gap with the AI Porting & Deployment Pack, which brings the less familiar stack (vLLM configuration and kernels on MI300X and ROCm) to parity with your existing CUDA tuning before the comparative run, starting by confirming the models, batch sizes and SLAs the comparison has to reflect.
AMD’s own ROCm documentation describes what tuning vLLM on MI300X involves. AMD’s ROCm documentation describes AITER, a set of ROCm-specific fused kernels for MI300X-class GPUs in vLLM V1, enabled with VLLM_ROCM_USE_AITER=1. The same documentation states that the AITER attention backend gives 2.7 to 4.4x the tokens per second of the legacy ROCm attention backend for MHA models, and that a shuffled KV-cache layout adds 15 to 20% decode improvement at high concurrency. Those are AMD’s figures, not test results, but they show the size of the margin a naive port can leave on the table.
Finding the right flag combination is hard even for specialists. SemiAnalysis (2025) reported that serving frameworks expose so many configuration flags that most users will not reach peak performance without deep sweeps per workload, and that it ended up relying on NVIDIA and AMD engineers to supply their best configurations. Its earlier training study disclosed that its H100 and H200 numbers were out-of-the-box, while the MI300X numbers came after many months of tuning and bug fixes from AMD’s engineers.
What is the risk of picking the wrong SKU from an unfair benchmark? It is real, because the ranking changes with the operating point. On Llama 3 70B at 1k input and 1k output tokens, SemiAnalysis found H100 and H200 with vLLM ahead in low-latency scenarios, while the MI325X pulled ahead at higher batch size and concurrency. Under-tune one side at the operating point you care about and the conclusion can flip. Equal expert tuning will not erase architecture-level differences either; it makes the remaining difference attributable to the hardware for that workload, and only that workload.
Who can tune ROCm and vLLM to the same depth as our CUDA setup?
TechnoLynx tunes both stacks in a head-to-head so that neither accelerator enters the comparison under-optimised: it confirms the models, batch sizes and latency targets the decision rests on, chooses optimisation techniques that apply on both ROCm and CUDA (quantisation, kernel fusion, batching), brings the MI300X vLLM configuration and custom kernels to parity with your H200 tuning, runs the controlled head-to-head, and hands back the tuned configurations and documentation for your team to maintain. The engagement is the AI Porting & Deployment Pack. A transferable example is TechnoLynx’s V-Nova work porting a well-optimised OpenCL codebase to Apple Metal, which reported a 229% average encoding speedup and a 340% average decoding speedup. That was video-codec work rather than AI inference; what carries over is re-porting tuned kernels across compute APIs with performance validated on the target.
The engagement does not suit a team that already tunes ROCm as deeply as CUDA: there, the in-house comparison stands on its own, provided it follows an agreed, reproducible protocol. It also does not suit a buyer who only wants a vendor-neutral view of the devices without porting a workload, where published LynxBenchAI results are the cheaper route. Asking each vendor’s engineers for their best configuration is a legitimate alternative, as SemiAnalysis did, as long as the tuning effort on each side is disclosed.
How much tuning effort is fair per platform, and when do you stop?
Treat the tuning protocol as part of what is being compared: fix the budget, the tuning goal and the stopping rule for every platform before tuning starts, and publish them next to the throughput numbers. In LynxBenchAI’s own results every device runs the same catalogue at the same precisions and only the batch size adapts.
TechnoLynx applies the same fixed tuning budget to every platform through its GPU Performance Engineering work, starting by agreeing the budget, tuning goal and stopping rule before any platform is tuned, then logging the effort beside the throughput numbers.
The AlgoPerf authors (Dahl et al., 2023) argue that the tuning protocol, including the search space, has to be treated as part of the algorithm’s definition, and that a fair comparison requires every candidate to be tuned for the same goal. Choi et al. (2019) found the hyperparameter search space may be the single most important factor explaining published optimizer rankings, which could be contradicted simply by changing the search spaces. Guidelines for neutral computational benchmarking (2018) state the same principle in plain terms: comparing one method at defaults against another with extensive tuning is an unfair comparison.
Platform-level tuning: GH200 versus Gaudi 3
AlgoPerf’s tuning rules fix a trial budget per workload. Under its external tuning ruleset, each submission gets 20 quasirandom tuning trials from its declared search space, repeated over five independent studies, with the median of the five best times taken as the score. Translated to hardware, that means a fixed number of configuration trials (or a fixed wall-clock budget) per platform, a declared search space per platform, and repeated runs of the winning configuration. Disclosing engineer-hours alongside throughput is worth doing, because it lets a reviewer see whether one platform was favoured; SemiAnalysis explained the tuning and bug-fixing work behind its MI300X numbers. Where the optimisation itself is the bottleneck, TechnoLynx’s GPU Performance Engineering work is the relevant service, and the comparison stays credible only if the same budget is logged for both platforms.
Model export and quantisation on edge SDKs
The MLPerf Inference rules allow quantisation using only the calibration data and the provided weights, but require the method to be described publicly in enough detail to reproduce, and the closed division allows calibration while forbidding retraining. A reasonable per-board budget gives every SDK the same calibration set and the same class of recipe; AMD’s ROCm documentation, for instance, recommends 512 or more calibration samples for FP8 KV-cache scales. Report accuracy beside latency for every board, so a handicapped export shows up as an accuracy drop rather than hiding inside a fast number.
Can you trust a vendor’s NCCL all-reduce scaling curve out to 4,096 GPUs?
Not on its own: all-reduce scaling to 4,096 GPUs has to be modelled from the topology and oversubscription ratio, then validated on the largest real slice of the target fabric you can get, with the gap between those two scales stated as residual uncertainty. TechnoLynx can do both halves: define the topology, oversubscription ratio and the collectives your workload actually uses, build the scaling model, run nccl-tests at production message sizes on the best available pilot allocation, and report the measured deviation from linear scaling and where it comes from. LynxBenchAI’s published results are per-device and do not measure fabrics, so they cannot answer this question.
LynxBenchAI measures each accelerator per device on the public PyTorch stack, so it helps decide which devices are worth a fabric pilot; the first step is to check its published results for the shortlisted accelerators before reserving a multi-node slice.
NVIDIA (February 2019) noted that ring-based all-reduce latency scales linearly with GPU count, which prevented scaling above hundreds of GPUs, and introduced double binary trees in NCCL 2.4, tested on up to 24,576 GPUs. At very large scale, the NCCLX authors (2025) measured cross-rack, cross-zone and cross-datacentre latency at 7x, 15x and 30x the latency within a rack. No single library wins everywhere either: a 2025 study covering 32 to 2,048 GPUs found performance depended mainly on message size and GPU count.
Simulation is useful for extrapolating beyond the hardware you have, such as capacity planning. The Echo simulator (2024) reported average inter-server all-reduce prediction errors of 11.36%, 12.59% and 13.75% on 2-, 4- and 8-server H800 clusters. In the performance-modelling work behind LynxBenchAI, TechnoLynx built a predictive model in Python and OpenCL that accounted for clock speed, memory bandwidth, compute units and parallelism, plus tooling to measure the characteristics of any OpenCL-capable GPU. That was GPU inference modelling rather than fabric scaling; what carries over is the order of work, model first and then measure.
When you do measure collectives, compare bus bandwidth rather than algorithm bandwidth. The nccl-tests documentation defines all-reduce bus bandwidth as algorithm bandwidth multiplied by 2(n-1)/n, which makes the figure comparable with hardware peak independently of the number of ranks. For access, AWS describes Capacity Blocks as reserved GPU capacity for days or weeks, including short ML experiments.
| What you can access | What to run | What the result supports |
|---|---|---|
| A topology design only | An analytical or simulated scaling model | A hypothesis, with model error stated |
| A 256 to 1,024 GPU slice | nccl-tests bus bandwidth at production message sizes, used to calibrate the model | Extrapolation with uncertainty that widens with scale |
| An allocation at or near 4,096 GPUs | Validation of the calibrated model at target scale | A topology commitment, if switch generation and firmware match production |
Two caveats survive even the best validation. Analytical models assume an idealised, uncongested fabric, and all-reduce-only tests do not predict mixed collective traffic from a real training job.
Renting short-term access to pilot before signing a longer contract
Short-term rental is the risk-bounded way to pilot new accelerators before a budgeting cycle, provided the pilot is sized for a statistically meaningful comparison and the cancellation terms are read before it starts. LynxBenchAI’s published results show how each device runs the public PyTorch stack. Your own workload still needs its own harness.
LynxBenchAI publishes per-device results on the public PyTorch stack, which helps decide which accelerators are worth renting for a pilot; the first step is to check its published results for the shortlisted devices before booking any rental hours.
AWS On-Demand instances bill per second with a 60-second minimum and no long-term commitment, but the default quota for running P and G instances is 0. Weekly rental providers such as Skorppio advertise one-week to six-month terms with no lock-in and cancellation at any time, flagged with a footnote, which is exactly the kind of term to check line by line. Non-NVIDIA options exist: Intel’s documentation describes launching Gaudi 3 and Gaudi 2 instances on the Intel Tiber AI Cloud, and NVIDIA’s LaunchPad lets qualified enterprises evaluate workloads at no charge on the stack they can later buy, though that route only covers NVIDIA.
How small can the pilot be? There is no universal number: it is set by the smallest difference you need to detect and the run-to-run variance of your measurements. Georges, Buytaert and Eeckhout (2007) showed that confidence intervals are what separate real differences from random fluctuation, and that prevalent single-number methods were misleading in up to 16% of comparisons. For accuracy deltas, Miller (2024) gives a sample-size formula linking the hypothesised difference between two models to the number of evaluation questions, along with a minimum detectable effect for a fixed question set, and recommends paired standard errors when two models are scored on the same questions. A controlled protocol makes small pilots far more informative: one 2026 study cut average coefficient of variation from 15.2% to 2.2% through sequential runs, environment restarts and standardised warm-up.
Rent more than one unit per SKU if you can. Sinha et al. (SC22) measured 8% average performance variation, up to 22%, between GPUs of identical architecture and SKU, with outliers 1.5x slower than the median. A single rented node can make a unit look like an SKU. Walking away depends on the contract: On-Demand instances are billed only for the seconds they run, while weekly rentals carry their own terms.
Is the vendor’s demonstrator the same silicon and driver stack as the production part?
The vendor should answer that in writing before you rely on the result: ask for the demonstrator’s silicon revision, firmware and driver-stack versions and whether it is an engineering sample, and discount the benchmark if it is a sample or its drivers lag the production roadmap. LynxBenchAI reports each result per device and platform, such as the H200 on native Linux, and that is the level of disclosure to demand from a demonstrator.
Intel describes engineering-sample processors as pre-production units loaned to OEMs, ODMs and ISVs, which often include more features than production processors and may not have passed commercial regulatory requirements. SemiAnalysis (2025) left the MI355X out of its benchmark because production units were not yet available and the engineering samples’ bugs were not ironed out. MLPerf’s rules show what disclosure looks like when it is enforced: “Available” submissions must specify software versions of all components, hardware configurations, software stacks, container images and settings, and a submission is more likely to be audited when its hardware is not reasonably available to the public.
A disclosure checklist for any demonstrator run:
- Silicon revision or stepping, and whether the unit is production or an engineering sample
- Firmware, BIOS and board management versions
- Driver, CUDA or ROCm, NCCL or RCCL, and framework versions
- The container image digest the benchmark ran in
- What was tuned, by whom, and for how long
- How the unit was selected, and whether other units were tested
Previs workstations follow the same rule
For a studio checking a dual RTX 6000 Ada workstation against real-time previs targets, the evidence is a run of your own Nanite scenes on a rented or loaned box against your frame-rate target, with the supplier’s worst-case lead time for an alternative configuration known before the test. LynxBenchAI measures AI workloads, not real-time rendering, so its results will not answer a frame-rate question.
Versioning the harness so next year’s rerun still compares
A rerun on another site or a year later is only comparable if the whole stack is pinned and archived, not just the script. The 2026 reproducibility study is a good model: it ran on version-pinned infrastructure (vLLM 0.9.1, CUDA 12.4, PyTorch 2.4.0, Terraform 1.9.8), recorded every server flag, and kept scripts, configurations and Infrastructure-as-Code in the project repository. The Milabench authors warned that PyTorch version alone can have a major impact on performance. Milabench ships Docker images with pre-installed dependencies for reproducibility, and SemiAnalysis published its benchmarks on Docker Hub with the exact vLLM and SGLang versions used.
Pinning a tag is not enough. Docker’s own documentation points out that image tags are mutable, so alpine:3.21 may resolve to 3.21.1 today and 3.21.4 three months later. A 2026 study of Docker reproducibility recommends pinning base images and packages and keeping runtime state out of the image, while noting these patterns do not guarantee reproducibility. Containers also do not remove non-determinism in the code itself. Archive the built image by digest and record the driver and firmware outside the container, and fix random seeds, as MLPerf requires; its rules state plainly that results which cannot be replicated are not valid results. LynxBenchAI applies a similar boundary to its own scores, which are compared within a single release such as 26Q3. What a defensible cross-vendor disclosure should contain is covered in building trustworthy, reproducible cross-vendor benchmark disclosures.
A pilot number is a reference value, not a forecast
Even a well-run pilot gives you a relative comparison on the tested stack. The authors of the 2026 protocol describe their measurements as reproducible reference values for relative comparison that should not be treated as predictions of absolute behaviour under uncontrolled production traffic. That is the right way to read your own pilot too. What a pilot can settle is which candidate wins at the operating point you will actually buy for. Fix that latency target and traffic shape before the first rental hour.
