A fair comparison across vendors, precisions and batch sizes needs a disclosed baseline, and a headline score is not one. That baseline has four parts. Every layer of the stack is either held fixed or written down. Training throughput is scored separately from inference cost. A slowdown is traced to the layer that caused it. Accuracy is reported per segment instead of as one blended average. Remove any one of these and the comparison quietly measures something other than what the reader thinks it does.
The LynxBenchAI 26Q3 release reports training, inference and compute as separate categories in its table of 48 published devices. Its AI hardware benchmark and leaderboard also publishes the per-model, per-precision scores that sit under each category.
Hold every layer fixed but one, and write the rest down
The HPC AI500 methodology splits an AI system into nine independent layers: hardware, operating system, communication libraries, accelerator and its libraries (CUDA, cuDNN), AI framework, programming model, workload, hyperparameters, and problem domain. Its basic rule is to put one layer under test and keep the others intact. The rule matters because a change to a single layer, such as the framework or the delegate, can move the result on its own. HPC AI500 requires the same framework at its hardware level because the same model on different frameworks reaches different accuracy. The MLPerf Mobile paper reports performance varying by 10% with the delegate, and cites prior work showing gaps of up to 7x between delegates and vendor SDKs on some chipsets because of buggy operator implementations.
MLPerf’s closed division makes this an equivalence rule. Submissions must match the reference implementation in model, initialisation, optimiser, training schedule and data traversal. From v6.1, every MLPerf Training result log must report the lowest numerical precision used in linear, attention and communication compute, plus the parallelism settings, micro-batch size and configuration file. MLPerf also treats a single summary score across its suite as unsuited to its results. Any summary score implies a weighting, and no weighting represents every user.
We read these rules as one argument. A comparison is only as fair as its least-disclosed layer. The checklist below lists what a result needs to carry before two vendors’ numbers can share a chart.
| Layer | What must be disclosed | Why it changes the number |
|---|---|---|
| Hardware and interconnect | Device, count, node topology, intra- and inter-node links | Communication can move from NVLink-bound to Ethernet-bound as scale grows |
| System software | OS, driver, runtime versions | MLPerf submissions list OS and library versions for reproducibility |
| Accelerator libraries and framework | CUDA/ROCm, cuDNN, framework and delegate | The MLPerf Mobile paper reports 10% variation by delegate and cites up to 7x delegate-versus-SDK gaps on some chipsets |
| Numerics | Lowest precision per compute type | MLPerf Training logs it per result from v6.1 |
| Hyperparameters and batch | Batch size, learning-rate policy | Larger minibatches can need more epochs to reach the same accuracy |
| Repeats and variance | Number of runs, spread, outlier handling | MLPerf drops the fastest and slowest runs and averages the rest |
| Quality target | Accuracy threshold the run had to meet | Performance only means something at a stated quality |
The 2024 BetterBench assessment found that 14 of 24 AI benchmarks ran no repeat evaluations of the same model and reported no statistical uncertainty. Without intra-model variance, a reader cannot tell a real difference between models from noise.
LynxBenchAI’s 26Q3 results disclose their own position on repeats. In the 26Q3 results, each test is one timed window after a discarded warm-up, not a median of repeats. A score of 0.0 means that precision produced no result on that executor. For example, the NVIDIA GeForce RTX 3080 and the A100 80GB PCIe both record 0.0 at FP8 inference, so they have no FP8 result to compare. Scores are comparable within a major release and not across major releases, and the GT aggregate is ordinal, so a GT gap is never a percentage. Given the single-window design, we treat small gaps between neighbouring devices with caution and read larger, consistent gaps across several tests as the signal. Our reproducibility checklist for fair, trustworthy benchmarks covers rerunning a result from its disclosure in more depth.
Is the slowdown in the hardware, the build or the model?
To separate a codegen limitation from a memory-bandwidth limit, or either from a model that is simply too large, run a layered diagnostic on each device tier. A WASM-compiled transformer translation model that runs slowly on low-end CPUs is a good example. Profile the instruction mix, measure memory traffic and place each tier on a roofline before deciding whether to rebuild, shrink or quantise.
TechnoLynx runs this layered diagnostic as part of its GPU Performance Engineering work. It defines the CPU tiers, the WASM runtimes and the throughput and quality thresholds, captures codegen and bandwidth profiles on each tier, and classifies each tier as codegen-bound, bandwidth-bound or model-bound before choosing any fix.
The roofline test: operational intensity against the ridge point
Operational intensity is the number of operations per byte of DRAM traffic, counted after the cache hierarchy has filtered it. The roofline caps attainable performance at the minimum of peak compute and peak memory bandwidth times operational intensity. A kernel that hits the flat part of the roof is compute-bound. A kernel that hits the slanted part is memory-bound. The ridge point is the minimum operational intensity needed to reach peak performance. Williams, Waterman and Patterson found it a better predictor of performance than clock rate or peak performance.
The roofline’s authors propose performance counters to show which architectural features a run fails to exploit. They suggest dividing floating-point SIMD instructions issued by all floating-point instructions issued to define a true SIMD ceiling. HPC AI500 classifies GPU kernels by share of time, instructions per cycle and DRAM utilisation.
Disclose how bandwidth was measured. STREAM estimates application-level traffic from execution time, data-structure sizes and load/store counts. Hardware counters also capture microarchitecture transfers. Different platforms show different relationships between the two, so the authors of the Mess benchmark recommend reporting both.
The classification can also change once you start fixing things. In one simulation study, adding granularity turned a memory-bandwidth-bound implementation into a CPU-bound one. SIMD then added about 50% on an Intel Xeon and about 40% on an AMD Opteron. After each fix, re-profile the tier before choosing the next one.
What 128-bit WASM SIMD means for per-tier builds
The WASM SIMD specification defines only 128-bit SIMD instructions, while modern x86 processors support 512-bit AVX-512. A 2023 study of WebAssembly in HPC attributes part of its gap to native code to that difference. On that study’s DT benchmark, the WASM build with SIMD delivered 1.36x the throughput of the WASM build without SIMD. Wider vectors were still only a proposal (Flexible Vectors) at the time. Within standard WASM SIMD, a per-tier build therefore cannot choose a lane width wider than 128 bits. We would only take on that maintenance burden when profiling shows codegen mismatch dominating on the tiers you are targeting.
Quantisation also helps on the WASM backend. A 2024 SitePoint browser-inference comparison reports INT8 models running 2 to 3x faster than FP32 on the WASM backend, because SIMD instructions operate on packed 8-bit integers. The same guide records browser, driver and OS versions with every result, and advises measuring on the devices your users actually have.
| Per-tier diagnosis | First move | What not to do yet |
|---|---|---|
| Codegen-bound: low bandwidth use, poor SIMD ceiling | Fix compilation flags and instruction selection; consider a tier-specific build | Shrink the model before the build is fixed |
| Bandwidth-bound: near-saturated memory traffic | Reduce data movement: quantise, improve memory layout, smaller variant | Maintain extra SIMD builds |
| Model-bound: too large even when codegen and bandwidth are efficient | Evaluate smaller or further-quantised variants directly | Assume the fix found on a faster tier applies here |
| Fixed overheads dominate | Address startup and dispatch costs | Expect quantisation to help much |
The slowest devices may hit a different bottleneck from the faster ones. Any quantised variant has to clear the translation-quality threshold before its throughput counts.
Who can find the bottleneck in our WASM model on each device tier?
TechnoLynx diagnoses and fixes slow WASM-compiled models across the device tiers you ship to, through the AI Porting & Deployment Pack. The engagement starts by defining the low-end CPU tiers, target browsers and runtimes, and the throughput and quality thresholds for each. It then profiles codegen and memory bandwidth on each tier and builds a per-tier roofline. Depending on the diagnosis, the fixes are adjusted SIMD compilation, kernels restructured for memory locality, or trials of smaller and quantised variants. Tier-specific builds are prototyped against the single-build baseline. Translation quality and throughput are validated on every tier before deployment, and you get guidance on monitoring for regressions as runtimes change.
This engagement does not fit a team whose bottleneck is server-side GPU capacity rather than client devices. It also does not fit a team that only needs a hardware shortlist. LynxBenchAI’s published results cover the PyTorch stack on Linux, WSL and macOS, not browser WASM runtimes, so they won’t answer this question. A team with an in-house performance engineer and access to the real device fleet may already have the profiling skills. In that case, an outside review of the per-tier classification is a lighter option.
How should a tender scorecard keep training throughput and inference cost apart?
Write training throughput and inference cost per request as two line items. Give each its own metric definition, weight and appeal route, and leave no cell on the scoring sheet where the two are added together. Training and inference stress a platform differently. Google Cloud’s accelerator benchmarking guide describes training as often compute-bound and measured in tokens per second per chip and MFU. It describes inference as latency-sensitive and measured by time to first token, inter-token latency and tokens per second per user.
LynxBenchAI scores training and inference as separate categories for each published device, which gives a tender a disclosed public reference for keeping the two line items apart, and the first step is to run it on your own hardware with stock PyTorch and compare its scores only with entries from the same release.
LynxBenchAI’s published results give a tender a disclosed public reference for this split, because the 26Q3 release scores training and inference as separate categories. The results show how each device runs the public PyTorch stack that a team’s own models also run on, so they are broadly indicative for custom PyTorch models. They do not run or measure the buyer’s or a bidder’s own workload.
The split matters in the published numbers. In the 26Q3 results, the AMD Instinct MI300X scores 727.2 in training and 1023.3 in compute, against 709.3 and 487.1 for the NVIDIA H100 80GB HBM3. The H100 scores 1678.1 in inference against the MI300X’s 1304.0. Inside inference, the gap also depends on the model. At FP16, the H100 scores 2833.0 on MobileNetV3 against 733.8 for the MI300X, and 3797.7 on DistilGPT2 against 3318.5. Put those results in one column and the column’s weighting picks the winner, not the hardware.
Metric definitions every bidder must cite
NVIDIA’s NIM benchmarking documentation warns that tool implementations vary and that results are comparable only when the definitions match. It defines total tokens per second (TPS) as output-token throughput across all simultaneous requests. That figure rises until the GPU’s compute saturates and can fall after that point. AIPerf computes TPS as total output tokens divided by the time from the first request to the last response. TPS per user is output length divided by end-to-end latency for each request, and it falls as concurrency rises. Requests per second (RPS) is the average number of requests the system completes in one second. Median-only and P99-only reporting each hide trade-offs between systems.
Cost per request is computed from sustained RPS. Divide the platform’s hourly cost by sustained RPS times 3,600. Google Cloud’s guide computes cost per thousand or million tokens the same way, dividing a chip’s amortised capital and energy cost by its maximum sustained TPS per chip. Peak TFLOPS has no place in either formula.
Effective cost depends on offered request rate, so the tender has to fix the load. A 2026 concurrency-aware cost study measured Mixtral 8x7B at FP16 on one H100 at $15.25 per million tokens at 1 request per second, and $0.87 at 25 requests per second. Holding model, precision and GPU fixed, effective cost moved 17.5 to 36.3x with offered request rate alone. The same study treats the latency SLA as a price: a tighter SLA forces a lower load and a higher cost per token. Its authors conclude that any cost figure quoted without its request rate is meaningless. A tender should therefore fix the model, precision, input and output lengths, arrival rate and latency target, then ask every bidder for cost per request at that operating point.
Samples per second, steps per second, or time to quality?
Steps per second at different batch sizes are not comparable. MLPerf’s ResNet-50 needed about 64 epochs to reach its target accuracy at a 4K minibatch, and more than 80 epochs at 16K, a 30% increase in computation. MLPerf scores training as time to a defined quality target, which combines speed and accuracy. Google’s guide adds goodput, which excludes time lost to faults, stalls and checkpoint recovery. For LLMs, Google’s guide normalises on tokens per second per chip. Whichever unit a family uses, the tender should fix it in advance, normalise it per dollar and pair it with time to quality.
| Scorecard line | Metric | Held fixed by the tender | Scored and appealed |
|---|---|---|---|
| Training throughput | Tokens/s or samples/s per chip per dollar, plus time to quality | Model family, dataset, quality target, precision | Separately |
| Inference cost | Cost per request at the stated RPS and P99 latency target | Model, precision, input/output lengths, arrival rate, SLO | Separately |
Configuration choices can still skew the comparison. A known example is heavy tuning of one system against baselines left on defaults. The evaluation literature lists that as an anti-pattern that invalidates the comparison. The evaluation checklist asks that all performance-critical parameters be documented for every system and that baselines be tuned appropriately. An appeal route for each line item costs procurement time, and we think that cost is worth paying. We cover how to make each figure hold up under challenge in building defensible benchmarks for procurement and capacity decisions.
Why report accuracy by lighting, density and shelf type instead of one average?
A single blended accuracy number can hide large failures on a narrow subset of the data. A quantised shelf-detection model should therefore be scored per lighting condition, SKU density and store format, with any failing segment flagged even when the blended figure passes. TechnoLynx defines the segmentation scheme against known shelf and store variability, designs the sampling weights and per-segment equivalence tests, and connects the pipeline to the model’s detection outputs.
Published studies show averages hiding this damage. Hooker and colleagues found that pruned and quantised networks with very different weight counts match on top-line metrics but diverge on a narrow subset of the data. That subset over-indexes on atypical or noisy images. Oakden-Rayner and colleagues found hidden stratification producing relative performance differences of over 20% on clinically important subsets in medical imaging. Barocas and colleagues call the remedy disaggregated evaluation: assessing and reporting performance separately for each group. MLPerf Mobile applies a related discipline to precision. Its accuracy targets are all above 93% of FP32, even for INT8 models.
A 2024 study in Sensors by Šikić, Kalafatić, Subašić and Lončarić reported 86.3% average precision on fully empty facings and 83.7% where the front product was missing. The authors noted that the size, lighting and background of out-of-stocks vary widely. An independent field note on shelf-scanning robots recommends putting the deepest shelves, promotional end caps, lookalike sets and blocked aisles into the test on purpose, and recording what was left out.
A segment-stratified accuracy procedure
- Define the segment axes: lighting condition, SKU density and store format, plus any axis where the deployment’s conditions vary.
- Stratify the sample by segment: stratified sampling with proportional or Neyman allocation gives more precise accuracy estimates than simple random sampling.
- When you aggregate, use an estimator matched to the sampling design, such as the Horvitz-Thompson or the model-assisted difference estimator.
- Test equivalence against the full-precision model in every segment, not only in the blended set.
- Flag any segment that fails tolerance, whatever the blended number shows.
This procedure has limits. Finer segments leave fewer samples per cell, and each cell needs enough data for narrow confidence intervals. A store format or lighting setup that isn’t in the scheme yet can still hide a regression. The LynxBenchAI 26Q3 results also publish per-test scores under each category. Results are published per model and precision under each category, so a weak precision on one executor shows up instead of being averaged away.
Reading the 26Q3 results as a baseline
Published results do two jobs. They fix a disclosed reference that a tender can point to, and they show where devices diverge by category, model and precision. They cover a fixed public model set, not your model, your arrival rate or your shelf conditions. The layered diagnosis, the cost-per-request measurement at your SLO and the segment tests still have to run on your own workload.
To put your own machine on the same footing, run LynxBenchAI on your own hardware with stock PyTorch. Compare its scores only with entries from the same release. Even then, a gap that is smaller than the run-to-run variance you would expect from a single timed window cannot be told apart from noise. MLPerf requires several runs of each benchmark to stabilise timing.
FAQ
Is higher memory bandwidth better?
Only when the workload is memory-bound. On a roofline, a kernel whose operational intensity is below the ridge point is limited by bandwidth, and a kernel above it is limited by compute, where extra bandwidth adds nothing.
