To show that two accelerator platforms got the same tuning effort, fix what is being compared and write down what was done to each. That means holding the model and numerical precision constant and scaling the workload until each platform saturates. Tuning parity is one of several separate validation jobs before committing to AI server hardware or a cluster design. SoC equivalence, fabric oversubscription, switch-port sizing and field lab trials each need evidence of their own, and treating them as one generic “evaluation” is how the wrong evidence ends up in front of the decision.

How do we prove two accelerator platforms got equal tuning effort?

Google Cloud’s accelerator benchmarking guidance warns that a model trained on one chip should not be expected to perform well out of the box on another, because experienced users co-design models with the hardware. Mila’s Milabench team ran its suite on NVIDIA A100 and H100, AMD MI300X and Intel Gaudi2, and put the gap between FLOP counts and real-world performance down to differences in software stack maturity. On mobile SoCs, the MLPerf Mobile authors report that, depending on the chipset, results through generic delegates and vendor SDKs sometimes differ by 7x because of poor or buggy op implementations.

So “equal effort” cannot mean “no effort”. It means the same declared level of effort on every platform. SPEC’s run rules give a useful template. A “base” result builds every benchmark with one common set of optimisations, and it is meant to reflect a user who wants performance but will not tune individual programs. A “peak” result allows optimisations chosen per benchmark. The Bosch deep learning hardware benchmark splits its procedure in a similar way, into unoptimised runs and optimised runs, distinguishing the optimisation of vendor hardware from software. Research on optimiser benchmarking argues that a fair assessment must count the computational cost of hyperparameter tuning, not only results at near-optimal settings.

How LynxBenchAI keeps the comparison fair

LynxBenchAI runs the same model catalogue at the same precisions on every device in a release, and adapts batch size to each device. Batch size is raised until throughput plateaus inside a noise band. A laptop GPU and a data-centre accelerator are therefore each measured at their own saturation point, not at one fixed workload that happens to suit one of them. A single fixed workload can favour one device class over another. An edge study on an Intel AI PC shows why: in video classification the NPU did better at small batch sizes and the GPU at larger ones, so a single fixed batch would have chosen the winner in advance.

Run-to-run variance also has to be stated. Recent work shows that GPU-rich systems vary significantly in performance, and that the variation grows with compute intensity because of power-management behaviour such as DVFS. SPEC deals with this by reporting the median of three runs. In the 26Q3 release, each LynxBenchAI test is a single timed window after a discarded warm-up, not a median of repeats, and its scores are comparable only within 26Q3. Small differences between neighbouring devices are therefore best read as unresolved, not as a ranking.

Category results show why one headline number is not enough. In the 26Q3 results, the AMD Instinct MI300X scores 1023.3 on compute against 487.1 for the NVIDIA H100 80GB HBM3, while the H100 scores 1678.1 on inference against the MI300X’s 1304.0. On training the two are close: 727.2 for the MI300X and 709.3 for the H100. Results also differ by platform on the same card. The GeForce RTX 5090 on native Linux scores 985.8 on inference and the same card under WSL scores 891.8.

Tuning-parity checklist before numbers are frozen

Check each item below before numbers are frozen:

1. Same model and precision on every platform. Only the workload size changes, and it is raised to each platform’s saturation point.
2. A declared tuning tier. Either one common set of optimisations for every benchmark (base-style) or optimisations chosen per benchmark (peak-style), as SPEC defines them.
3. Recorded tuning effort. Record the tuning effort spent on each platform.
4. Pinned, disclosed versions. Milabench could not run every vendor on the same PyTorch version, and notes that the PyTorch version can have a major effect on performance. SPEC requires compiler and driver names and versions in the disclosure.
5. Backend choice stated. Say whether a vendor SDK or a generic framework path was used, since the framework choice has a substantial effect on hardware performance.
6. Accuracy checked after optimisation. Hoefler and Belli recommend always measuring and reporting accuracy after performance optimisation.
7. Estimates labelled. Mark any estimate clearly as an estimate, as SPEC requires.

For disclosure, the SPEC ACCEL and SPEC CPU 2026 rules require configuration details sufficient to reproduce the results. Milabench covers scaling as well. Its tasks run on a single GPU, on a multi-GPU node and across multiple nodes, and every available GPU is used so that full-node performance is measured. The wider process of evaluating AI compute candidates with benchmarks, pilots and fair comparisons covers where these numbers fit in a procurement decision.

To see this method on a device you already own, LynxBenchAI benchmarks AI performance on your own hardware with stock PyTorch. It does not run or measure your own workload. The published results are on the LynxBenchAI benchmark and leaderboard.

Projecting multi-year battery life from six weeks of bench data

Six weeks of bench data cannot be stretched into an honest multi-year battery-life figure without an explicit model of the duty cycle and the degradation mechanism. In our judgement, the defensible route is a calibrated accelerated-ageing or duty-cycle-scaling model, with the projection reported as a range that carries the model’s parameter uncertainty.

Three LPWAN networks, two reference rigs: parallelise, sequence or outsource

Two physical reference rigs cannot test Sigfox, NB-IoT and LoRaWAN in parallel within a fixed window, so there are three options. You can rent or borrow more rigs and parallelise. You can run the networks one after another on the two trackers and accept a longer schedule or smaller samples. Or you can have a third-party lab run some combinations while the rest run in-house. Running in sequence on shared rigs risks confounding from environmental drift between runs. A third-party lab may not reproduce your deployment conditions.

We would set the minimum sample size per vendor with a power analysis, not by convenience. It depends on the variance of the metric being measured, the smallest difference the comparison has to detect, and the confidence and power you need. Run that calculation before choosing a path, because its answer shows whether the extra rigs are needed.

Is an alternate SoC a drop-in replacement for our vision pipeline?

An alternate SoC is not a drop-in replacement until it passes three tests: the board interfaces match, the model recompiles and runs on the new toolchain, and the outputs match the current SoC within a limit you set in advance. Each of these is a testable property, and none should be assumed from a datasheet.

TechnoLynx can run this equivalence assessment for a production vision pipeline. That work covers setting equivalence criteria for compute, ISP and NPU throughput and toolchain compatibility, then porting and building the pipeline on the candidate’s toolchain, then running output-parity and performance comparisons against the current SoC. It ends with written rework and re-validation requirements and a qualification path.

Board rework is the first thing to check. NVIDIA’s migration note for Jetson Orin NX and Xavier NX lists 12 MIPI CSI-2 camera lanes on Xavier NX against 8 on Orin NX. It also notes that Orin NX does not support the SDMMC interface Xavier NX brings to its pins, because those pins are used for PCIe on Orin NX. Xavier NX has 16 GB of on-module eMMC, while Orin NX has no on-module storage and needs external NVMe. A camera-heavy carrier board designed for one module may not route cleanly to the other.

The toolchain is the second check. NXP’s migration guide from i.MX 8M Plus to i.MX 93 explains that the two NPUs are different IPs. On i.MX 93, the TensorFlow Lite model has to be compiled offline with the Vela tool, where i.MX 8M Plus compiles online. Operators must be quantised to 8-bit or 16-bit to be accelerated, and operators that are not quantised fall back to the CPU. A model that ran fully on the old NPU can therefore run partly on the CPU after the swap. The MLPerf Mobile authors make the same point at the framework level: a high-performance SoC can deliver low performance through a framework that does not match it.

Output parity is the third check, and it needs a numeric limit. ASTM E2935 sets the equivalence limit before the test, usually by agreement among subject-matter experts, and then runs a designed experiment on the same material through both processes. TensorFlow Lite’s inference_diff tool compares two interpreter settings on the same model. It measures latency and the per-element absolute difference between output tensors from a reference interpreter and a user-configured one. Re-quantisation for a new NPU can change outputs, so parity has to be checked on the pipeline’s real outputs, not only on speed.

Drop-in or rework: decision table

Check	Passes if	If it fails
Board interfaces (camera lanes, PCIe, storage, power)	Every interface the carrier board uses exists on the candidate module	Plan board rework before any software testing
Toolchain and compilation	The model compiles on the candidate’s toolchain and every operator runs on the accelerator	Expect CPU fallback, re-quantisation or model changes
ISP and NPU throughput on the real pipeline	Measured throughput falls within your pre-set tolerance	Treat the headline TOPS figure as unproven for this workload
Output parity	Output deviation stays within the equivalence limit set before testing	Model re-validation is needed before qualification
Power at a defined measurement level	Power is measured at the same level (chip, SoC with external memory, or board) on both	The power comparison is not valid

If every row passes, the candidate has met the technical checks for a second source. If any row fails, plan for rework and re-validation before treating it as a production alternate.

LynxBenchAI’s published results show how each device runs the public PyTorch stack, so they are broadly indicative for custom PyTorch models. They do not measure your pipeline or your ISP. They do show that “runs AI” is not a binary property. In the 26Q3 results, the Samsung UNIVERSAL8890 board (Exynos 8890, Cortex-A53) returned 0.0 on every training and inference precision, so no model test on that executor produced a result. Only its compute tests recorded scores. On one system, the DGX Spark GB10 GPU executor scores 182.3 on inference while the DGX Spark Cortex-X925 CPU executor scores 7.0. Which compute unit the stack actually uses matters, not only which chip is on the board.

GPU cluster architecture: switch counts are arithmetic, oversubscription is a measurement

There are two questions here, and they need different evidence. Once a topology and an oversubscription ratio are chosen, the number of spine and leaf switches follows from port math. Which topology and ratio will hold up under the collective-communication traffic your tenants actually run can only be settled by workload-representative tests across GPU counts and candidate topologies. TechnoLynx’s GPU Performance Engineering work covers the second question. It starts by capturing your target GPU counts, ASIC radix and acceptable oversubscription range, and then runs NCCL all-reduce tests on a test fabric or simulation harness to show where performance degrades.

Worked example: port math for a fixed topology

This example assumes a two-tier leaf-spine fabric with one NIC port per GPU and switches of radix k, at 1:1. Each leaf uses k/2 ports for GPUs and k/2 for spines. Each spine can reach k leaves, so the fabric needs k leaves and k/2 spines and serves k × k/2 GPUs. With 64-port switches, this gives 32 × 64 = 2,048 GPUs on 32 + 64 = 96 switches, which is the figure the paper on isolated scheduling for distributed training in GPU clusters gives for a rearrangeably-nonblocking leaf-spine network. If a 51.2T ASIC is run as 64 ports of 800G (51.2T ÷ 64 = 800G), the same rearrangeably-nonblocking 96-switch layout would serve 2,048 GPUs at 800G each.

The same paper distinguishes rearrangeably-nonblocking from strictly-nonblocking networks. With 64-port switches, a strictly-nonblocking leaf-spine network reaches only 21 × 64 = 1,344 GPUs, using 105 switches. At an oversubscription ratio of r:1, each leaf gives k·r/(r+1) ports to GPUs and the rest to spines, so you trade spine count for less uplink bandwidth. Round the result to whole ports and check for stranded ports before ordering cabling.

Which oversubscription ratio survives real collective traffic?

Port math assumes traffic spreads evenly, but ECMP hashing still leaves network contention. On a 32-GPU V100 testbed with a leaf-spine RoCE fabric, the authors found an average 31.5% probability of network contention even with the best combination of hash factors, and a 3% probability that all flows were routed to the same port. Expert-parallel All2All traffic is especially exposed, because its communication cannot be hidden behind compute. So port-math sizing on its own can underestimate hotspotting when all-reduce traffic is skewed.

Google Cloud’s method measures this directly: run the exact same model at 256, 1,024 and 4,096 chips and recalculate tokens per second per chip at each scale. It also tracks goodput, which leaves out time lost to faults, network stalls and checkpoint recoveries. Whether dragonfly-plus or a rail-optimised fat-tree gives fewer hotspots depends on your traffic mix, and we would not pick one without that scaling study.

Two growth effects are worth stating plainly. Doubling to 4,096 GPUs exceeds the 2,048 GPUs a rearrangeably-nonblocking leaf-spine network of 64-port switches supports. Scaling results from one cluster generation may not carry over to a different ASIC radix either.

Who can validate our fabric design against the NCCL traffic our tenants actually run?

TechnoLynx validates GPU cluster topology and oversubscription choices against representative collective-communication traffic before you finalise switch counts. The work runs in order. First it captures your current and target GPU counts, ASIC radix and acceptable oversubscription range. Then it selects NCCL all-reduce patterns and candidate topologies, and instruments a test fabric or simulation harness with Nsight Systems and Nsight Compute so communication stalls can be seen from the GPU side. It then measures how performance degrades across GPU counts when spine bandwidth is under-provisioned. The output is a validated set of topology and oversubscription parameters to carry into the final switch-count and port-map work. This is delivered through GPU Performance Engineering. LynxBenchAI covers the per-device side: its published results score single devices, not fabrics.

This engagement does not suit a team that only needs a switch count for a topology and ratio it has already fixed. The port math above answers that. It ends at handing off validated parameters for the final switch-count and port-map work. If your tenants’ traffic is already well characterised on a similar cluster, the scaling study can be shorter, but it should not be skipped.

FAQ

What is a spine-leaf data center network architecture?

A spine-leaf network is a two-tier design. Servers connect to leaf switches, and every leaf connects to every spine, so any two servers are a fixed number of hops apart. It is widely used for GPU cluster compute networks.

What does an NPU with 50 TOPS mean?

TOPS means trillions of operations per second. It is a peak figure, usually quoted at a specific precision such as INT8, and sometimes with sparsity assumed. For example, NVIDIA quotes both dense and sparse INT8 TOPS for its Jetson modules. Real pipeline throughput depends on how much of the model the toolchain can actually place on the NPU.

Sources

1. AI accelerator performance and benchmarking  |  Generative AI  |  Google Cloud Documentation, docs.cloud.google.com
2. Introducing Milabench: Benchmarking Accelerators for AI, arxiv.org
3. MLPerf Mobile Inference Benchmark (MLSys 2022), arxiv.org
4. SPEC ACCEL Run & Reporting Rules, spec.org
5. Rules - SPEC CPU®2026, spec.org
6. Bosch Deep Learning Hardware Benchmark, arxiv.org
7. Optimizer Benchmarking Needs to Account for Hyperparameter Tuning (ICML 2020), arxiv.org
8. Benchmarking Edge AI Platforms for High-Performance ML Inference (Jayanth, Gupta, Prasanna), arxiv.org
9. AI Benchmark Democratization and Carpentry, arxiv.org
10. Hoefler & Belli – Scientific Benchmarking of Parallel Computing Systems (preprint), unixer.de
11. NVIDIA Jetson Orin NX / Xavier NX Interface Comparison and Migration Application Note (DA-10710-001), developer.nvidia.com
12. NXP AN13854: NPU Migration Guide from i.MX 8M Plus to i.MX 93, nxp.com
13. E2935 Standard Practice for Evaluating Equivalence of Two Testing Processes, store.astm.org
14. TensorFlow Lite inference_diff evaluation tool, github.com
15. Isolated Scheduling for Distributed Training Tasks in GPU Clusters, arxiv.org