You can only compare an AI benchmark number across devices if the entry tells you how it was produced: the precision, the batch size, the exact model checkpoint and sequence length, the framework, runtime, CUDA and driver versions, and the clock and platform conditions. Each device’s workload also has to be scaled to that device’s own saturation point instead of fixed at one size for everyone. If any of that is missing, two numbers side by side on a chart are really two different experiments.

LynxBenchAI applies that rule in its own method. Every device in the published LynxBenchAI results runs the same model catalogue at the same precisions, and only the batch size changes, raised until throughput levels off. LynxBenchAI also keeps the driver, framework and runtime versions each result ran on in its submission record.

What every AI benchmark entry must disclose before its number counts

Before a number goes on a comparison chart, the entry has to state the precision mode, batch size, model checkpoint, input shape or sequence length, the whole software stack, and the conditions it ran under. If it doesn’t, the comparison is unverified, not fair.

If your own model needs measuring, TechnoLynx can run that PyTorch workload on your shortlisted devices with matched, disclosed precision and batch settings. Each result comes with a methodology record covering the stack, checkpoint and sequence length.

LynxBenchAI fixes the model and the precision in its methodology and adapts only the batch size. Its published results show how each device runs the public PyTorch stack your own models run on. That makes them a broad guide for custom PyTorch models too, but they don’t measure your own workload or application.

Minimum disclosure fields for a comparable entry

Field	Why it changes the number	Who already requires it
Precision mode, including TF32 and reduced-precision accumulation settings	On Ampere, FP32 convolutions and matrix multiplications can run in TF32, and PyTorch uses TF32 for cuDNN by default	MLPerf requires the quantisation method to be described publicly in enough detail to reproduce it
Batch size	Tensor Cores are activated when batch and layer dimensions are multiples of 8 (FP16) or 16 (INT8)	MLPerf lets submitters choose batch sizes per scenario
Model, checkpoint and pre/post-processing	Undisclosed preprocessing can produce “silent errors”: the evaluation runs but some predictions come out wrong	MLPerf’s closed division requires the same models, datasets and quality targets
Input shapes and sequence length	TensorRT engines with dynamic shapes are benchmarked at one specific shape inside a configured range	trtexec takes the shapes explicitly through its --shapes flag
Framework, runtime and library versions	Frameworks built on the same cuDNN stack still showed very different AlexNet performance	MLPerf Available submissions must give the software version of every component, plus stacks and containers
Driver and CUDA version	NVIDIA’s TensorRT docs say not to use output whose GPU, driver, CUDA version and clock state weren’t recorded as a baseline	SPEC ACCEL lists device driver names and versions as required fields
Clock, power cap and thermal state	Two runs of one engine on one GPU can differ by tens of percent if the clock floats in one and is locked in the other	trtexec guidance is to capture nvidia-smi -q before every benchmark run and a live trace during it
Platform and driver mode	On Windows, WDDM mode tends to give worse and less stable inference results than TCC mode	SPEC requires changes to daemons or system tuning to be documented in the disclosure
Run protocol: warm-up, duration, repeats, summary statistic	trtexec’s defaults set a minimum warm-up and run length, and both can be changed	Hoefler and Belli (SC15) ask authors to report every measurement, synchronisation and summarisation technique

These requirements already exist in practice. SPEC’s ACCEL run rules say a published result must describe all performance-relevant conditions, and a full disclosure must give enough information for someone else to reproduce the result independently. SPEC’s power methodology adds that the system description has to be detailed enough for another party to replicate the configuration, including any tuning done to reach the result.

The usual gap is the software side. When Hoefler and Belli (SC15) surveyed published papers, they found that most described the hardware but not the software environment. They also pointed out that naming a well-known system isn’t a description, because routine software upgrades on that system change what you’d measure.

A release tag does not guarantee the same conditions for every entry

A release name tells you which results are meant to be compared with each other. It doesn’t tell you every entry in that release ran under identical conditions, so check each entry. The 26Q3 results include native Linux, WSL and macOS platforms in one table. In 26Q3, the RTX 5090 on native Linux scores 985.8 for inference and the RTX 5090 under WSL scores 891.8. Both entries are the GeForce RTX 5090. The platform isn’t: one ran on native Linux, the other under WSL. LynxBenchAI scores are meant to be compared within the 26Q3 release only.

Vendor numbers and quantisation sign-off

The same check applies when a number comes from a vendor. It is the first step in validating vendor and hardware claims before you buy. For quantisation sign-off, TechnoLynx builds a reproducible A/B harness comparing full-precision and quantised outputs on frozen scenarios with defined pass/fail thresholds. A pooled average can hide one segment that fails parity.

Fixed workloads favour one device class; scaling to saturation does not

A fixed, one-size-fits-all workload introduces systematic bias across device classes. Batch size is capped by GPU memory, so the largest usable batch differs from device to device. Any one fixed size favours whichever device it happens to suit. The MLPerf Inference paper notes that most inference systems need a minimum, architecture-specific batch size before they use their compute fully. TorchBench tested inference batch sizes by doubling from one to find the value with the highest GPU utilisation. An MLPerf-based edge study adjusted the offline batch size for each system to avoid running out of memory and to maximise throughput.

LynxBenchAI holds the model architecture and numerical precision constant on every device and scales only the batch size, raising it until throughput levels off within a defined noise band, so a laptop GPU and a data-centre accelerator are each measured at their own saturation point. MLPerf leaves the batch choice to each submitter. LynxBenchAI makes it part of the method itself, so no device gets a hand-picked operating point. For a LynxBenchAI result, the batch size is set by the saturation rule, not one shared number.

Fair comparison also requires bounded optimisation effort. LynxBenchAI prepares the benchmarked models once, before any target device is known, and ships the same prepared artefacts to every vendor’s hardware. Where two vendors’ kernels disagree on what is permissible, the stricter rule applies to all of them. A workload that fails or times out is recorded as zero, not dropped. In the 26Q3 results, the Samsung Exynos 8890 board records 0.0 on every training and inference test, and the Apple M1 records 0.0 for DistilGPT2 training at both FP32 and BF16. A 0.0 means that precision produced no result on that executor.

Does the same precision on an RTX 3090 and an RTX 4090 give comparable numbers?

Not automatically. Two cards with the same precision label can run in different numeric modes depending on architecture and framework defaults, so the label alone doesn’t make the comparison like-for-like. LynxBenchAI compares cards on the public mainstream stack as distributed, with optimisation bounded for all. If the paths differ in maturity, report the result as indicative.

The mechanism is concrete. On Ampere, TF32 mode converts the inputs of convolutions and matrix multiplications from FP32 just before the multiply. So two FP32 results can be two different numeric modes, depending on architecture and framework defaults. PyTorch’s documentation says some recent GPU architectures truncate intermediate accumulations to reduced precision for speed, and that this can be switched off. Tensor Cores only engage when dimensions hit the required multiples, so at a batch size off those multiples the fast path isn’t used. Per-layer profiling shows this. trtexec’s --dumpLayerInfo and --dumpProfile print each layer’s data types, tactic names and runtime, and Nsight Systems shows why a layer is slow at the kernel and stream level.

In the 26Q3 results, the RTX 3090 produced no FP8 inference result on any model (0.0), while the RTX 4090 scored 778.0 at FP8 on Flan-T5 Small. In the 26Q3 results, INT8 isn’t automatically the faster option either. The RTX 4090 scored 1333.4 at FP16 and 650.2 at INT8 on Flan-T5 Small inference, and the RTX 3090 scored 625.5 and 402.0. Read those figures as what the public PyTorch stack does at each precision on each device, not as the limit of the silicon. An MLPerf-based study of quantisation on edge processors found that without instruction-set support or library optimisation, quantisation can actually hurt inference performance.

LynxBenchAI measures publicly available software that anyone can obtain. A submitter may use a newer driver or an optimised PyTorch build, as long as every component is a public release, so a score describes how the mainstream stack performs as shipped.

Pinning the driver, CUDA and framework versions between runs

Repeat runs on one machine stay comparable when the driver, CUDA, kernel and framework versions are fixed and logged with every run. PyTorch’s documentation says fully reproducible results are not guaranteed across releases, individual commits or platforms. Even on one machine, cuDNN’s benchmarking mode may pick different convolution algorithms on later runs. NVIDIA’s TensorRT docs say a timing cache should only be reused on matching CUDA, TensorRT, GPU model and clock frequency.

A procedure that holds up over a review period:

1. Before the first run, capture the GPU’s product name, power cap and clock settings with nvidia-smi -q.
2. With every run, record every software component’s version and trace clocks, power and temperature with nvidia-smi dmon.
3. On Windows, keep drivers out of updates with the Group Policy “Do not include drivers with Windows Updates”, or turn off Automatic Updates through the NoAutoUpdate registry value.
4. Lock the GPU clock for deterministic comparisons, and let it float only when you’re measuring peak behaviour.
5. If any version changes, re-run the baseline instead of comparing across mismatched stacks.

Pinning has limits. It doesn’t remove variance from power throttling, which on low-power-limit parts such as the T4 and A2 makes short-window measurements unstable. Holding updates for weeks also leaves security patches unapplied, which is a trade-off to accept knowingly. In LynxBenchAI, the submission record carries the driver, framework and runtime versions each result ran on.

Median, range or one number: what should a benchmark table publish?

Don’t average results across mixed driver or backend submissions. Summarise repeated runs of one configuration with a confidence interval or a measure of variance, not an isolated number.

LynxBenchAI publishes one score per test per executor, with model and precision fixed across devices.

Several methodology sources recommend showing variation. Kalibera and Jones (ISMM 2013) recommend always giving confidence intervals for results. Hoefler and Belli (SC15) ask authors to report confidence intervals for nondeterministic measurements. The lm-eval authors note that reporting variance over several runs is seldom done. The AIPerf documentation suggests five runs as a starting point and treats a coefficient of variation under 10% as trustworthy. Established suites build this in: SPEC ACCEL reports the median of three runs per benchmark, and TorchBench runs each model ten times and reports the run with the median execution time.

Why LynxBenchAI publishes one value per result

Each LynxBenchAI test is one timed window after a discarded warm-up, not a median of repeats. The batch for that window is set by raising it until throughput plateaus inside a defined noise band. A single window doesn’t show run-to-run variance, so small gaps between neighbouring devices aren’t evidence of a difference. In 26Q3, the RTX 3080 scores 240.5 for inference and the RTX 5060 Ti scores 240.4. If your decision depends on a margin that small, run the measurement several times on your own hardware and look at the spread.

Range and median reporting has costs. It takes more space, casual readers find it harder to scan, and any normalisation choice builds in assumptions about which configurations count as comparable.

What happens to a submission that leaves configuration fields blank?

The MLPerf Inference rules say plainly that results which cannot be replicated are not valid. SPEC ACCEL requires any public result that doesn’t follow its run rules to be clearly marked as an estimate, and any SPEC member may ask for the raw files behind an unmarked public result. Audits find real problems. MLPerf’s first inference round received over 600 submissions and cleared 595 as valid, and its review found about 40 issues in roughly 180 closed-division results.

For a community table, use the fields in the disclosure table above as a gate for entry. A row that leaves a field blank is shown as unverified, or left out, not averaged in with complete rows. Contributors can still misreport fields, so a template reduces the problem without removing it.

LynxBenchAI sets or records most of these fields itself. Model, precision and batch policy are set by the methodology, and the submission record captures the driver, framework and runtime versions. The public-software rule carries a stated limitation. It is declared, not technically enforced: the client doesn’t check that a submitted stack is publicly available, so the rule can be audited after the fact but not at run time.

Who can measure our shortlisted GPUs under one disclosed configuration?

LynxBenchAI measures each covered device under one methodology: a fixed model catalogue, fixed precisions, batch scaled to saturation, and the stack versions recorded with every result. The 26Q3 results cover 48 devices across training, inference and compute, using MobileNetV3, Flan-T5 Small, DistilGPT2, DistilBERT and GraphGPS models plus dense and sparse matrix-multiply tests. If your shortlist is covered, the device pages on the LynxBenchAI leaderboard give the figures. For your own machine or an uncovered device, LynxBenchAI benchmarks your own hardware with stock PyTorch.

There are limits. LynxBenchAI shows how devices run the public PyTorch stack. It doesn’t run your model, your preprocessing or your serving path. If the decision depends on your own workload, the options are a TechnoLynx engagement that measures it under matched, disclosed settings, or your own harness. For TensorRT deployments, trtexec plus the environment recording NVIDIA documents is a sound base for that harness.