A benchmark result is evidence a purchase committee or an auditor can rely on only when it describes a configuration someone else could rebuild and rerun: the exact device, its driver and firmware, the framework and runtime versions, precision, batch size, the workload, who ran it and how many times. Without that, a number is a claim, however reputable the source. MLPerf and SPEC already hold their own results to this standard: the MLPerf Inference rules state that results which cannot be replicated are not valid, and the SPEC CPU 2026 run rules require public results to disclose enough to be reproduced independently. The discipline belongs to the method rather than to any one artefact, which is why the same reasoning carries from GPU procurement to camera-based pose tracking.
A score describes a stack, not a chip
What gets measured is one executor, meaning a device plus its software stack, so every LynxBenchAI score covers the framework, kernels, driver and runtime as well as the chip. That matters because the software layer moves results a long way. A 2026 cross-layer study of LLM inference on Qualcomm Snapdragon phones found that frameworks on the same hardware, using the same kernels, can differ by about 2x, and that custom-kernel paths can fall more than 10x behind vendor-optimised stacks in prefill. In that study, two frameworks that both route through Qualcomm’s QNN stack differed by nearly 2x because one ran the full graph on the NPU and the other kept self-attention on the CPU.
The MLPerf Mobile authors (MLSys 2022) report over 10% between delegates in their own table, and cite earlier work showing up to 7x between delegates and vendor SDKs on some chipsets. Output quality moves too. A 2025 study found that changing evaluation batch size, GPU count and GPU type produced up to 9% variation in accuracy for a reasoning model under bfloat16 with greedy decoding, and traced the cause to non-associative floating-point arithmetic.
Our own release shows the software environment moving scores too. In the 26Q3 results, the GeForce RTX 5090 on native Linux scores 985.8 for inference, while the same card under WSL scores 891.8. The RTX 3080 on native Linux scores 240.5 for inference and the RTX 3080 under WSL scores 231.7. The release lists each pair as separate entries with different scores.
How do you confirm a run used the public stock driver and runtime, not a vendor-tuned build?
You confirm it after the run, from the record, because nothing that happens at run time proves it. LynxBenchAI measures software anyone can obtain: every component must be a version released to the public. A build made for LynxBenchAI, or tuned against it and shipped to nobody else, falls outside the methodology, so a score describes the mainstream stack as distributed.
The rule is declared, not technically enforced. The client does not check that a submitted stack can be obtained publicly; it records the driver, framework and runtime versions each result ran on, which makes the rule auditable after the fact instead of at run time. In practice, you check each recorded version against the vendor’s public release history, and any version you cannot find there is the thing to question.
MLPerf and SPEC set their own availability rules for the same problem. The MLPerf Inference rules make an audit more likely when the hardware and software are not reasonably available to the general public, and they require “Available” submissions to specify the software version of every component, the hardware configuration, the software stacks and the containers. SPEC CPU 2026 requires components that ordinary customers can obtain, with production versions generally available within three months of the result’s first public release. LynxBenchAI requires every component to be a publicly released version and does not enforce this at run time; the version record makes it auditable afterwards. What a version record lets a second party do with a score is covered in interpreting and reproducing workload benchmark results.
Can a third party benchmark our workload on every candidate platform and defend it to auditors?
Yes, provided the harness runs your representative workload the same way on every candidate, pins and records each platform’s driver and SDK versions, and is built so someone else can rerun it after the next firmware update.
LynxBenchAI covers the measurement side of that question: its published results show how each covered device runs the public PyTorch stack your own models run on, so they are broadly indicative for custom PyTorch models too. LynxBenchAI does not run or measure your workload, so a decision that depends on your exact model needs a dedicated run, and the sensible first step is reading the LynxBenchAI results for the devices on your shortlist.
Rerunning a benchmark needs an artefact rather than a description: code, prompts, datasets and instructions. A 2025 study of LLM runtimes on Apple Silicon is a good example of the bar: it publishes commit hashes, OS and toolchain versions, exact commands, environment files and instructions for regenerating every figure from raw logs. The same authors warn that minor updates can shift kernel behaviour and driver-level memory accounting, which is why they pin every commit. Bench360 (2025) puts the whole configuration in one YAML file and seeds every random source with a fixed global seed. MLPerf puts the rules on query arrival, timing and accuracy into LoadGen, outside submitter code, which its authors say makes compliance and auditing easier.
How do we stop a cross-vendor comparison turning into a software-maturity contest?
There are three honest ways to normalise across TensorRT, the Qualcomm AI stack, the Hailo compiler and their peers, and each one answers a different question. The first is a common public stack wherever every platform supports one, which is LynxBenchAI’s choice: it measures out-of-the-box behaviour. The second lets each vendor use its own toolchain under fixed task semantics. The MLPerf closed division requires a model equivalent to the reference, allows calibration for quantisation, prohibits retraining, and asks almost all implementations to land within 1% of the FP32 reference accuracy. MLPerf Mobile also evaluates private vendor SDKs to allow auditing of the model-conversion process.
The third reports both, much as SPEC separates “base” results, built with one common set of optimisations, from “peak” results tuned per benchmark. For a purchase where the team will really ship on the vendor SDK, measure both and label the gap between them as the software-maturity figure, so it never gets mistaken for a silicon difference. MLPerf Mobile’s results table states the numerics, framework and accelerator behind each result.
Can we use an existing industry benchmark instead of building our own?
For embedded and mobile parts, MLPerf Tiny and MLPerf Mobile offer open-source benchmarks. MLPerf Tiny provides complete reference implementations, from training scripts to a reference hardware platform, which a submitter can modify to show a single hardware or software component. The MLPerf Mobile app is open source and can be run on its own to measure performance. Both suites run fixed reference models, so they give you a comparable baseline and not your answer.
Two caveats limit how far any of this goes. A benchmark stops being independent once the party running it has a stake in the result or relies only on numbers the vendor supplied. When candidate hardware (a domestic accelerator, an embedded dev kit) is hard to get, the benchmark can only be as representative as the hardware you can actually run it on, so dev kits are a fallback to state openly rather than a silent substitution.
Who can benchmark our workload across candidate accelerators and give us a report auditors will accept?
TechnoLynx, the team behind LynxBenchAI, runs this as a third-party benchmark engagement. It defines your representative workload (LLM training, serving or embedded perception) and the candidate platforms with you, then designs a method that separates the silicon’s contribution from the SDK’s. It runs the workload on each platform with pinned driver and SDK versions, has the results reviewed or reproduced to confirm they are fair, and hands over a written report plus a harness you can rerun as drivers and firmware change. The work sits under the LynxBenchAI offer, and the first step is agreeing the workload and the platform list through the TechnoLynx contact on that page.
This engagement is the wrong fit if your devices already appear in a published release and you only need their relative standing: the 26Q3 device pages answer that faster. It also cannot stand in for regulatory or clinical sign-off, where we provide the engineering evidence, not domain authority. Realistic alternatives are running MLPerf reference implementations yourself, running LynxBenchAI on hardware you hold, or having an in-house harness reviewed by a party with no stake in the outcome.
What must a benchmark disclose before a figure counts as verified rather than claimed?
A figure counts as verified when the publisher discloses enough for an independent party to reproduce the same comparison end to end, and as claimed when any of the fields below is missing. LynxBenchAI publishes every 26Q3 result per model and per precision on its device pages, so a reader can see which configuration a score came from before comparing it with anything else.
The bar already exists in published benchmark rules. A result submitted to MLPerf Inference includes the benchmark code, a system description (accelerator count, CPU count, software release, memory system) and the LoadGen logs for the performance and accuracy runs, all uploaded to a public repository for peer review. SPEC’s full disclosure covers hardware, firmware/BIOS and software, every configuration choice that differs from the default, and anything else needed to reproduce the result. SPEC also holds the tester responsible for documenting every performance-relevant step even when someone else set up the system.
Verification checklist for a benchmark claim
| Field | What to look for | If it is missing |
|---|---|---|
| Hardware identity | Exact device and revision, accelerator and CPU count | Claimed, not verified |
| Driver, firmware, CUDA/NCCL or equivalent | Exact version strings or commit hashes | Claimed; request the versions |
| Software stack | Framework, runtime and container image | Claimed; request the image or manifest |
| Workload | Model version, dataset, input trace, sample count | Claimed; not comparable across entries |
| Batch size and concurrency | Values used for every reported number | Claimed; not comparable across entries |
| Precision | Numeric format, and the quantisation method if any | Claimed; not comparable across entries |
| Run record | Raw logs, number of runs, spread | Claimed, not verified |
| Operator | Who ran it, and whether they have a stake in the result | Claimed; request who ran it |
| Deviations | Every departure from the reference setup | Treat as an estimate |
The deviations row follows existing practice: MLPerf’s open division requires every submission to document how it departs from the closed division, and SPEC requires non-compliant results to be clearly marked as estimates. More on these fields is in full configuration disclosure.
Were all these GPUs tested at the same batch size, precision and trace?
Two entries are comparable only if batch size, precision and workload trace match; otherwise the comparison is apples to oranges, however tidy the chart looks. In the 26Q3 results, the A100-SXM4-80GB and the GeForce RTX 3090 show 0.0 for every fp8 inference test, which means that precision produced no result on those executors, while the H100 80GB HBM3 scores 1362.7 for fp8 DistilBERT inference. An fp8 comparison between those devices simply does not exist, and an honest chart says so instead of substituting another precision.
Full disclosure has limits. It makes reproduction possible, not certain. Run conditions matter too, which is why MLPerf Mobile prescribes a room temperature of 20 to 25 °C and cooldown breaks between tests to limit thermal throttling. MLPerf-style rules fit standardised benchmarks cleanly, so a custom internal workload needs a bespoke bar of the same kind.
Which number is throughput and which is latency, and do vLLM and llama.cpp define TTFT the same way?
Throughput counts tokens (or requests) per unit of time, and latency is the time a single request waits; time to first token (TTFT) and time per output token (TPOT) are both latencies. Tools do not agree on the formulas behind those names, and the vLLM benchmark documentation says outright that metric terminology is not standardised, so results should be compared by measurement point and formula rather than by metric name.
| Metric | One published definition | Where definitions diverge |
|---|---|---|
| TTFT | vLLM: from the request’s arrival at the frontend, which currently starts when tokenisation begins | An Apple Silicon study (2025) measures TTFT client-side and excludes tokenisation unless noted |
| TPOT | vLLM: (end-to-end latency − TTFT) / (output tokens − 1) | A prefill/decode study divides decode time by the output length, not by output length − 1 |
| Inter-token latency | vLLM: one sample per streamed output event, recorded separately from per-request TPOT | vLLM records TPOT as zero for requests of one token or fewer, while vllm bench serve excludes them |
| Throughput | Decode throughput (tokens after the first, divided by decode time) or end-to-end throughput (tokens divided by prefill plus decode) | Prefill throughput counts input tokens, decode throughput counts output tokens |
Client-side TTFT in vLLM is roughly server queue time plus server TTFT plus frontend and network overhead (tokenisation, chat templates, detokenisation, transport). A TTFT measured at the client and one measured at the server can therefore both be correct and still disagree. The same check applies to llama.cpp output: find the formula and the measurement point before putting its numbers beside vLLM’s. Two controls matter on reruns. Repeating vllm bench serve against the same server can reuse prompts left in the prefix cache and inflate throughput, so vary the seed or reset the server between runs. MLPerf Inference LLM submissions report only aggregate throughput within TTFT and TPOT bounds (for Llama 2 70B, TTFT ≤ 2000 ms and TPOT ≤ 200 ms at the 99th percentile), so the trade-off between prefill and decode is not visible in them.
What exact versions produced this vendor’s latency numbers, and will the vendor let us rerun them on our own rig?
Cite a vendor’s throughput or latency figure in appraisal documentation only once it discloses hardware revision, firmware and driver versions, precision mode, batch size and who ran the test; if it does not, re-validate the figure on your own rig before using it.
LynxBenchAI records the driver, framework and runtime versions behind every result it publishes, so a vendor’s figure can be set beside a disclosed configuration on the same device, and the first step is reading the LynxBenchAI results for the devices on your shortlist.
TechnoLynx does that re-validation under the LynxBenchAI offer: it fixes the required disclosure fields against your workload’s characteristics, builds a versioned harness on your own test rig with matching batch sizes and precision modes, reproduces the vendor’s figure, quantifies any gap and documents every configuration field for audit-grade sign-off, and the first step is agreeing the workload and the platform list.
Independent reproduction is already part of published benchmark practice. MLPerf Mobile requires the system under test to be commercially available, collects the benchmark app’s log files unedited, and has results independently audited by rebuilding the vendor’s app on a factory-reset device; a result stands if the auditor’s numbers fall within 5% of the submitted ones. An MLPerf Inference audit asks the submitter for two days of hardware access, one for a pre-agreed list of tests and one for follow-up tests.
A vendor that will not disclose is giving you information too. Its figure stays in the “claimed” column, and the evidence for the appraisal has to come from your own rig. Even a fully disclosed number may not transfer when the vendor’s workload differs from yours, so read every comparison next to its disclosure fields. How to package those fields so competing vendors and reviewers can check them is the subject of building trustworthy, reproducible cross-vendor benchmark disclosures.
How many repeated runs does a credible benchmark need, and should it report confidence intervals?
No single run count is universally right, but a credible result reports more than one run, a measure of spread and, where differences are small, a confidence interval. The established conventions differ in detail. SPEC CPU 2026 runs each benchmark three times and takes the median, or twice and takes the slower run. MLPerf Tiny reports the median of five runs. The MLPerf Inference paper describes five runs for the server scenario, taking the minimum. The Apple Silicon study ran 10 trials per point and reported mean ± standard deviation.
Benchmarking research explains why the reporting matters. Georges, Buytaert and Eeckhout (OOPSLA 2007) found that common practices, such as reporting the best run, made too strong a claim in up to 16% of comparisons, and that mean and median were consistently more reliable than best, second-best or worst. Kalibera and Jones report that 71 of the papers they surveyed gave no measure of variation for their results. For differences inside the margin of experimental error, skipping statistical rigour can lead to the wrong conclusion.
A workable procedure is to fix the seed and the request ordering, run it repeatedly, as the conventions above do, and compute the coefficient of variation and a confidence interval. NVIDIA’s AIPerf documentation treats a coefficient of variation under 10% as trustworthy and reads confidence intervals that do not overlap as strong evidence of a real difference. Keep adding runs until the interval reaches the width you need, which is what Georges et al.’s JavaStats tool automates. A single run should carry an explicit “n=1, no CI” label so nobody reads a zero-width interval as precision.
Apply the same reading to our own release. Each 26Q3 test is one timed window after a discarded warm-up, not a median of repeats, so small gaps between neighbouring devices should be read with that in mind. Compare scores within the 26Q3 release only.
Is there a published benchmark for driver-camera occlusion we can anchor our model choice to?
Before anchoring a model choice to a public occlusion benchmark, check its camera geometry and occlusion pattern against your rig, and treat it as directional until then. LynxBenchAI does not score keypoint accuracy or occlusion robustness; for this decision its published results answer only the hardware half, namely how a device runs the chosen model family on the public PyTorch stack.
If a candidate benchmark does exist, put it through the same checklist as a hardware claim: camera position, occlusion types, dataset version and metric definitions. Supplier results that leave those out tell you little, however relevant the supplier. The benchmark’s date matters as much as its setup, and in-cabin validation on your own footage remains the final check.
Can an occlusion benchmark built for driver monitoring be reused for outdoor sports pose tracking?
Reuse the methodology, not the artefact: occlusion and blur injection, keypoint error metrics, confidence-under-occlusion scoring, fixed inputs and disclosed configuration all carry over, while the driver-cabin assumptions about lighting, motion and camera geometry do not. TechnoLynx builds and validates pose-estimation pipelines of this kind through its Computer Vision R&D offer.
Before trusting absolute robustness numbers in the new domain, run a small transfer check: apply the adapted protocol to a representative sample of real match or training footage and confirm that the metrics behave as expected. Cabin benchmarks assume controlled lighting and limited motion blur, so their thresholds and metric weights may need re-weighting for outdoor athletes. The check is only as good as the footage, which needs to cover real conditions and not just clean clips.
Preprocessing follows the same logic. Set motion deblurring and exposure normalisation from the blur and lighting variance you measure in the athlete footage rather than copying the driver-cabin pipeline. Tune deblurring carefully, because applying too much can introduce artefacts that hurt keypoint accuracy.
How far should public MLPerf numbers be expected to differ from our own workload on our own hardware?
The distance depends on how closely your model, precision, batch and concurrency pattern, serving scenario and software stack match the submission. MLPerf lets submitters optimise the reference models and run them through their preferred toolchain. Its authors also show that performance varies drastically across its four scenarios (single-stream, multistream, server, offline). Bench360 (2025) found the best inference engine to be highly scenario-dependent and concluded there is no single best setup for local inference. We read an MLPerf result as what an optimised stack achieved on that configuration, and nothing more.
LynxBenchAI scores every covered device on the same models and precisions in each release, so the 26Q3 device pages show how much a device’s standing depends on the architecture tested. For hardware the release does not cover, LynxBenchAI benchmarks AI performance on your own hardware with stock PyTorch; it measures the device, not your model.
Will the same hardware still suit us if the next model is a transformer instead of a CNN?
Not necessarily, and the 26Q3 results show why. The H100 80GB HBM3 scores higher than the Instinct MI300X on fp32 MobileNetV3 training (564.1 against 275.4), while the MI300X scores higher on fp32 DistilGPT2 training (1571.3 against 1159.0). On fp16 inference the H100 scores higher on both MobileNetV3 (2833.0 against 733.8) and DistilGPT2 (3797.7 against 3318.5). The order depends on the model and the category, so a hardware choice made on one architecture needs checking against the architecture you expect to run next.
Weighting a mixed production load in one suite
For a mixed load (an H.264, HEVC and AV1 ladder plus upscaling and scene classification, say), weight each stage by its share of production GPU time, not by how easy it is to script. Live-event concurrency spikes behave differently from steady VOD batch transcoding, much as MLPerf separates server and offline scenarios, so keep them as separate scenarios and report each one rather than folding them into a single number. Per-title encoding optimisation belongs in a workload of its own when it runs as a separate job with its own resource profile.
Defining a fleet-representative workload instead of relying on vendor submissions
A standard workload definition must pin exact artefacts, because, as the MLPerf authors note, a name such as ResNet-50 does not uniquely or portably describe a model. Record the model weights, dataset version, input trace, precision and batch settings in the same versioned configuration as the harness so that a representative subset can be rerun across the fleet later.
Metrics that follow customer throughput rather than marketing TOPS
Prefer metrics measured on real models over peak operation counts, and report dense and sparse figures as separately labelled numbers, never with a sparse figure inside a dense headline. The 26Q3 results report Dense MatMul and Sparse MatMul as separate compute scores, alongside per-model training and inference scores. The 26Q3 table carries no power figures, so TOPS-per-watt comparisons need a separately disclosed power measurement.
Disclosure is what lets anyone check a number, from a GPU tender to an occlusion test for a pose model. The question that remains is whether anyone will actually do the checking. A version record, a rerun count and a confidence interval are the minimum worth publishing.
FAQ
Do you always use 1.96 for a 95% confidence interval?
No. 1.96 is the normal-distribution value, suitable when you have roughly 30 or more measurements. With fewer, Georges et al. (2007) switch to the Student’s t distribution, and for five runs NVIDIA’s AIPerf documentation uses a t critical value of about 2.776.
What happens to a 95% confidence interval when you add more runs?
The interval generally narrows as the number of measurements grows. How quickly depends on the workload: Georges et al. saw some configurations reach about 1% width in under ten measurements, while others stayed above 3% even after 30.
