A defensible choice of AI compute infrastructure needs evidence bound to your own workload, not to a specification sheet. Datasheet peaks and a vendor’s headline benchmark will not carry it. The method has three steps. Split the workload into its real sub-requirements. Measure each one the way it will actually run: throughput under your latency bound, sustained performance, real utilisation, queue wait, and the task metric that carries the risk. Then document the runs so another party can reproduce them. A budget committee, an auditor or a losing bidder will test that record, and when they do, it is the only part of the decision that holds up.

The same method applies whether the choice is a language model for contract review, a perception model for a safety case, extra nodes for a university GPU cluster or the technical specification in a public tender. What changes from one to the next is which sub-requirement carries the risk, so the order below runs from what to measure, through how to test each kind of need, to the paper trail that defends the result.

What AI compute infrastructure consists of, and what to measure before you buy

AI compute infrastructure is several layers that fail independently: the executors (each a device plus its software stack), node memory, the interconnect fabric, storage and the scheduler that decides who gets the hardware. The tender for CINECA’s LISA system sets a mandatory MPI point-to-point latency below 3 microseconds and a target storage read throughput of at least 9 TB/s. It also requires node power measurement at 95% accuracy and job-level profiling that reports accelerator utilisation for under 5% performance overhead.

We would not test every layer to the same depth before committing. We would measure four things first. The first is what each candidate executor sustains on your model shapes and precisions. The second is how the system behaves under your latency bound. The third is how much of the allocated hardware your jobs actually use, and the fourth is where time is lost in the queue. Synthetic tests such as HPL or a matrix-multiply kernel are well suited to assessing distinct features of a system. The team behind the JUPITER Benchmark Suite (2024) argues that they should be complemented by application benchmarks, because real applications use the hardware dynamically and show its real strengths and limits.

Two related tasks build on these measurements: turning them into a cost-per-request model, and writing a workload specification that fixes batch size, sequence length and precision separately for training and inference.

Why spec sheets mislead when you size AI compute infrastructure

Peak FLOPS, memory bandwidth and theoretical throughput are structurally insufficient predictors of real AI performance. The factors that dominate (memory locality, on-chip data reuse, execution strategy, software behaviour) depend on the workload and cannot be read off a datasheet. The published evidence points the same way from several directions:

• Reddi et al.’s MLPerf Inference paper (2019) found that SSD-ResNet-34 needs 175 times more operations per image than SSD-MobileNet-v1, yet its measured throughput was only 50 to 60 times lower. That is a consistent threefold gap between operation count and observed performance.
• A 2024 study of pre-training on academic hardware found that estimates built from model FLOPs and GPU specification throughput were six times too optimistic against a naive empirical baseline.
• A characterisation of five large GPU clusters presented at SC22 collected over 18,800 hours of data. It found 8% average performance variation (22% at most) between GPUs of identical architecture and SKU, with outliers up to 1.5 times slower than the median GPU.

In the 26Q3 results, a device’s ranking in one category does not carry over to the others. In the 26Q3 results, the AMD Instinct MI300X scores 1023.3 in the compute category (dense and sparse matrix-multiply kernels), against 611.1 for the NVIDIA B200. Yet the B200 scores higher in training (1089.7 against 727.2) and in inference (2224.5 against 1304.0). Consumer cards show the same inversion. The NVIDIA GeForce RTX 4090 scores 367.9 in compute against 279.6 for the RTX 5090, while the RTX 5090 scores higher in training (515.3 against 335.9) and inference (985.8 against 659.4). The software stack shifts results too: in 26Q3 the NVIDIA GeForce RTX 5090 (WSL) scores 470.3 in training, against 515.3 on native Linux. Some precisions produce no result at all on some executors. The NVIDIA A100-SXM4-80GB produced no fp8 result on any inference model that has an fp8 test in 26Q3, each recorded as 0.0.

This gap is why LynxBenchAI exists. It is there to stop systems that look optimal on a specification sheet from being chosen and then failing in real use. Running real software is the reference standard, and estimating from specifications is only a fallback for when that cannot be done. LynxBenchAI’s published results show how each device runs a public PyTorch stack across training, inference and compute tests at several precisions, so they are broadly indicative for custom PyTorch models too. They do not run or measure your own model or application. In 26Q3, the training and inference tests use MobileNetV3, Flan-T5 Small, DistilGPT2, DistilBERT and GraphGPS (Peptides). Each test is one timed window after a discarded warm-up, not a median of repeats, and scores compare only within the 26Q3 release. Use the results to build a shortlist, then test your own workload on the shortlisted executors.

Which measurements should a budget committee, auditor or safety reviewer see before sign-off?

Throughput should be measured under a latency bound, not alone. Latency and throughput are intimately related. Reddi et al. (2019) identify latency-bounded throughput as the measure of inference performance in industrial use cases. A system can deliver excellent throughput and still perform poorly once a latency constraint applies. In MLPerf’s server scenario, queries arrive at random and each must meet a latency bound. That makes large batches harder to form, so throughput falls: for some systems by as little as 3% relative to the offline scenario, for others by 50%. The loss is not even across models on the same system. One system lost about 50% or more on all three models tested, while another lost up to 40% on machine translation but about 10% on the vision models. An accelerator may stand out in one scenario while underperforming in another.

The table below lists the measurements we would put in front of a reviewer, with what each one shows that a headline number hides.

Measurement	What it tells the reviewer	What the headline number hides
Latency-bounded throughput at your bound	Capacity at the service level you will actually run	Server-scenario throughput ran 3% to 50% below offline throughput across systems in MLPerf Inference (Reddi et al., 2019)
Tail latency (90th or 99th percentile) from a statistically sized run	Whether the bound holds for nearly every request	MLPerf needs 270K queries to support a 99th-percentile guarantee on its vision tasks
Sustained performance over a minimum run length	Behaviour once power management and frequency scaling settle	MLPerf requires every run to last at least 60 seconds for this reason
Hardware utilisation of the accelerators you already allocate	Whether you need more hardware or better use of what you have	Near-full allocation coexisted with about 52% hardware utilisation in a production cluster (Jeon et al., 2019)
Queue wait split by cause	Capacity shortfall versus scheduling policy	Fragmentation caused around 80% of waiting time in the same cluster
Task metric matched to the risk (per-clause accuracy, safety-weighted recall, false-positive rate at a fixed true-positive rate)	Whether the model choice is acceptable for its use	Accuracy on complex contract clauses varied far more across models than on standardised ones (ISDA, 2025); high recall can still be the wrong safety trade-off
Measured performance of the delivered units	Whether what arrived matches what was tested	Identical-SKU GPUs varied by 8% on average, with outliers 1.5 times slower than the median (SC22)

How do we split a workload into sub-requirements and test each against our own data?

Workload decomposition means giving each sub-need its own matched evaluation: the same inputs for every candidate, a metric chosen for that sub-need’s risk, and an acceptance threshold agreed before the run. Where the decision is which language model to adopt, TechnoLynx’s LLM Selection Pack is the decision engagement. LynxBenchAI covers the hardware side, showing how candidate executors run public PyTorch models. The authors of LegalBench (2023) make the general point well: any generalised benchmark gives only a preliminary picture, and practitioners deploying a model should collect and validate data specific to their own use case. Two very different examples show what this looks like in practice.

Comparing a 32B open-weight model with a 70B hosted model on 40-page contracts

Run both models on the same set of full-length contracts and score them clause by clause, not document by document. ISDA’s benchmark of LLMs on extracting collateral agreement clauses shows why. Clauses with standardised phrasing, such as base and eligible currency, were extracted accurately by almost all models. Minimum transfer amount clauses varied much more, because models confused them with threshold language. An average over every clause would hide exactly the clause types where the two models differ, so report accuracy separately for each clause type and contract template.

First check that both models can take a whole contract in their context window. ISDA left out models such as Mixtral 8x7B because their context windows could not hold the full documents, and it did not test context-extension strategies. Test separately whether a 32B open-weight model keeps its quality across 40 pages before the accuracy comparison counts. Data handling is one driver of the open-weight option. ISDA included open-source models that can be hosted internally because many firms do not allow confidential contract data off-premises.

We would not use a universal acceptable accuracy delta. Set the threshold before the run, scaled to the cost of a missed or wrong clause at the contract volume you will actually process. Choose the 32B model if its delta against the 70B baseline falls inside that tolerance, and keep the hosted model if it does not. ISDA found hallucination uncommon and misidentified clauses the main failure, but judged human review essential because outputs can still contain incorrect or fabricated details.

Comparing a GNN relational model with a tuned CNN on occluded merges

We would give a safety reviewer recall on the safety-critical cases and the false-positive rate, both on the same occluded-merge validation set. Raw accuracy is not enough. Standard object-detection metrics do not consider whether an object could interfere with the vehicle. One study of safety-aware detection metrics weighted objects by distance, collision trajectory and time to collision, and found that the best detectors by nuScenes ranking were not always preferable on safety. With a 0.5-metre match threshold, 567 of 1,500 safety-weighted rankings differed from the plain average-precision ranking. Risk Ranked Recall (2021) takes a similar approach, grouping objects into three ranks by an objective collision-risk model and reporting recall for each rank.

A single high recall figure does not make a model safe either. A review of out-of-distribution detection for safety assurance notes that even performing correctly 99% of the time does not guarantee safety. For a safety-critical application, few false negatives matter more than a high AUROC, although too many false positives slow the system down and erode operator trust. The same review recommends targets you can check, such as “a false positive rate of X% at a true positive rate of Y%”, instead of “a robust detector”. Partial occlusion is a named operating condition to test robustness against. When comparing performance across datasets, they should match on other semantic attributes and be large enough for statistically significant results.

We would treat a GNN’s occlusion advantage over a per-object baseline as unknown and build your own matched test. Choose the GNN only if it raises safety-critical recall on occluded merges without an unacceptable rise in false positives, and confirm the result in on-vehicle replay testing. Compute constraints belong in the safety requirements as well. The Risk Ranked Recall authors treat compute latency from sensor input to actuation as a property of the vehicle’s hardware and software, and the safety review notes that very few papers provide run-time calculations and fewer still consider worst-case execution times.

Does our queue need more AI compute infrastructure, or a different scheduling policy?

Queue delay has two distinct sources: a group exhausting its quota, and fragmentation under locality requirements. Run that split before buying anything. LynxBenchAI’s published results come in at the last step: once your scheduler and utilisation logs confirm a real capacity shortfall, shortlist devices from those results, which show how each runs a public PyTorch stack, then test your own jobs on the shortlist. They cannot diagnose the queue itself, because that evidence lives in your scheduler and utilisation logs.

LynxBenchAI shows how each candidate device runs a public PyTorch stack, so once your logs confirm a real capacity shortfall, the first step is to shortlist devices from its published results and then test your own jobs on that shortlist.

Jeon et al.’s analysis of a production multi-tenant GPU cluster at USENIX ATC 2019 shows the split clearly. It separates fair-share delay (a group has used up its quota) from fragmentation delay (a job is within quota, but no set of GPUs with the right locality is free). Across all jobs, fragmentation accounted for around 80% of waiting time. The same study found that GPUs in use averaged only about 52% hardware utilisation even though most GPUs were allocated. The 2026 GFS paper similarly cites studies putting average GPU utilisation in production clusters at rarely above 30–40%. A cluster that looks fully allocated can still leave much of its GPU capacity unused.

Scheduling policy alone can change effective capacity by amounts as large as a new purchase. A Slurm simulator study (2017) found that disabling node sharing on an 8,000-core cluster added 45% to the time needed to finish the same workload, which the authors equate to needing a 45% larger cluster. In the other direction, a workload analysis of NSF’s HPC resources (2018) estimated that 10% to 20% more nodes would let 95% of backlogged jobs run immediately, assuming allocations stayed at their current level. Measure actual wait times from job records, not from the scheduler’s own predictions. On ARCHER2 and Cirrus, Slurm’s estimated start times were most commonly accurate only to an hour or more.

To find which departments and workload types are hit hardest, break the job records down by field of science, institution and job size, as XDMoD does for wait time and other metrics. In one virtual cluster of Jeon et al.’s study, 25% of jobs using more than four GPUs waited at least 10 minutes, compared with 10% of single-GPU jobs. A dedicated station helps a group whose delay comes from lack of capacity. It does nothing for delay caused by priority rules or fragmentation, so fix the policy first where the logs point there.

Putting a number on the cost in delayed grants, papers or student milestones needs a second dataset: surveys or interviews that link wait times to specific project deadlines. Treat self-reported impact as approximate, because people remember delays selectively. Many centres do not collect this kind of evidence. A study of R1 institutions cites survey work finding that only 41 of 69 centres even measure usage of their systems, 33 academic outcomes and 32 ROI.

When is one dedicated GPU cluster easier to defend than on-prem plus cloud burst?

A single one-time purchase is easiest to defend when demand growth is flat and predictable and burst access is unreliable or expensive. When forecast growth exceeds on-prem headroom within the committee’s planning horizon, a small on-prem base plus shared or cloud burst is the easier case to make. LynxBenchAI’s published results help size the on-prem base, because they show how candidate devices run the public PyTorch stack your departments’ models are built on: shortlist devices from them, then check the shortlist against samples of each department’s own workload before the RFP is scored.

LynxBenchAI shows how candidate devices run the public PyTorch stack, which helps size the on-prem base; the first step is to shortlist devices from its published results, then check them against each department’s own workload.

Utilisation of NSF’s HPC resources grew by about two orders of magnitude between 2011 and 2018 for most NSF directorates. Separately, capacity at university clusters grew at a compound annual rate of 17.8%, which a 2025 survey of US research institutions puts down to three-to-five-year refresh cycles and limited capital. We would not assume that six departments’ combined GPU demand outgrows one system within two years. Treat that as a forecast to build from your own queue and utilisation logs, not as an established fact.

Universities already run shared and burst designs. Roughly 70% of US R1 campuses operate condo programmes, in which investors get priority queues and idle cycles are back-filled for general fair-share users. At one institution described in the same survey, Slurm gives each account an investment QoS with priority proportional to its purchase and a burst QoS that reaches idle nodes at roughly one-fortieth the initial priority.

Introl, a GPU infrastructure vendor, argues in an undated blog post that cloud breaks even at about 40% utilisation and that on-premise wins above 60%. That is a vendor’s claim, not a measurement of your workload; it also argues that organisations running GPUs more than 10 hours daily save money on-premise. Score the options on representative workloads from every department.

JUPITER’s procurement shows workload-representative scoring at exascale. It used a total-cost-of-ownership, value-for-money approach in which value was the number of reference workloads the system would execute over its lifespan. Application performance was weighted above synthetic tests, because synthetic benchmarks cannot give a realistic picture of resource consumption over the system’s life. The suite balanced CPU and GPU benchmarks at about 1:5. Bidders used it to prepare and commit their performance numbers, and the procuring entity selected the vendor on those values together with other criteria. The same approach scales down to a six-department purchase: a small reference set from each department, run on each shortlisted option.

What evidence pack defends the spec if a losing bidder challenges the tender?

We would build the defence of a GPU or edge-compute specification against a bidder challenge on three documents. The first is a technical specification that is demonstrably vendor-neutral in wording and criteria. The second is an independent benchmarking report with its methodology, hardware and software versions and raw data. The third is a short review memo that counsel and auditors can file with the tender.

Who can review our compute tender and benchmark the candidates independently?

TechnoLynx reviews a drafted technical specification for vendor-neutral wording and criteria coverage. TechnoLynx’s AI Readiness Scorecard scores an AI programme against a named published rubric with an evidence trail. TechnoLynx then designs and runs an independent benchmarking engagement across the relevant accelerators and backends using your own workload shapes. The methodology, configuration and raw results go into a report you can attach to the tender file, and the findings are summarised in a short written review memo addressed to legal counsel or auditors. LynxBenchAI’s published results give a public reference for how each device runs the PyTorch stack, alongside the engagement’s measurements of your own workload shapes.

This work is adjacent to TechnoLynx’s core delivery, not at its centre. It covers the engineering evidence and does not replace a procurement lawyer’s judgement on your jurisdiction. If the goal is internal learning and not legal defensibility, a review memo alone may be enough, without commissioning new benchmarks. Where the tender references published results from standard suites (MLPerf, SPEC), read their full disclosures, because those results cover specific configurations that do not necessarily match what is being bought. The independence of whoever runs the benchmarks must be real and documented. If it is only nominal, a challenger can attack the evidence pack itself, and a review report is only as strong as the transparency of the method and data underneath it.

How public tenders document vendor neutrality

Neutral wording is a legal requirement in some jurisdictions, not a courtesy. A guideline from the German Ministry of the Interior’s Procurement Office and Bitkom cites § 97 GWB and § 31(6) VgV. Under them, the subject of a procurement must be described in product-neutral terms, with brand names allowed only in justified exceptions. On that basis, server tenders may not name particular processor manufacturers or models. The same guideline sets the quality bar for benchmarks: detailed documentation of both procedure and results, so that another party can reproduce them, and the latest version of the benchmark, since outdated versions risk buying devices that underperform during their period of use.

It considers bidders’ own benchmark statements adequate for many award criteria, and reserves measurement on the actual equipment for criteria such as power consumption, where datasheets give only maximum values. For AI compute we hold the stricter line: evidence has to be tied to the buyer’s workload, because a bidder’s numbers on its own configuration can differ from performance on yours for reasons a datasheet does not show. The guideline itself accepts that published results come from specific configurations that may not match the servers being bought.

Two large public procurements show what the documentation can look like. The LISA tender separates mandatory requirements, which reject a non-compliant offer, from targeted requirements, which do not. Its benchmark suite combines a subset of MLPerf training tasks with HPL. Committed HPL and HPL-MxP values are scored by a stated formula that normalises them between the tender’s set minimum and maximum. JUPITER defined every benchmark in JUBE workflow scripts that fix execution parameters. It also included generated benchmark descriptions in the committed procurement documentation, together with hashes of the archived benchmark repositories. Safety assurance uses the same idea of an auditable trail. Under ISO/IEC/IEEE 15026-1, an assurance case is a reasoned, auditable artefact that connects claims to evidence and explicit assumptions. A compute tender can apply that structure: each requirement maps to a measurement, a threshold and the raw data behind it.

Keep the benchmark for acceptance day

Once the contract is signed, the evidence pack should become the acceptance test. JUPITER’s suite, built for vendor selection, is to be used again in the acceptance procedure. Acceptance is also the point to check unit-to-unit variance. Identical-SKU GPUs can run up to 1.5 times slower than the median, so the delivered units should be measured against the numbers the decision relied on, not assumed to match them. Run the same matched tests on the delivered hardware, compare each unit with its published device page or with your pre-award runs, and record the gap. Without that check, the evidence pack shows only that the spec was fair, not that the purchase delivered what was specified.

FAQ

How do AI benchmarks work?

A standard suite such as MLPerf Inference fixes the task, reference models, quality targets and run rules, then measures performance under defined scenarios. MLPerf Inference uses single-stream, multistream, server and offline scenarios and reports no single summary score.