A fair hardware vendor evaluation runs in a set order. First, check each vendor’s licence for clauses that restrict publishing benchmark results, and do it before anyone is named. Then source the boards through channels you can defend, put every candidate on a comparable software stack so you know whether you are measuring silicon or firmware, and only after that weigh the measured results against whole-life cost, supply and support. The evaluation also needs an honest plan for dev kits that arrive late and for a deadline that lands before the tests are finished.

What a hardware vendor evaluation has to measure beyond the spec sheet

LynxBenchAI treats benchmarking as evidence for a decision someone is about to act on, tied to a workload rather than to a specification sheet, because systems that look best on paper are the ones that get chosen and then fail in real use. For procurement, that means evidence on practical performance with your own workload, software readiness and reproducibility, not the specification sheet.

Sustained throughput is the first thing we would measure beyond the spec sheet. A 2026 edge-inference study (arXiv 2603.23640) found that the gap between peak hardware specifications and sustained real-world inference under thermal load is still poorly characterised. In that study the iPhone 16 Pro lost roughly 40% of its peak throughput within three iterations, while on dedicated hardware other limits took over: the RTX 4050 was bounded by its battery power ceiling and the Hailo-10H by on-module memory bandwidth. The same authors note that MLPerf Inference v5.1 measures peak performance without tracking thermal throttling or sustained-workload behaviour. If the deployment runs for hours, a ten-second burst number answers the wrong question.

Remote attestation support is the second. NVIDIA’s 2021 guidance for edge security architects names secure and measured boot, remote attestation and drive encryption as critical parts of edge security, because most edge systems lack physical security. RFC 9334 sets out the roles: an Attester produces Evidence about itself, a Verifier appraises it against policy, and a Relying Party uses the result to decide whether to trust the device. That Evidence is built from signed measurements of hardware, firmware, BIOS and software, starting from a root of trust. We would score attestation as a demonstrated capability (the device produces Evidence that a verifier you choose can appraise) rather than as a checkbox on a datasheet, since the protection behind the attesting environment ranges from simple process isolation to dedicated hardware. RFC 9334 describes that range explicitly, from minimal protection through a TEE to specialised hardware that resists decapping and glitching attacks.

Who can review our hardware spec and vendor claims with no stake in which vendor wins?

TechnoLynx reviews a draft tender specification, or a vendor’s performance, thermal or capability claim, against the workload you actually run, through its AI Governance and Trust engagements. The first step is to establish the real workload, thermal or capability requirement behind the document. After that comes the review scope or a vendor-neutral pilot protocol, then the review itself or a pilot run against your own samples. Findings on vendor-lock risk, claim accuracy and capability gaps are documented, and the checked specification or test protocol is handed over for the tender. Whoever you use, ask them to state any reseller, referral or partnership tie to the candidates. A review is only as independent as that answer.

This work does not replace a procurement panel’s formal scoring. UK Cabinet Office guidance (2021) expects bids to be scored independently by at least two evaluators with a moderator. It is also not legal or regulatory sign-off. Reasonable alternatives exist. A specialised IT procurement consultant is one route, because supplier expertise creates an information asymmetry that can end in over-specification and unnecessary add-ons. An internal review team is another: SEBoK suggests combining people close enough to the project to know it with people far enough removed to see it independently. A pilot benchmark adds time and cost that a tender timeline may not allow.

Independence carries weight: in a US Federal Acquisition Regulation example, a company that wrote the data system specifications and equipment performance criteria for a hardware competition should be excluded from the initial follow-on hardware competition. Benchmarking has a similar conflict of interest. Raasveldt and colleagues (DBTest 2018) describe a conflict of interest in running benchmarks: researchers have an incentive to configure the system they compare against less carefully.

Reading the tender specification before it goes out

A pre-issue review asks two questions: are we asking for capability the workload does not need, and are we missing something a vendor will not volunteer? The Cabinet Office guidance requires criteria to be objective, proportionate, relevant and unambiguous. It also warns against copying criteria from earlier programmes without checking they fit this one. Treat ‘Must have’ requirements with the same care, because failing one usually means exclusion, which can shut out workable alternatives.

Checking a pod design’s PUE and cooling claims

The Green Grid’s PUE white paper states that PUE cannot be calculated accurately from nameplate ratings and that actual energy measurements are required. The Green Grid recommends a year of monitoring, or at least a month at verified typical loading. It discourages comparing PUE across facilities without proper analysis. ISO/IEC 30134-2:2026 defines PUE measurement categories and rules for unmeasured energy, so the first question for any claimed figure is which category it was measured under.

Final sign-off on a rewritten, vendor-neutral specification

A specification rewritten to remove vendor lock still needs a last independent read. We would check it for product names, proprietary interfaces and one vendor’s metric definitions stated as requirements. The Cabinet Office guidance calls peer review of quality criteria good practice before they are finalised. SEBoK recommends agreeing entry and exit criteria for a review before it starts.

Testing a vision system’s capability claim on your own samples

A claim that a camera system can tell two defect types apart should be tested on your own samples before a purchase order, using the same samples and the same pass criteria for every vendor. A fair comparison needs both systems to perform exactly the same function. The test should also reproduce a scenario a user would reasonably run in production. Keep in mind that a pilot on a sample set may not generalise to full production volume or the full range of defect variation.

Do vendor licences restrict publishing benchmark results?

Some do, and you need to know which before your comparison names a product. A DeWitt clause is a provision in a proprietary licence that bars publishing information about the software, such as benchmarks, without the vendor’s approval. Oracle created the original clause, and it became a standard feature of most database vendors’ licences. That is why research papers so often compare anonymous systems labelled ‘DBMS-X’.

The clauses also appear in the software licences that come with AI hardware. Hailo’s end-user licence bars using its Licensed Software ‘for any benchmarking to be publicly published’. It also bars disclosing any technical features, performance, benchmark test or comparative analysis to a third party without written permission. Hailo’s licence defines that Technical Data as its Confidential Information. Other vendors have similar terms:

• NVIDIA’s Nsight Systems 2024.4 licence bars disclosing benchmarking or performance data relating to the software or NVIDIA GPUs when comparing directly with non-NVIDIA platforms, unless NVIDIA gives prior written permission.
• Dell’s US federal EULA (May 2026) requires Dell’s prior written consent to share or publish benchmarking results of its software.
• Microsoft’s Universal License Terms require prior written approval to disclose benchmark results for any Server Product.
• Cohesity’s evaluation terms (August 2025) treat benchmarking results and the evaluator’s impressions of performance as confidential.

You cannot simply avoid these contracts. As David A. Wheeler points out, the supplier is usually the only legal source of the product you want to benchmark. Enforceability has been challenged in court. In 2003 the New York Attorney General’s office persuaded a court to prevent Network Associates selling software under terms that banned disclosing benchmark results without permission. Terms also change: Intel’s microcode licence prohibited publishing benchmark results, and after complaints Intel said it would update the licence. Databricks removed its own clause and introduced a ‘DeWitt Embrace Clause’, which invalidates a competitor’s own clause if that competitor benchmarks Databricks, so Databricks can benchmark and name it. Open-source licences cannot carry DeWitt clauses.

For each candidate, we would check which component the clause attaches to (SDK, compiler, firmware or the device itself), whether it covers internal testing or only publication, and whether a comparison against other vendors triggers it. Recheck the current terms just before you publish, because the version you agreed to months ago may not be the one in force.

Before you name vendors by product, have qualified counsel read each named vendor’s current agreements. That is a legal question, and technical diligence does not answer it. WBA pairs an information-only disclaimer with a fact-check window for assessed companies, and SPEC requires a disclosed basis for comparison. Where a clause blocks naming, research papers report anonymised systems instead. How to structure the publication itself is covered in building trustworthy, reproducible cross-vendor benchmark disclosures.

Running every candidate through the same pipeline on an equivalent setup

A comparison is fair when every candidate does the same work, on an equivalent setup, with the configuration written down well enough that someone else could rerun it. Raasveldt and colleagues, citing Jain, give differing hardware or software configuration as an example of a benchmarking ‘game’, Jain’s term for deliberate manipulation: the results are not comparable. A badly configured system can perform significantly worse than a properly configured one, so unequal tuning effort produces a misleading result even when nobody intends it.

The checklist below draws on a published fair-benchmark checklist, the MLPerf Inference rules and the ethical vendor benchmark manifesto:

1. Same model and the same processing on every candidate. MLPerf’s Closed division requires pre-processing, post-processing and the model to be equivalent to the reference, and it permits calibration for quantisation but no retraining.
2. Quantisation that can be reproduced. Any quantisation method has to be described publicly in enough detail to reproduce, using only the supplied calibration data.
3. Equalise everything you can. Use the same operating system, client languages and benchmarking tools, and research each product well enough to use its fairest configuration.
4. Record every version. List the software version of every component, the hardware configuration, the software stack, the containers and the settings.
5. Record the optimisation settings. Record compilation flags and system parameters for every candidate.
6. Control warm-up and thermals. Separate cold runs from warm ones. The 2026 edge study let each device equilibrate for 10 minutes at ambient temperature, discarded a warm-up inference and confirmed thermal stability before timing.
7. Repeat and report spread. Do several runs and check the standard deviation.
8. Verify outputs. Confirm that each system produces the same results, or the speed comparison is measuring different work.
9. Make it rerunnable. Any third party should be able to run the benchmark from published code and instructions. MLPerf’s rules are blunter: results that cannot be replicated are not valid results.

Can you buy Hailo-8 or RB6 boards without going through a vendor review programme?

For these two platforms, public purchase channels exist. Stock still has to be confirmed on the day you order. Hailo’s own distributor list includes Mouser, Farnell, EBV and Raspberry Pi, among others, by region. One retailer, UP Shop, listed the Hailo-8 M.2 2280 module at $199 marked ‘Stock Available Now’, with support offered only in combination with UP products validated for the module. When Qualcomm announced the RB6 platform in May 2022, Robotics 24/7 reported that the RB6 development kit was available through Thundercomm.

The decision logic is simple. If a distributor publicly lists the board with a lead time inside your deadline, buy it there. If nothing is listed, budget time to work with the vendor or its partner programme. Swapping in a board that is easier to buy is a legitimate option. It changes the comparison, though, and may no longer answer the hardware question you started with. Stock listings are snapshots, so verifying them with named distributors at the time of purchase is part of the evaluation plan.

When the dev kits will not arrive before the review deadline

A missing board is a logistics problem before it is a technical one. Pursue retail stock and a loan in parallel, and if neither lands in time, scope the test to the smallest setup that still runs your pipeline, labelled clearly as partial.

Situation	Route	What the evaluation report must say
Distributor or retail stock with a lead time inside the deadline	Buy	Nothing beyond the normal configuration record
No stock in time, but a lab, integrator or distributor can lend a unit	Borrow, after reading the loan terms	The loan conditions and any configuration the lender fixed
Neither route lands in time	Minimal carrier-board setup for the missing SoC	That results for that board are partial, and why

The Qualcomm Robotics RB5 Core Kit contains the mainboard with the RB5 SoM, a wall power adapter and a USB Type-C cable, while the Vision Kit adds the vision mezzanine and camera modules. The camera modules come only with the Vision Kit. For an M.2 accelerator, the host becomes part of the result: the UP Shop listing for the Hailo-8 module itself says actual performance depends on the host platform and the inference model. A minimal substitute may also not reflect production thermal and power behaviour, which weakens its comparability with fully kitted boards.

Read loan terms before you plan tests around them. Under Cohesity’s evaluation terms, for example, title to loaned hardware stays with the lender and the evaluator may not modify the hardware or its configurations during the evaluation period. Hailo’s terms permit access to its samples only for internal, non-commercial, non-production purposes. Return deadlines and configuration limits like these can conflict with your review schedule, so schedule the loaned board’s runs first.

Sanity-checking a vendor’s spine-leaf topology and switch count

Yes, an independent team can audit a proposed spine-leaf fabric against your target scale. The audit only counts if the reviewer has no stake in how many switches you buy. The same applies to resellers: their recommendations need checking for independence from vendor partnerships and margin structures.

SEBoK describes a technical review as analysis by sufficiently independent and knowledgeable stakeholders against pre-established criteria, and this review needs the same. Add written records that justify each conclusion, which the Cabinet Office guidance requires of scores in case the award is challenged. Two limits apply. The audit is only as good as the growth assumptions you supply, and a design review does not replace latency and throughput testing once the hardware is deployed.

Which boards can actually ship inside a loaner or embargo window?

Treat ‘announced’ and ‘shipping’ as different states. For this cycle’s shortlist, keep only boards with confirmed distributor stock or a written loaner commitment inside your window, and move still-sampling parts to the next cycle unless the vendor commits to your dates. LynxBenchAI published results answer part of this for the devices they cover. The 26Q3 results cover 48 devices, almost all of them GPUs and CPUs, so a GPU on your shortlist can be checked on its device page before you chase a loaner. None of the edge NPU boards discussed on this page appears in the 26Q3 results. For hardware the release does not cover, LynxBenchAI benchmarks real AI performance on a machine you already have in hand.

NVIDIA does not specify lifecycle information for Jetson developer kits, which it describes as tools for software development and prototyping, not production. For Jetson products, NVIDIA states an official end-of-life notice will be sent at least eight months before the last shipment. CNX Software reported a Connect Tech notice saying NVIDIA was accelerating end-of-life timelines for the Jetson TX2 NX, TX2i, AGX Xavier and Xavier NX as LPDDR4 supply tightened. Distributor channels also differ by region. Even with hardware in hand, check whether an embargo limits what you disclose and when.

How much of a throughput gap comes from firmware and runtime rather than the chip?

A single end-to-end number does not separate the two. A raw cross-vendor throughput gap mixes runtime, firmware and serving-stack overhead with silicon capability, so attributing it requires pinning the stack and holding the workload fixed. The 2026 edge study shows why. Each platform ran a different inference framework (vLLM, MLC-LLM, MLX, hailo-ollama) and a different nominally 4-bit quantisation format. The authors conclude that the observed differences reflect hardware and software combined, rather than hardware alone. They attribute the S24 Ultra’s inflated prefill time largely to the framework, and the RTX 4050’s determinism partly to vLLM’s scheduling rather than to the GPU. They state that alternative software stacks may give materially different results on the same devices.

Our own published results show a platform effect on a single card. In the 26Q3 results the NVIDIA GeForce RTX 5090 on native Linux scores 985.8 for inference and 597.5 GT, while the same card under WSL scores 891.8 and 545.7. The same card is listed under both platforms in the 26Q3 results. (GT is an ordinal aggregate, so read the order, not a percentage.) Precision support belongs to the executor too: the NVIDIA GeForce RTX 3080 produces no FP8 inference result in 26Q3.

To pin two vendors to a comparable stack, use the same model and a reproducible quantisation method on both, and record every software version, hardware configuration and setting. We expect isolation to be partial. Hailo’s Dataflow Compiler, for example, is co-designed with the hardware architecture of its neural network processor. A controlled study also pins only one configuration, and its attribution may not carry over to later driver releases or to your workload.

The choice depends on the question. If you are judging silicon headroom, run the controlled study and read off the hardware-attributable part of the gap. If what matters is as-deployed throughput on the stack you will actually ship, an end-to-end benchmark is the right measurement, as long as it is labelled as a platform result.

Scoring hardware vendors against cost, supply and support

Measured performance is one input to the award, not the whole of it. The UK Cabinet Office’s bid evaluation guidance defines the target as the most economically advantageous tender, judged against published award criteria, which may not be the lowest-cost bid. Value for money there means the best mix of quality and effectiveness for the least outlay over the period of use. Price evaluation should consider whole-life cost, covering acquisition, maintenance, management, operation and disposal.

Price per quality point (PQP) is one way to combine the two. Divide each bid’s price by its quality score (a whole number, for example points out of 100) to get a price per quality point. The guidance credits PQP with discouraging unsustainable lowest-price bidding and making value for money easier to assess. Disclosed minimum price or quality thresholds keep weak bids out of the calculation. Weightings should be tested with the market before they are fixed. Finance models should be evaluated independently of the technical specification.

On the quality side, an IT procurement guide from Commercial Consulting lists independent testing data, total cost of ownership, vendor stability over a three-to-five-year support horizon, supply chain reliability, and support and warranty terms as the criteria to weigh. For accelerators such as Hailo’s, support includes the compiler toolchain. Beri.net’s analysis of Hailo after the Microchip acquisition notes that the Dataflow Compiler licence is revocable and that the compiler is needed every time the model changes. The same analysis cites Intel’s Neural Compute Stick 2: the hardware kept working, but new toolkit support stopped at the OpenVINO 2022.3.x LTS line. We would score toolchain continuity as a separate supply risk, because a compiler that stops moving ends your ability to retrain and redeploy long before the silicon wears out. The framing of procurement risk around that score is covered in de-risking and defending an AI compute procurement.

If the evaluation cannot finish before the procurement deadline

Commit to what you can undo. In practice that means evaluation units, loans or dev-kit quantities, not a volume order, and contract terms with a defined end date. Cohesity’s evaluation terms, for example, default to 60 days, or 90 days for technology partners. That makes a smaller, time-boxed trial a reasonable thing to ask a vendor for, not a special favour. Watch for orders that cannot be reversed. In the Connect Tech notice CNX Software reported, all new purchase orders for products with the affected Jetson modules became non-cancellable and non-returnable. Previous-generation equipment can also bridge the gap where performance requirements allow.

The evidence to bring when asking for a smaller trial is the evaluation you have already done. Bring the written protocol, the partial results labelled as partial, and a named list of what still has to be measured on the vendor’s hardware. Records that justify each score are also what supports the award if it is challenged. A report that says plainly which board ran on a substitute carrier, which numbers stayed confidential under a licence and which stack versions were pinned will hold up under scrutiny that a cleaner-looking table would not.