A defensible clinical AI vendor evaluation for radiology triage rests on two things, both built before any vendor presents: a documented scoring rubric whose weights your clinical, technical and compliance leads sign off in advance, and a matched-condition test that runs every vendor’s model on the same held-out local cases through your own PACS test environment. Vendor-reported accuracy comes from evaluation sets of different size and make-up, and vendor-reported latency comes from different hardware and inference stacks, so neither can be compared as supplied.

TechnoLynx sets up and runs that comparison as part of its AI engineering for life sciences and healthcare work, and the first step is to fix the evaluation protocol, the held-out case set and the accuracy and latency metrics that apply across every vendor’s stack before any appliance is connected. LynxBenchAI measures AI performance on the GPU hardware itself, across accelerators, backends and precisions.

How should a hospital evaluate clinical AI vendors before buying a radiology triage tool?

Evaluate the vendors on your own cases, under conditions you control, against criteria and weights you fixed before seeing their numbers. This is not excess caution. In an independent head-to-head study published in Radiology in 2026, devices from seven manufacturers each analysed the same 5,235 consecutive chest radiographs requested from primary care at a single UK centre, with the multidisciplinary team decision as the reference standard. Area under the ROC curve ranged from 0.80 to 0.94, sensitivity from 20.8% to 77.8%, specificity from 58.9% to 98.4% and positive predictive value from 1.5% to 28.4%. Compared with radiologist reporting, the devices added between 10 and 2,039 extra false-positive tumour results, and the devices barely agreed with one another (κ = 0.24).

All seven were commercial AI devices for lung cancer detection. A spread that wide on one shared case set shows why vendor-reported figures need checking on your own cases. The authors of a 2026 Frontiers in Radiology paper put the right question plainly: does the model perform reliably here, “on these scanners, with these protocols, for these patients”, inside this workflow?

The work runs in a fixed order, because each step locks something the next one depends on:

1. Define the clinical use case and check that each tool’s regulated intended purpose matches it.
2. Agree the minimum acceptable performance for the proposed setting, with set definitions of sensitivity, specificity and clinical utility.
3. Assemble a representative local test set and a reference standard to compare AI output against.
4. Run every vendor’s model on that set in your test environment, and in shadow mode on live data where possible.
5. Score, weight and aggregate, validating and reporting the weights and the results.
6. After go-live, monitor for drift, software version changes, scanner upgrades and protocol changes.

Is there a standard scorecard for clinical AI vendor procurement?

A rubric becomes defensible through its method (justified criteria, stakeholder-derived weights, recorded evidence), not through adopting someone else’s numeric split. Published templates exist for the criteria and the scoring scale.

LynxBenchAI gives the hardware line of the rubric measured throughput, latency, memory, energy and cost per unit of work on the accelerator itself. It measures the hardware with stock PyTorch, so it does not replace running each vendor’s own model in your harness. Comparing price against performance across specific cards is a separate question from choosing a triage vendor.

The Healthcare Financial Management Association’s AI vendor evaluation form (Arizona chapter) is one concrete template. It scores each criterion on a five-point scale, from 1 “Unacceptable: does not meet expectations” to 5 “Excellent: fully meets or exceeds expectations with strong evidence”, and it names a selection panel of clinical workflow, IT, revenue cycle or finance, quality, compliance and business-system reporting leaders. Its criteria include model performance (accuracy, sensitivity, specificity and robustness across diverse patient populations), clinical validation evidence, and integration with EHRs and PACS. The ECLAIR guidelines from European radiology authors add a checklist, plus ten top questions to put to solution providers covering technical, financial, quality and safety considerations. The Royal College of Radiologists’ 2024 UK deployment guidance contributes the questions a generic form misses: PACS and RIS integration status, how the algorithm was trained, when the tool should not be applied, and whether results will generalise to your population.

A procurement rubric for triage appliances, adapted from those sources:

Dimension	What gets measured	Evidence that counts
Clinical accuracy	Sensitivity, specificity, PPV, NPV, AUROC, false-positive burden and subgroup performance on the local held-out set	Harness results at the agreed operating point
Latency and throughput	Median and tail (p90) per-study latency on the vendor’s own stack, with conditions disclosed	Harness logs plus stack disclosure
Integration	PACS and RIS integration fit and support for interoperability standards	Vendor responses to your data map, test-environment behaviour
Hardware and GPU cost	Total cost of the appliance over its service life: hardware, support, power, refresh	Quotes, plus published hardware measurements where available
Regulatory evidence	Intended purpose, conformity marking or clearance, documented limitations, risks and training data	Manufacturer documentation
Published validation	Peer-reviewed or real-world evidence for the use case	Publications, other sites’ deployment experience

Borrowing a generic SaaS or LLM procurement rubric without adapting it to clinical risk gives you a model that looks rigorous and does not hold up for this domain. The hardware and GPU cost row reflects our engineering judgement rather than a published standard.

What weighting between accuracy, latency, hardware cost and regulatory evidence holds up in front of a board?

The weighting that holds up is one derived from this use case’s clinical risk, elicited from the panel by a stated method, and justified in writing. ISPOR’s MCDA Task Force guidance (2016 summary) treats weighting as capturing stakeholders’ preferences between criteria. It advises choosing the elicitation method with the cognitive burden, the precision needed and the differences between stakeholders in mind, and it asks that weights be validated and reported.

Our judgement for a triage tool, where a missed finding delays care, is that accuracy on the local set and regulatory evidence should carry most of the weight. Latency works better as a gate than as a large weighted score. Either the appliance returns a result inside the turnaround the worklist needs or it does not, and a vendor that is faster beyond that point has gained little clinically. Hardware cost belongs in the weighted total, but it should never be able to buy back a failed accuracy threshold.

ISPOR names the additive model as the most common aggregation formula in healthcare MCDA. In an additive total, a cheap and fast vendor can outscore a more accurate one. So put the minimum acceptable sensitivity and specificity in as pass/fail gates ahead of the weighted score. The RISED pre-deployment framework (2026) prefers binary PASS / FAIL / INCONCLUSIVE verdicts over continuous scores that obscure whether a threshold has been met. Its default thresholds, including a 500 ms latency trigger for profiling the inference pipeline, are literature-grounded starting values that should be recalibrated for the target use case.

Before setting the regulatory weight, check what is actually regulated rather than assuming everything is. A regulated triage model is a medical device with a stated intended purpose. Whether the evaluation harness, test environment and scoring workbook fall under regulatory requirements is worth checking rather than assuming. The line that does matter is between validating locally, which is expected, and retraining or modifying the model, which will almost always need vendor involvement and regulatory review.

Documenting the scoring so it survives an audit later

An audit-proof scoring record captures what was decided, on what evidence, by whom and when, whichever vendor wins. Raji and colleagues’ internal algorithmic audit framework (2020) describes a “transparency trail” of documentation left at each step, organised into five stages (scoping, mapping, artifact collection, testing and reflection), each with its own documents. Medical device makers already work under documented design controls and keep a design history file of the product’s development. A hospital’s procurement record benefits from the same discipline.

The record should hold:

• each weight with its written rationale and the elicitation method used;
• the scores within each criterion and how they were validated;
• the case-set composition, the reference standard and the confirmation rule;
• the full stack disclosure behind every latency and accuracy figure;
• the gates, which vendors passed or failed them, and any deviation from the plan;
• named sign-offs from each panel role, dated before scoring opened.

Treating that record as an engineering output, rather than a policy document drafted afterwards, is what makes it something a compliance owner can sign against. See also our page on building defensible evidence packs for regulatory and procurement review. We would treat a consistent checklist for scoring each vendor’s regulatory package (clearance status, validation protocols, audit trail) as separate work feeding the regulatory-evidence row above. In the UK, RCR guidance also expects mitigation plans for false negatives and false positives to be recorded in the local DCB0160.

Why can’t vendor accuracy and latency figures be compared directly?

Vendor figures cannot be compared directly because each one was produced on a different dataset, hardware platform and inference stack. A 2026 review of efficient deep learning for medical imaging found that variation in hardware, runtime engine, numerical precision, batch size and warm-up protocol can cause order-of-magnitude differences in reported latency, which are easily mistaken for architectural superiority. The review’s authors call latency a hardware–compiler co-design outcome that needs explicit measurement under controlled conditions, and they note that performance rankings are bound to the specific hardware–software stack.

LynxBenchAI measures throughput, latency, memory, energy and cost per unit of work on the GPU hardware itself with stock PyTorch, across accelerators, backends and precisions, so the hardware line of a comparison rests on stated conditions rather than brochure figures.

A quantised model can keep strong aggregate AUC while becoming overconfident in wrong predictions or less responsive to rare pathologies. If a vendor ships an INT8 TensorRT engine in the appliance but reports accuracy from a full-precision run on a reference GPU, the brochure number does not describe what you would be buying. Add calibration error and rare-class sensitivity to the accuracy metrics, particularly for quantised models.

The fix is not to force every vendor onto identical hardware. The same review proposes a deployment-first standard that does not mandate uniform hardware but anchors each claim to a disclosed stack, protocol and measurement procedure. For a vendor comparison, that means each vendor’s stack runs natively and every figure arrives with its conditions attached:

• Task: modality, input resolution, preprocessing pipeline
• Hardware: device model, memory, power mode, thermal constraints, system load
• Software stack: runtime engine (TensorRT, ONNX Runtime, OpenVINO and so on), versions, backend configuration
• Precision: FP32, FP16 or INT8, and the quantisation method
• Inference protocol: batch size, warm-up runs, timed runs, I/O scope
• Latency reporting: p50 and p90, variance across runs, throttling notes

Different GPU hardware can legitimately produce different latency, and that difference belongs in the report rather than being normalised away, since the appliance is the product you are buying. Whether a given latency budget holds on specific target hardware under worst-case load is a separate validation exercise from the vendor comparison itself.

Running every vendor’s model on the same held-out cases in your PACS test environment

Run the same frozen case set through each vendor’s appliance from a single orchestrator, using an identical submission path into each, and harmonise the outputs to a common evaluation target before scoring. MHub.ai’s 2026 evaluation shows the pattern: three lung segmentation models ran on the same 422-patient public CT cohort with an identical command in which only the model identifier changed, and DICOM handling was standardised. Lobe-level outputs from two of the models were merged into left and right lung masks, so all three were scored against expert left- and right-lung annotations, available separately for about 300 patients. The principle carries over to commercial appliances: the harness owns the cases, the routing and the scoring, and the vendor owns only its inference path, which the harness should not alter.

PACS integration adds variation of its own. The RCR notes that implementations from the same PACS and RIS providers vary with local configuration, and recommends sharing a detailed data map so vendors can show how they would meet your requirements. Two complementary runs follow from the RCR guidance. One is shadow mode, where the AI runs in the background on real patient data and its findings are not used clinically. The other is a local enriched set of positive and negative cases, which checks that the software works as intended with your population, scanners and protocols. Shadow mode also gives you local false-positive rates, which manufacturers do not normally supply because those rates depend on the prevalence of findings in your referral population.

Keep the held-out cases inside the hospital network, put each appliance on an isolated test segment, and limit and log vendor remote access during the trial. Which data may leave the environment, and under what agreement, is a question for your compliance team and counsel. Confirming that any cloud component of a vendor’s offering sits within eligible regions and business associate agreement coverage is a separate check.

Which reference standard should label the held-out test set, and how long should you wait for confirmation?

Label the held-out set with the strongest reference standard the clinical question allows, decided before any vendor output is seen, and fix the confirmation window in writing. The published comparisons show what that looks like in practice. The seven-manufacturer chest radiograph study used the multidisciplinary team decision. Project AIR’s lung nodule validation used a chest radiologist’s reading based on CT, and its bone age validation used three to five expert readers. The RCR describes suitable ground truth as biopsy-proven disease or long follow-up with no adverse consequences.

The right confirmation window depends on the condition. The test is whether a case labelled negative has had a realistic chance to declare itself; for slow-growing disease that means waiting for follow-up imaging or outcome data before the label is final.

At the 1.4% lung cancer prevalence in the seven-manufacturer study, its 5,235 radiographs contained roughly 73 cancers (0.014 × 5,235), which is a modest base for comparing sensitivities. An enriched data set more reliably identifies limitations, though the RCR warns it can overstate algorithm performance. Specificity needs a negative sample representative of the referral population, including conditions that mimic the disease. Plan scanner-, protocol- and patient-group subgroup analyses too, since aggregate accuracy can conceal systematic underperformance in subgroups. The RISED authors showed an AUROC of 0.96 hiding failed reliability and sensitivity checks, and an inconclusive subgroup verdict that only a larger cohort could resolve.

Finally, do not accept each vendor’s default operating point as given. Evaluate each model at thresholds tuned for your population, because a vendor-provided operating point is not universally optimal.

Can an independent party run the evaluation instead of the vendors?

Yes, and published projects already do it. Project AIR compares commercial products head-to-head by applying them to the same hidden dataset, with radiologists reading the same cases for comparison. In its 2024 Radiology study, nine of 17 eligible AI products were validated on data from seven Dutch hospitals. Project AIR’s method enables direct comparison between competing products and with human readers.

The RAISE lifecycle paper notes that independent validation studies can be run by deploying hospitals, research institutions, AI marketplace companies or medical societies. The RCR adds that a bid should state up front whether independent validation will be required. Put that requirement in the tender, so vendors agree to the hidden-set protocol before the comparison starts.

Who can run a matched-condition comparison of triage vendors for our hospital?

TechnoLynx designs and runs the matched-condition comparison for a hospital choosing between triage appliance vendors. TechnoLynx first frames the evaluation protocol, held-out dataset and accuracy and latency metrics. TechnoLynx then builds a harness that sends the same cases through each vendor’s appliance without altering its inference path, runs it, checks that the accuracy and latency differences are reproducible and statistically meaningful, and delivers an independent report, with re-runs as vendor models update. This sits within TechnoLynx’s AI engineering for life sciences and healthcare work.

Scope matters. An independent party makes sense when the board wants the harness run by someone with no stake in the outcome. Whoever you choose, ask them to disclose full stack conditions for every figure. We produce the engineering and the evidence; we do not give regulatory sign-off, a legal opinion on HIPAA or clinical labels. The reference standard stays with your radiologists or multidisciplinary team. Where a Project AIR-style independent validation already covers your use case, or where published independent head-to-head studies or benchmark datasets fit your population, check their methodology and results for relevance to your local use before relying on them.

The winning score describes one model version on one stack

A vendor comparison is a dated measurement, not a lasting verdict. The Frontiers authors ask for monitoring after deployment for software version changes, scanner upgrades, protocol changes and discordance with radiologists. RAISE adds runtime monitoring of latency, throughput and inference time against service agreements. That makes the harness and the frozen case set worth keeping after procurement closes. When the chosen vendor ships a new model or the appliance’s runtime changes, re-run the same cases under the same disclosed conditions. That tells you whether the result your board signed against still holds.