A medical-CV team ships a model that scores 0.94 AUC on its internal test set, then watches performance collapse the moment it runs on scans from a different hospital’s scanner. The single held-out test set never saw that failure coming, because it came from the same site, the same acquisition protocol, and largely the same patient population as the training data. This is the gap MedPerf — an open benchmarking platform from MLCommons — is built to close: it measures how a model behaves across clinical sites and patient populations without ever moving the test data off the premises where it was collected. The reason this matters is not academic. For medical-device computer vision under Software as a Medical Device (SaMD) rules, generalisability evidence is a regulatory requirement, not a nice-to-have. The FDA wants to see that a diagnostic model holds up on data it was never tuned to, across the demographic and equipment variation it will actually encounter in the field. And the data that would prove this is scattered across institutions that are legally and practically unable to hand it over. What problem does federated benchmarking solve that a single held-out test set cannot? The naive validation story is clean: collect a dataset, split off a held-out portion, report accuracy on it, and call that your evidence. It fails for one structural reason — the held-out set is drawn from the same distribution as everything else you touched. It shares the scanner make and model, the reconstruction kernel, the referral patterns that decide who gets imaged, and the local prevalence of the condition you are detecting. A model can memorise those confounders and still post an impressive number. What breaks in production is population shift and acquisition shift. Move to a hospital that runs a different CT vendor, serves an older or more comorbid population, or uses a different contrast protocol, and the model’s operating point drifts. We see this pattern regularly: a detector that looked strong on internal data loses several points of sensitivity on external sites, and the loss is uneven across subgroups. A single test set cannot surface this because it was never allowed to be different. Federated benchmarking inverts the assumption. Instead of pooling test data centrally and hoping it is representative, it evaluates the frozen model at each site, on that site’s own data, and reports the spread. The evidence you get is not one number but a distribution across environments — which is exactly the shape of question an FDA reviewer is asking when they probe whether your device generalises. How MedPerf works, and what it means in practice MedPerf treats the model, the data, and the evaluation as three separable roles that never need to sit in the same place. The mechanism is deliberately boring, which is what makes it credible. A benchmark committee defines the task. It specifies the clinical question, the reference standard, the input format, and the metrics — the contract every participant runs against. The model owner packages the model as a container (typically a Docker image wrapping a runtime such as PyTorch or ONNX Runtime, with the preprocessing baked in so behaviour is reproducible). Each data provider runs the container locally, behind their own firewall, against their own labelled test data. The raw images and labels never leave the institution. Only aggregate metrics come back — per-site AUC, sensitivity, specificity, subgroup breakdowns — to a central aggregation point. No patient-level data, no images, no protected health information crosses the boundary. The divergence from the naive path is entirely about data movement. The single-test-set approach assumes you can bring the data to the model. MedPerf assumes you cannot, and structures the whole evaluation around that constraint. For a team that has ever tried to negotiate a data-sharing agreement with a hospital’s legal and IRB office, that inversion is the difference between a benchmark that can happen and one that stalls for a year. This is the same “ship the computation to the data” discipline that governs privacy-constrained CV work generally — the logic that underpins on-device and edge evaluation in our work on client-side ML deployment applies just as directly when the constraint is HIPAA rather than latency. How does MedPerf produce multi-site evidence without centralising patient data? The privacy guarantee rests on what physically crosses the network boundary. Because the container executes inside the data owner’s environment and emits only summary statistics, the platform sidesteps the need for a data-use agreement covering raw records. A data provider is agreeing to run a piece of code and return a small table of numbers — a far lighter legal ask than transferring a de-identified imaging archive, which in many jurisdictions is not de-identifiable enough to satisfy the institution’s counsel anyway. The trade-off is trust in the evaluation harness rather than in data transfer. The committee’s benchmark definition, the container, and the metric computation all have to be auditable, because no one at the aggregation point can re-derive the numbers from source data — they never see it. This is a real shift in where verification effort goes. It is closely related to the discipline we describe in validation evidence that holds up to FDA review: the statistical claim is only as good as the frozen, documented procedure that produced it. What metrics and artefacts does a MedPerf benchmark return? A run produces a set of per-site and pooled metrics plus the provenance needed to defend them. The table below maps the common outputs to the validation question each one answers. MedPerf artefact What it reports FDA-relevant question it addresses Per-site performance metrics AUC / sensitivity / specificity per participating institution Does the device hold up across independent clinical environments? Subgroup breakdowns Metrics stratified by age, sex, scanner, or protocol Is performance equitable across the intended population? Detection-quality metrics mAP, IoU-based scores for localisation tasks For detection devices, how good is localisation, not just presence? Benchmark manifest Frozen task definition, reference standard, metric code Is the evaluation reproducible and auditable? Container digest Immutable hash of the evaluated model image Was the submitted model exactly the one that was tested? For localisation-heavy tasks, the detection-quality outputs matter as much as the classification ones — the distinction between presence and precise localisation is why we treat [email protected] as it maps to FDA validation evidence as a separate line of evidence from the tighter localisation demands captured in the mAP50-95 detection metric. A MedPerf benchmark can report both; which one carries weight depends on the clinical claim. None of these numbers are benchmarked here as reproducible figures — the point of the artefacts is that your run produces your numbers, defensible because the manifest and container digest let a reviewer confirm exactly what was measured. Where MedPerf fits in a validation pipeline versus internal testing MedPerf does not replace internal testing; it sits downstream of it. Internal evaluation — the fast iteration loop against your own held-out data — is where you find and fix the obvious failures. MedPerf is where you find the ones your own data structurally cannot show you. Treating it as a substitute for either internal testing or a formal clinical study is a category error. The sequence that works, in our experience across regulated-CV engagements, looks like this: freeze a model candidate, run it internally to confirm the operating point, then run it through federated evaluation across external sites before the submission is written, so that any population-shift gap surfaces while there is still time to act on it. Programmes that discover a subgroup performance cliff during MedPerf evaluation, rather than at submission review, avoid the retrofit scramble — the 6–12 month clearance delay (an observed pattern across medical-device timelines, not a fixed regulatory figure) that comes from having to go back and assemble generalisability evidence after a reviewer flags its absence. The economic anchor is the evidence-gathering phase itself. Negotiating a central data-sharing agreement with each site, one at a time, is measured in months of legal and IRB review. A coordinated federated benchmark collapses that into a single agreement-light run because the legal ask per site is so much smaller. That compression is the ROI, and it is why the results feed directly into the generalisability evidence package a regulatory-pathway engagement assembles. What are the practical limitations of MedPerf for a team pursuing FDA clearance? The honest boundary conditions matter more than the strengths for anyone planning around this tool. It is a benchmarking platform, not a clearance. MedPerf produces evidence; it does not produce a regulatory decision. The results are an input to a submission, weighed alongside the clinical study design and risk analysis. Site recruitment is still real work. The legal ask is lighter, but you still need willing institutions with labelled data and the operational capacity to run a container. Federation does not conjure participants. Label quality is not federated away. The reference standard at each site has to be trustworthy. Heterogeneous labelling practices across institutions can widen the metric spread in ways that reflect annotation noise, not model behaviour. The harness must be auditable. Because no one re-derives the metrics from raw data, a flaw in the container or the metric code is invisible until someone inspects the definition. The manifest-and-digest discipline is what keeps this honest. Aggregate-only output limits root-cause analysis. When a site reports a poor number, you cannot pull the failing cases to inspect them. Diagnosing why a subgroup underperforms often needs a separate, consented investigation — which is where structured failure attribution like causal trees for treatment-effect estimation in medical-device CV earns its place. FAQ How does MedPerf work? MedPerf, an open platform from MLCommons, separates the model, the data, and the evaluation into roles that never co-locate. A benchmark committee defines the task, the model owner ships a container, each data provider runs it behind their own firewall against local data, and only aggregate metrics return to a central point. In practice it means you can measure a model across many sites without ever moving patient data. What problem does federated benchmarking solve that a single held-out test set cannot? A single held-out set is drawn from the same distribution as your training data — same scanners, protocols, and population — so it hides population and acquisition shift. Federated benchmarking evaluates the frozen model at each independent site on that site’s own data and reports the spread across environments. That distribution across sites is the shape of generalisability evidence an FDA reviewer actually asks for. How does MedPerf produce multi-site generalisability evidence without centralising patient data? Because the model container executes inside each data owner’s environment and returns only summary statistics, no images or protected health information cross the network boundary. This reduces the legal ask per site from a full data-transfer agreement to permission to run code and return a small table of numbers. The trade-off is that trust shifts to the evaluation harness, so the benchmark definition, container, and metric code must be auditable. What metrics and artefacts does a MedPerf benchmark return, and how do they map to FDA validation evidence? A run returns per-site and subgroup performance metrics (AUC, sensitivity, specificity, and detection scores like mAP for localisation tasks), plus a frozen benchmark manifest and an immutable container digest. The performance metrics answer whether the device holds up across sites and populations; the manifest and digest make the evaluation reproducible and confirm exactly which model was tested. Together they form a defensible generalisability package. Where does MedPerf fit into a medical-device CV validation pipeline versus internal testing? MedPerf sits downstream of internal testing, not in place of it. Internal evaluation is the fast loop where you fix obvious failures; MedPerf surfaces the failures your own data structurally cannot show. The effective sequence is to freeze a candidate, confirm the operating point internally, then run federated evaluation across external sites before writing the submission. What are the practical limitations of MedPerf for a team pursuing FDA clearance? MedPerf produces evidence, not a clearance, and the results are only one input to a submission. Site recruitment is still real work, label quality varies across institutions and can widen the metric spread, and the aggregate-only output means you cannot pull failing cases for root-cause analysis. The evaluation harness must also be auditable, since no one re-derives the numbers from raw data. The deeper point is that generalisability is not a property you bolt on at submission time — it is a measurement you either designed for or did not. A model that has only ever been scored against data from its own origin is not undertested by a little; it is untested on the question that decides whether it is safe to deploy. If you are scoping a medical-device CV programme, the question worth asking early is blunt: which sites, which populations, and which scanners will your evidence actually cover — and how will you measure that without the data ever leaving the building? That is the framing our computer vision engineering practice works from, and it is where MedPerf results become the backbone of a regulatory-pathway evidence package rather than a last-minute patch.
