A radiology group runs your segmentation model against a published leaderboard number, sees a strong pooled AUC, and still refuses to sign. That refusal is rational. The pooled number tells them how the model did on somebody else’s scanner mix and somebody else’s patient distribution — not theirs. This is the gap MedPerf was designed to close, and it is also the gap most teams misread when they first encounter it.

The naive read of MedPerf is that it is a leaderboard for medical AI: run the model, post an aggregate metric, call it validated. That framing quietly discards the one property a site reviewer actually needs. MedPerf is a federated evaluation framework — a harness that scores a model inside each participating site’s own data enclave, without the test data ever leaving that enclave, and returns per-site results rather than a single pooled figure. The difference is not cosmetic. It changes what kind of evidence you walk away with.

How should you think about MedPerf in practice?

MedPerf, developed under the MLCommons umbrella, coordinates evaluation across multiple clinical sites without centralizing patient data. The model, packaged as a container, travels to the site. The data does not travel anywhere. At each site, the harness runs three pinned components in sequence: a data preparation step that maps local data into the format the model expects, the model inference itself, and a metrics computation step that scores the predictions against local ground truth. Only the resulting metrics — not the images, not the labels — return to the aggregator.

That containerized, three-stage structure is the load-bearing part. Because the data preparation and metrics pipelines are fixed and versioned artifacts, every site scores the same model the same way against its own data. You are not comparing a number computed by one lab’s homegrown script against another lab’s. You are comparing the same measurement procedure applied to genuinely different distributions. In our experience with clinical-imaging evaluation, that procedural consistency is what a reviewer trusts before they trust any specific score.

What this means in practice: MedPerf produces a table of results indexed by site, each carrying the distribution characteristics of that site — its scanner vendors, its acquisition protocols, its patient demographics. That table is the artifact. Not the best row, not the mean.

Why does federated evaluation produce more portable validation evidence than a pooled benchmark AUC?

A single pooled AUC is an average over a distribution you did not choose and cannot inspect. Suppose a model reports an AUC of, say, 0.93 across a five-site consortium. That figure is compatible with the model performing at 0.97 on four sites and 0.78 on the fifth — a spread that would matter enormously to the fifth site, and which the pooled number erases entirely. Averaging collapses exactly the variance a validation reviewer is paid to interrogate.

Federated evaluation surfaces that variance instead of hiding it. Because MedPerf returns site-stratified results, a prospective customer can look at the rows that resemble their setting — the sites running the same CT vendor, the same field strength, a comparable case mix — and read a number that reflects conditions close to their own. This is the portability property (an observed pattern across clinical evaluation work, not a benchmarked constant): evidence generated as per-site strata carries meaning from one customer to the next, because each reviewer can locate themselves in the strata. A pooled number carries almost nothing, because it describes a population that no single site belongs to.

There is a subtlety worth naming. Site-stratified results are more portable, but they are not automatically transferable. A site whose scanner mix and protocols are genuinely unrepresented in the evaluation cohort still needs its own evaluation. Stratification lets a reviewer decide whether they are represented; it does not manufacture representation where none exists. That judgment stays with the reviewer, which is the whole point.

The three-stage pipeline and distribution matching

The MedPerf data preparation stage is where distribution matching actually happens, and it is easy to underrate. A model trained on one institution’s DICOM conventions will silently misbehave on another’s unless the inputs are normalized the same way: pixel spacing, orientation, windowing, intensity ranges, and label taxonomy all have to line up. The preparation container encodes that normalization once, as a versioned artifact, so that a discrepancy in results reflects a real distribution difference rather than a preprocessing accident.

This matters for a reason that has little to do with MedPerf specifically and everything to do with how clinical imaging models fail. Most apparent “performance drops” at a new site are preprocessing mismatches, not model weakness. When you separate the preparation stage into its own pinned component, you make that failure class visible and debuggable instead of letting it contaminate the score. It is the same discipline we describe for logging what feeds a model in W&B Reports for Clinical Imaging Validation: the evidence is only as trustworthy as your ability to say exactly what went into it.

The metrics stage is similarly pinned. Whether the pack reports Dice, Hausdorff distance for segmentation, or the detection metrics discussed in mAP50 vs mAP50-95 for a clinical imaging validation pack, the computation is identical across sites. A reviewer can reproduce it because the metric definition is a fixed artifact rather than a paragraph in a paper.

How do site-stratified MedPerf results enter a validation pack?

This is where MedPerf stops being a benchmarking curiosity and becomes procurement infrastructure. A clinical-grade medical imaging AI validation engagement produces a validation pack — the assembled evidence a site reviewer adjudicates before deployment. MedPerf is one mechanism for generating a specific slice of that pack: the distribution-matched, site-stratified performance evidence.

Here is the concrete decision surface a reviewer works through.

What a reviewer reads from a MedPerf strata table

Question the reviewer asks	What the strata table shows	Evidence class
Does the model hold on my scanner mix?	Per-site rows filtered to matching vendor/protocol	benchmark (metrics reproducible at that site)
How wide is the performance spread across sites?	Min/max/spread across all rows, not the mean	benchmark
Am I represented in the cohort at all?	Presence or absence of a site resembling mine	observed-pattern (reviewer judgment)
Was a low score a real drop or a preprocessing bug?	Data-prep container version + normalization spec	benchmark (versioned artifact)
Can I reproduce this number myself?	Pinned model + prep + metrics containers	benchmark (containerized, re-runnable)

The ROI is direct: site-stratified results let a reviewer confirm the model holds on their own distribution before procurement, compressing an evaluation cycle that otherwise re-litigates aggregate metrics at every site. Federated scoring also removes a common weeks-long blocker — the data-transfer and de-identification overhead of shipping a de-identified test set out of the enclave. When the data never leaves, that whole workstream evaporates. Building this kind of reproducible, reviewer-facing reliability evidence is the throughline of our work on production AI reliability.

How does keeping data inside the enclave interact with HIPAA/GxP evidence?

Running evaluation inside the site’s enclave means PHI stays where it already lives, under controls the site has already validated. That is more than a convenience. It changes the compliance posture of the evaluation itself: there is no new data-egress event to justify, no external processing agreement to negotiate for the test set, no de-identification pipeline whose own validity you would then have to defend.

But keeping data in place is a data-governance property, not a completeness claim. MedPerf gives you performance evidence generated under the site’s existing controls; it does not by itself produce the access logs, change-control records, and process documentation a GxP or HIPAA audit expects. Those intersect with the workflow-evidence concerns of clinical AI governance, and they have to be assembled alongside the MedPerf output rather than assumed from it. The federated design removes one common evidence gap; it does not close all of them.

Where does a MedPerf benchmark stop and a regulatory submission begin?

This is the boundary most worth stating plainly, because conflating the two is how teams over-promise. A MedPerf evaluation is evidence — reproducible, site-stratified performance measurement. A regulatory submission is an argument built on evidence, and it demands categories of proof MedPerf never touches.

MedPerf does not establish clinical validity in the sense a regulator means — that a performance metric maps to a patient-relevant outcome. It does not cover the full quality-management-system documentation, the intended-use statement, the risk analysis, the human-factors and usability evidence, or the labeling. A strong MedPerf strata table is a genuine input to a submission, not a substitute for one. Treating it as the latter is a category error that a reviewer will catch immediately.

What still has to be generated outside MedPerf

• Clinical-outcome linkage — evidence that the measured metric corresponds to a decision that helps patients.
• QMS and process documentation — design controls, change management, traceability.
• Risk management — hazard analysis, mitigations, residual-risk justification.
• Prospective or reader-study evidence — where the intended use requires it.
• Post-market surveillance plan — how drift will be caught after deployment, which connects to the broader monitoring discipline we cover across the production-AI reliability work.

MedPerf earns its place by doing one thing well: producing distribution-matched, reproducible, site-stratified performance evidence without moving data. The discipline is knowing where that evidence ends.

FAQ

How does MedPerf actually work?

MedPerf, coordinated under MLCommons, ships a containerized model to each clinical site and runs three pinned stages there — data preparation, inference, and metrics — while the data stays inside the site’s enclave. Only metrics return to the aggregator. In practice you walk away with a table of per-site results, each carrying that site’s scanner mix and patient distribution, rather than one pooled score.

Why does federated evaluation produce more portable validation evidence than a pooled benchmark AUC?

A pooled AUC averages over a distribution the reviewer did not choose and cannot inspect; a 0.93 average can hide a 0.78 site. Federated evaluation returns site-stratified rows, so a prospective customer can read the rows that resemble their own setting. That is why per-site evidence carries meaning from one customer to the next while a pooled number describes a population no single site belongs to.

What role does the MedPerf data preparation and metrics pipeline play in matching a site’s distribution?

The preparation container normalizes local DICOM conventions — pixel spacing, orientation, windowing, label taxonomy — as a versioned artifact, so a score difference reflects a real distribution difference rather than a preprocessing accident. The metrics container pins the scoring definition identically across sites. Together they make distribution differences visible and reproducible instead of contaminating the number.

How do site-stratified MedPerf results enter a clinical imaging validation pack?

They form the distribution-matched, site-stratified performance slice of the validation pack a reviewer adjudicates before deployment. A reviewer filters the strata table to rows matching their scanner mix, reads the spread rather than the mean, and confirms the model holds on their distribution before procurement — compressing an evaluation cycle that would otherwise repeat at every site.

How does keeping data inside the site enclave interact with HIPAA/GxP evidence requirements?

Evaluating inside the enclave keeps PHI under controls the site has already validated, removing the data-egress, external-processing, and de-identification workstreams a shipped test set would require. It is a data-governance property, not a completeness claim: the access logs, change control, and process documentation a HIPAA/GxP audit expects still have to be assembled alongside the MedPerf output.

Where does a MedPerf benchmark stop and a regulatory submission begin?

MedPerf produces reproducible performance evidence; a submission is an argument built on evidence that also needs clinical-outcome linkage, QMS documentation, risk analysis, human-factors evidence, and labeling. A strong strata table is a genuine input to a submission, never a substitute for one, and treating it as the latter is a category error a reviewer catches immediately.

What are the practical limits of MedPerf — what evidence still has to be generated outside it?

MedPerf does not establish clinical validity in the regulatory sense, does not cover QMS or risk documentation, and does not manufacture representation for a site absent from the evaluation cohort. Clinical-outcome linkage, prospective or reader-study evidence where required, and a post-market surveillance plan all have to be generated outside it. Stratification lets a reviewer decide whether they are represented; it does not create representation where none exists.

The useful question to leave with is not “what score did the model get” but “can this reviewer, on their own scanner mix and their own patient distribution, reproduce a number they are willing to sign against” — and MedPerf is the mechanism that keeps that question answerable without the data ever leaving the room.