Ask an imaging AI team for validation evidence and you will usually get a slide: one headline AUC, one confusion matrix from the public test set. That is what the training pipeline exports by default, so that is what gets packaged. Then the clinical reviewer asks the only question that matters first — which patients and which scanners is this number from? — and the pack has no answer.
A clinical-grade validation evidence pack is the document set that answers that question before it is asked. It has six sections, and each one exists because a reviewer has previously sent a pack back for its absence. This piece is the contents inventory: what goes in each section, what evidence backs it, and which sections carry forward into monitoring once the model is live.
What does the pack actually contain, section by section?
The short version, in the order a reviewer reads it:
| Section | What it holds | Evidence class |
|---|---|---|
| 1. Evaluation cohort definition | Case count, inclusion and exclusion rules, scanner and vendor mix, acquisition protocol distribution, patient demographic breakdown, site provenance | Documented dataset inventory |
| 2. Overall performance | Headline metrics with confidence intervals, on the named cohort only | Measurement on the stated cohort |
| 3. Per-stratum performance | The same metrics cut by scanner, protocol, site, and the demographic axes that matter clinically; weakest stratum named explicitly | Measurement, per subgroup |
| 4. Operating point and threshold rationale | The chosen threshold, the sensitivity/specificity trade it encodes, and who decided that trade and why | Design decision with a named owner |
| 5. Failure taxonomy and known limitations | Named failure modes with example counts, plus the cohorts and conditions the pack does not characterise | Reviewed error analysis |
| 6. Monitoring design | Which strata are watched in production, on what signal, at what cadence, and what the alert triggers | Deployment specification |
The pack’s job is not to make the model look good; it is to make the model’s boundaries legible. A reviewer who can see the edges will pilot inside them. A reviewer who cannot see the edges will assume the worst and ask for a re-run.
Section 1 — the cohort, documented rather than referenced
Naming a public dataset is not cohort documentation. The reviewer needs the distribution, because representativeness is a property of the distribution and not of the dataset’s reputation. In practice that means a table: how many cases per scanner vendor and model, per acquisition protocol, per site, per relevant demographic band, with the exclusions stated as rules rather than as a case count.
This is where most packs fail, and the failure is expensive. When the evaluation cohort is documented only by name, a pilot that surfaces one unlisted scanner generation invalidates the whole performance claim — not because the model necessarily degrades on it, but because nobody can say whether it does. That is the re-validation cycle a complete cohort section avoids.
Sections 2–4 — all three performance surfaces, not one
The answer to “overall metrics, per-stratum cuts, or threshold rationale?” is all three, and they are not interchangeable. Overall metrics tell the reviewer the ceiling. Per-stratum cuts tell them where the floor is. The threshold rationale tells them which of those two numbers the deployed system will actually produce, because the operating point — not the AUC — determines what a radiologist sees on a Tuesday afternoon.
We treat the weakest stratum as a headline, not a footnote. A pack that reports strong aggregate performance and quietly buries a thin-slice CT subgroup at ten points lower is the pack that generates a surprise in pilot. Reporting it up front converts the same fact into a scoping decision: exclude that protocol from the pilot, or gate it behind a second read. Both are acceptable. Discovering it in week three is not.
Where the buyer has their own image set, the strongest form of this section is per-stratum results computed on their images rather than ours. That is the measurement that removes the argument about transfer.
Section 5 — writing limitations so they strengthen the pack
There is a real fear here: that a documented failure taxonomy gives a reviewer ammunition. In our experience the opposite holds. Clinical reviewers assess model risk with or without your help; an explicit taxonomy is the difference between them assessing the risks you have measured and them imagining the ones you have not.
The section that works has three parts. First, named failure modes — motion artefact on a specific sequence, low-contrast lesions below a size threshold, a particular reconstruction kernel — each with an example count from the evaluation cohort rather than an adjective. Second, the conditions the pack does not characterise at all, stated as such. Third, for each named mode, what the deployed system does about it: a confidence gate, a routing rule, an exclusion, or nothing but a documented acceptance.
The pack also stops somewhere, deliberately. It presents engineering validation evidence — measured performance on a documented cohort under a stated protocol. It does not assert regulatory clearance, and it does not claim clinical accuracy in the sense a clinical study establishes. Saying so in the pack is not a hedge; it is what keeps the rest of the document credible.
Section 6 — what carries into production
Sections 1 and 3 are the ones with a second life. The cohort table becomes the reference distribution that production images are compared against, and the per-stratum results become the monitoring plan: the strata carrying the weakest measured performance are the strata that get the tightest thresholds and the shortest review cadence. That reuse is the practical argument for doing the stratification properly at validation time — it is the same work twice, or it is a monitoring design built on guesses.
Three things have to be added, because the pack cannot supply them:
- Input-distribution drift signals. Scanner software upgrades and protocol changes arrive without announcement. Monitoring watches the acquisition metadata mix, not only the model’s outputs.
- Outcome feedback wiring. Radiologist agreement, override rates, and downstream confirmation are the only signals that catch degradation the input distribution does not predict.
- An escalation path with an owner. An alert that reaches nobody with authority to pause a workflow is telemetry, not monitoring.
The pack itself is a scoped deliverable — we build it as part of a [production AI monitoring harness](Production AI Monitoring Harness) engagement, so the monitoring design in section 6 is specified by the same people who computed the strata in section 3. The broader question of how imaging AI moves from a research metric to a clinically reviewable deployment is the subject of our life sciences and medical imaging work, which covers the engagement shape this deliverable sits inside.
GxP-scoped versus internal workflow
Scope changes the pack’s weight, not its contents list. For a non-regulated internal workflow — triage prioritisation, worklist ordering, a research read — the six sections stand as written and the threshold rationale can be a short decision record.
Under GxP scope, the same sections need a documented trail rather than a stated conclusion: versioned datasets with access records, a change history on the evaluation code, signed-off threshold decisions, and traceability from each reported number to the run that produced it. The evidence does not change; its provenance requirements do. Teams that build the pack with versioned data snapshots and pinned evaluation code from the start absorb that difference cheaply. Teams that reconstruct it afterwards usually cannot.
Frequently Asked Questions
What does a clinical-grade validation evidence pack contain in practice, section by section? Six sections: evaluation cohort definition, overall performance, per-stratum performance, operating point and threshold rationale, failure taxonomy and known limitations, and post-deployment monitoring design. Each answers a question a clinical reviewer asks in that order, and a missing section is the usual cause of a returned pack.
How is the evaluation cohort documented — scanner mix, acquisition protocol, patient demographics — so a reviewer can judge representativeness? As a distribution table rather than a dataset name: case counts per scanner vendor and model, per acquisition protocol, per site, and per clinically relevant demographic band, with inclusion and exclusion stated as rules. Representativeness is a property of that distribution, so a reviewer cannot judge it from a citation alone.
Which performance results belong in the pack: overall metrics, per-stratum cuts, operating-point and threshold rationale, or all three? All three, because they answer different questions. Overall metrics give the ceiling, per-stratum cuts locate the floor, and the threshold rationale determines which of the two the deployed system actually produces. The weakest stratum belongs in the summary, not a footnote.
How should the failure taxonomy and known-limitation section be written so it strengthens rather than weakens the pack? Name each failure mode concretely with an example count from the evaluation cohort, state plainly which conditions the pack does not characterise, and pair each mode with the mitigation the deployment applies. Reviewers assess risk regardless; an explicit taxonomy means they assess measured risks instead of imagined ones.
What part of the pack carries forward into the post-deployment monitoring design, and what has to be added there? The cohort table becomes the reference distribution and the per-stratum results become the monitoring priorities, with the weakest strata getting the tightest thresholds. Three additions are needed in production: input-distribution drift signals from acquisition metadata, outcome feedback such as override and agreement rates, and an escalation path with a named owner.
Where does the pack stop — what does it deliberately not claim about regulatory status or clinical accuracy? It presents engineering validation evidence — measured performance on a documented cohort under a stated protocol — and says so explicitly. It does not assert regulatory clearance and does not substitute for a clinical study establishing clinical accuracy.
How does the pack change when the imaging deployment is GxP-scoped versus a non-regulated internal workflow? The contents list is identical; the provenance requirements are not. GxP scope adds versioned datasets with access records, change history on evaluation code, signed threshold decisions, and traceability from every reported number to the run that produced it — cheap if built in, often impossible to reconstruct later.
If your current validation evidence is one slide, the useful next step is not a better slide. It is answering, in writing, which scanners and which patients produced the number on it — and seeing how much of the pack that question already forces you to build.
Clinical Grade Validation Evidence: assembling the evidence pack
Notified bodies expect traceability from risk analysis through to test protocol design—gaps here delay certification more than any technical shortfall. Revisit it when your workload shifts.
