A validation-set construction protocol is a document written before any performance number exists. It states who was included, who was excluded and why, which scanners and acquisition protocols are represented, how studies were partitioned at the patient level, and what disease prevalence the cohort carries relative to the site the model will run in. Most teams do not have this document. They have split logic in a notebook, a scanner list in a slide, and an AUC.
That gap is not a documentation nicety. It is the reason the same cohort conversation gets re-litigated at every hospital procurement.
Why is the construction protocol treated as plumbing?
Because the metric feels like the deliverable. The validation set is the thing you build in order to produce the number, so it inherits the status of infrastructure: necessary, uninteresting, finished once it works. Under that framing, the protocol is an implementation detail — recoverable from code if anyone ever asks.
Someone always asks, and code is the wrong answer. A clinical reviewer at a deploying site is not auditing your pipeline. They are trying to decide whether a sensitivity of 0.91 has any bearing on their patients, their GE and Siemens mix, their contrast protocols, their case load. To make that decision they need the population the number was measured on, described in prose they can compare against their own. A notebook cell that reads train_test_split(groups=patient_id) tells them nothing they can act on, even though it encodes a decision — patient-level partitioning — that materially changes whether the number is honest.
Treating the protocol as an artefact means one thing in practice: it is authored before the numbers exist, reviewed as a document, versioned, and shipped. Writing it first is what keeps it truthful. A protocol reconstructed after a favourable result has already been seen is a rationalisation, and experienced reviewers read it as one.
What the protocol has to state to be adjudicable
The test is not completeness for its own sake. Each element below exists because a reviewer can reach a different conclusion depending on its answer.
| Element | The reviewer question it answers | What its absence costs |
|---|---|---|
| Inclusion and exclusion criteria, with rationale per exclusion | Which patients could enter this cohort at all? | Silent exclusions read as cherry-picking; unstated rationale invites the assumption of one |
| Scanner and vendor mix | Is my fleet represented, and at what share? | Reviewer cannot estimate transfer to their hardware |
| Acquisition-protocol coverage | Were slice thickness, contrast phase, sequence and dose settings like mine present? | Performance may hinge on a protocol the site does not run |
| Patient-level partitioning | Could studies from one patient appear on both sides of the split? | Leakage inflates the metric; the number is not recoverable |
| Prevalence in the cohort vs the deploying site | Does the reported PPV apply to my population? | Threshold and workflow decisions get made on the wrong number |
| Named cohort gaps | Which scanner classes or subgroups are not covered? | The gap surfaces at procurement instead of scoping |
Two of these rows carry more weight than the rest.
Patient-level partitioning is where the most common unrecoverable failure lives. Clinical imaging datasets contain multiple studies per patient — follow-ups, contrast phases, repeat acquisitions after motion. A study-level random split places correlated images from the same patient on both sides of the boundary, and the model is partly scored on anatomy it has already memorised. The resulting metric is not slightly optimistic; it is measuring the wrong thing, and no amount of downstream slicing repairs it. Stating the partitioning unit explicitly is a two-line commitment that lets a reviewer close the question in seconds instead of asking for code.
Prevalence is where the arithmetic bites hardest. Sensitivity and specificity are prevalence-invariant; positive predictive value is not. A cohort enriched to 30% positives — the usual consequence of assembling enough pathology to measure anything — will report a PPV that has no relationship to a screening population running at 1–2%. The honest form is to document the cohort prevalence, state the deploying site’s expected prevalence as a separate figure, and recompute PPV at that prevalence rather than shipping the enriched one. Reviewers who catch this themselves stop trusting the rest of the pack, which is a rational response.
Distribution matching is a claim about coverage, not similarity
“Distribution-matched” is often used loosely to mean the validation cohort resembles the target population. That framing is too weak to review. The useful version is a coverage statement per stratum: this scanner class appears at this share, this acquisition protocol at this share, this age band and this body habitus at these shares — measured against the deploying site’s own distribution where that distribution is known.
The consequence is more interesting than the arithmetic. Once coverage is stated per stratum, portability stops being a matter of assertion. A reviewer can lay their own scanner mix beside the documented one and estimate the performance gap before deployment rather than discovering it after. Where a stratum is thin, the protocol says so with a case count, and thin strata get named as gaps rather than averaged into a pooled number that hides them.
Naming an uncovered scanner class in the protocol feels like handing the reviewer a weapon. In our experience it does the opposite. An unnamed gap becomes a late-stage discovery, and late-stage discoveries trigger renegotiation and unplanned re-validation cycles; a named gap becomes a scoping item with a decision attached — accept the limitation, collect a supplementary cohort, or exclude that scanner class from the deployment. Across the validation engagements we have run, gaps disclosed in the construction protocol are the ones that get resolved cheaply (observed pattern across TechnoLynx engagements; not a benchmarked rate).
Where the protocol stops
The construction protocol answers one question: what population was this measured on, and how was it assembled. It does not adjudicate labels — who read each study, under what information conditions, and how reader disagreements were resolved belongs to a separate ground-truth procedure with its own error rate and audit trail. It also does not defend intended use, risk classification or device claims; that is a regulatory submission’s job, aimed at a different audience making a different decision.
Keeping those boundaries clean is what makes the protocol portable. It also has to stay consistent with the data-provenance record — the cohort was assembled from PHI under some lawful basis, and the de-identification and access history behind it live in the governance workflow evidence rather than here. A construction protocol that contradicts the provenance record is worse than one that is silent.
The structural argument for why an evidence pack travels between hospital sites while its numbers do not is developed in our work on production AI reliability, and the full document anatomy of the pack this protocol opens is set out in what a clinical imaging validation pack contains beyond the benchmark report.
Frequently Asked Questions
What does it mean in practice to treat the validation-set construction protocol as an artefact rather than an internal implementation detail?
The Validation-Set Construction Protocol Is Itself makes this clear: it means the protocol is authored before any performance number exists, reviewed as a standalone document, versioned, and shipped with the evidence pack. The practical test is whether a site reviewer could read it without access to your code or your team and reach an independent judgement about the cohort. If the answer is no, it is still plumbing., all five, plus a rationale for every exclusion and an explicit list of known cohort gaps. Each element exists because a reviewer can reach a different conclusion depending on its answer — an unstated partitioning unit leaves leakage open, an unstated prevalence leaves PPV unusable. Completeness here is not thoroughness for its own sake; it is what converts open-ended questioning into a bounded document review.
How does the protocol prevent the leakage and distribution-matching failures that make a benchmark number unusable at a new site?
Leakage is prevented by committing to the partitioning unit in writing — patient-level, not study-level — so correlated studies from one patient cannot straddle the split. Distribution failures are surfaced by stating coverage per stratum with case counts rather than asserting the cohort “resembles” the target population, which lets a reviewer compare their own scanner and protocol mix directly.
How should documented prevalence in the validation cohort be reconciled with the deploying site’s prevalence, and what does that imply for reported PPV?
Document both figures separately and recompute positive predictive value at the deploying site’s expected prevalence. Sensitivity and specificity carry across; PPV does not, so an enriched research cohort will report a PPV far above what a screening population at 1–2% prevalence would see. Shipping the enriched figure without that recomputation is the fastest way to lose a reviewer’s trust in the rest of the pack.
Three principles for defensible hold-out sets
Treat your validation protocol as code: version it, peer-review sampling logic, and document every exclusion criterion before a single label is collected. Revisit it when your workload shifts.
