A validation pack whose evidence stops at deployment day defends a clinical-grade claim only on the day it was written. The fix is not a better monitoring dashboard. It is deciding, before the first site goes live, which distribution shifts are monitored against which validation-set strata, what the thresholds are, who adjudicates a flagged case, and how that adjudication is versioned back into the document the next reviewer reads.
Most teams do the opposite. Monitoring lands in an ops tool — uptime, p95 latency, a score-histogram panel in Grafana — and the pack stays frozen as a pre-deployment artefact. The two never meet. The gap is invisible at site one, because the people who ran the validation are still in the room and can produce a bespoke analysis on request. It becomes structural at site two, when a new reviewer asks how you know the model still works and the only readable answer is a set of retrospective numbers from a different scanner fleet.
What does post-deployment drift evidence entering the pack mean in practice?
It means the pack has a section that was designed to be filled in after go-live, and a mechanism that fills it. Concretely, four things are specified in advance:
- Monitored strata. Not “the input distribution” in the abstract, but named strata that already exist in the validation-set construction protocol — scanner vendor and model, acquisition protocol, slice thickness, site, and any patient subgroup the pack reports performance on separately.
- Signal type per stratum. Input-distribution shift, output-distribution shift, and measured outcome are different claims with different evidentiary weight, and the pack should not blur them.
- Thresholds and the response they trigger. A number that does nothing when crossed is telemetry, not evidence.
- Adjudication and write-back. Who reviews a flagged event, on what timeline, and which pack section their finding lands in — with a version stamp.
A drift signal is only interpretable if the stratum it moved in is a stratum the pack already reported performance on. That is the load-bearing sentence in the whole design. If monitoring reports “mean model score dropped 4%” and the validation report is organised by scanner vendor, nobody can convert the alert into a statement about clinical performance. The two structures have to be the same structure. This is why the monitoring plan is written against the validation-set construction protocol rather than against whatever fields the inference logs happen to expose — a dependency we develop further in the pack’s document anatomy and section ownership.
Which drift signals belong in the pack
The three signal classes answer different reviewer questions and carry different costs to obtain. Distinguishing them is also how you separate genuine drift from ordinary case-mix variation, which is expected and should not trigger anything.
| Signal class | What it detects | Reviewer question it answers | Evidence weight |
|---|---|---|---|
| Input distribution | Acquisition and population shift — new scanner, changed protocol, altered referral mix | “Is the model still seeing the data it was validated on?” | Cheapest and fastest; a leading indicator, not proof of harm |
| Output distribution | Change in prediction or score distribution not explained by input shift | “Is the model behaving differently?” | Strong signal, weak explanation; needs input-side context to interpret |
| Measured outcome | Model output versus adjudicated ground truth on a sampled subset | “Has clinical performance actually moved?” | Highest weight; requires reader time, so it is sampled, not continuous |
The practical arrangement we see work is a cheap continuous layer on the first two classes and a scheduled, sampled outcome-measurement layer on the third, with the sampling rate raised when an input or output alert fires. Outcome measurement is where the pack’s ground-truth adjudication protocol gets reused — the readers, the information conditions, and the disagreement-resolution procedure should be the same ones the ground-truth adjudication evidence section already documents. Reusing the protocol is what makes a post-deployment number comparable to a pre-deployment number instead of a differently-constructed one.
Case-mix variation is separated from drift by the same mechanism: because strata are named, a shift in the proportion of cases per stratum is reported as case mix, while a shift in performance within a stratum is reported as drift. Pooled monitoring cannot make that distinction at all, which is why pooled dashboards generate alerts nobody can act on.
Adjudication is the part that becomes pack content
An alert is not evidence. The record of what a named person concluded about that alert is evidence. The minimum adjudication record that belongs in the pack:
- The triggering signal, stratum, threshold, and timestamp
- The sampled cases pulled for review, and the sampling rule
- Who reviewed them, under what information conditions
- The finding: confirmed performance shift, case-mix artefact, data-pipeline defect, or unexplained
- The action taken — none, threshold revision, retraining, restriction of intended use, or regulatory notification
- The date the pack section was revised and by whom
Two of these deserve emphasis. “Data-pipeline defect” is a common finding in practice — a DICOM tag mapping changed, a preprocessing step silently altered windowing — and a pack that only offers “retrain” as a response misclassifies infrastructure bugs as model degradation. And the action record is what connects the pack to change control: any retraining or threshold change produces records that HIPAA and GxP change-control expectations already govern, so the adjudication record should be written once in a form both readers accept rather than twice in two systems.
The measure worth tracking here is detection-to-adjudication lag — the interval between a threshold being crossed and a documented finding existing. It is the specific quantity that post-market clinical-validation findings in AI-enabled medical devices tend to expose: not the absence of monitoring, but the absence of a documented response within a defined window. Tracked as a standing reliability measure alongside the count of monitored strata with defined thresholds, it turns a vague commitment to “monitor and retrain” into something a reviewer can inspect. (Framing based on patterns we see across engagements; the lag targets themselves are set per deployment, not from a published benchmark.)
Versioning across sites without breaking portability
The failure mode at scale is site-specific numbers overwriting portable structure. Site one’s scanner fleet produces a drift history; if that history becomes the post-deployment section, the pack you send to site two is now a document about somebody else’s hospital.
The split that holds up: the monitoring plan is portable — strata definitions, signal classes, threshold logic, adjudication roles, write-back procedure. The monitoring record is per-site and appended, never merged. Each site’s record carries its own version stamp and its own stratum table, and the pack’s index lists them. A reviewer at site three then reads one portable plan plus three worked instances of it, which is a stronger artefact than a single averaged history — it demonstrates the loop has run under different conditions. We treat the same portable-versus-site-specific distinction as the organising principle for the whole pack, not just its drift section.
Where this stops is the regulatory boundary. Drift evidence in the pack supports a claim about performance at deploying sites under a stated intended use. A confirmed shift that changes intended use, risk profile, or device claims is no longer a pack revision — it is a change notification or resubmission question, and the adjudication record’s action field is where that determination gets made explicitly rather than by omission. The line itself is covered in where the validation pack ends and a regulatory submission begins.
Specifying this loop is part of the broader reliability engineering we describe under production AI reliability, where drift thresholds and detection-to-adjudication lag sit as standing measures rather than incident-driven ones.
The uncomfortable question to ask of any pack before the first site goes live: if performance moved in one named stratum six months from now, which section of this document would change, who would change it, and how long would that take?
Frequently Asked Questions
What does post-deployment drift evidence entering the validation pack mean in practice?
Post Deployment Drift Evidence behaves predictably once you see the mechanism. The useful way to read Post Deployment Drift Evidence is this. It means the pack contains a pre-specified section that post-deployment findings are written into, plus the mechanism that writes them. In practice that is four decided items: which validation-set strata are monitored, which signal class is watched per stratum, what threshold triggers what response, and who adjudicates a flagged event and versions their finding back into the document.
Which drift signals belong in the pack — input distribution, output distribution, or measured outcome — and how are they distinguished from ordinary case-mix variation?
All three, kept separate because they carry different evidentiary weight: input shift is a cheap leading indicator, output shift is a strong signal with weak explanation, and sampled outcome measurement against adjudicated ground truth is the highest-weight claim. Case mix is distinguished from drift structurally — a change in the proportion of cases per stratum is case mix, while a change in performance within a stratum is drift.
How are monitored strata tied back to the validation-set construction protocol so that a shift is interpretable rather than just visible?
The monitoring plan reuses the construction protocol’s strata verbatim — scanner vendor and model, acquisition protocol, site, reported patient subgroups — rather than whatever fields the inference logs happen to expose. A signal is interpretable only if the stratum it moved in is one the pack already reported performance on; otherwise the alert cannot be converted into a statement about clinical performance.
Who adjudicates a flagged drift event, and what record of that adjudication belongs in the pack?
Adjudication roles are named in the monitoring plan before go-live, and outcome review reuses the same readers and information conditions as the original ground-truth adjudication protocol. The record carries the triggering signal and threshold, the sampled cases and sampling rule, the reviewers, the finding — confirmed shift, case-mix artefact, pipeline defect, or unexplained — the action taken, and the pack revision date.
How is the pack versioned as post-deployment evidence accumulates across multiple sites, without letting site-specific numbers overwrite the portable structure?
The monitoring plan stays portable and single; the monitoring record is per-site and appended, never merged into an average. Each site’s record carries its own version stamp and stratum table, so a later reviewer reads one plan plus several worked instances of it — which evidences that the loop runs under varied conditions rather than hiding that variation in a pooled history.
How does the drift section pre-empt post-market clinical-validation gaps seen in AI-enabled medical devices?
The recurring gap is not missing telemetry but the absence of a documented response within a defined window. Specifying thresholds, adjudication ownership, and a write-back path makes detection-to-adjudication lag a measurable quantity a reviewer can inspect, which is the thing post-market findings actually probe.
Where does drift evidence in the pack stop and a regulatory change notification or resubmission begin?
Pack drift evidence supports a performance claim at deploying sites under a stated intended use. When a confirmed shift touches intended use, risk classification, or device claims, it becomes a change-notification or resubmission question — and the adjudication record’s action field is where that determination is made explicitly rather than left to omission.
Three signals that drift evidence matters now
Drift evidence earns its place in the pack when stakeholders can trace a model behaviour change directly to a distribution shift, not just correlation. If Post Deployment Drift Evidence is on your roadmap, the next step is to map it onto your own constraints rather than copy a reference architecture.
