A validation pack answers whether a model performs acceptably against a defined intended use, on defined data, under defined acceptance criteria. An evidence pack answers whether an auditor can trace that conclusion to a signed-off, change-controlled, per-step record of the workflow actually running. Those are two different questions, produced by two different groups, on two different clocks — and the most common failure we see in regulated AI deployments is treating them as one deliverable produced once by whichever team finishes first.
The symptom is easy to spot. Someone copies the model performance tables out of the validation report, pastes them into the compliance binder under a heading called “Validation”, and the regulated step is declared covered. Six months later the model is retrained, the binder still shows the old tables, and nobody can say which validated conclusion was in force for the batch released last March.
What does the relationship between the evidence pack and validation evidence mean in practice?
It means the evidence pack cites validation evidence rather than containing it. The validation pack is a dated conclusion about a model version against an intended-use statement. The evidence pack is a per-step map of the regulated workflow, and each regulated step that depends on model output carries a pointer: which validated conclusion applies here, which model version it covers, which protocol produced it, who signed it, and over what date range it was in force.
That pointer is the interface. Get it right and audit prep pulls validation conclusions by reference instead of re-deriving them — a structured citation per regulated step rather than a multi-week evidence hunt across ML tracking systems and QA document control. Get it wrong and you have good validation results with no defensible trail connecting them to the workflow that actually ran.
The distinction matters because the two artefacts also fail differently. A validation pack fails when the acceptance criteria were poorly chosen or the test data did not represent the deployed population. An evidence pack fails when the criteria were fine but nobody can prove which version of them governed a given regulated action on a given date. Fixing the first is a modelling problem; fixing the second is a records problem, and it cannot be fixed retroactively.
Who owns what, and who signs each
The ownership split is the part teams argue about, usually because both groups are half right. QA owns the pack because it is a quality record. The ML team owns validation because they ran it. Neither statement resolves what happens when the model changes.
| Dimension | Validation pack | Evidence pack |
|---|---|---|
| Question it answers | Does this model version meet acceptance criteria for its intended use? | Can an auditor trace each regulated step to a governed, signed record? |
| Primary owner | Validation / CSV function, with ML engineering supplying protocol execution | Quality function, with the workflow owner supplying step-level artefacts |
| Signature | Protocol approver and validation lead | Change-control approver per regulated step |
| Unit of scope | One model version against one intended-use boundary | One regulated workflow, step by step |
| Lifecycle trigger | Model change, threshold retune, data-distribution shift | Any change to a regulated step, including non-model changes |
| Time semantics | A dated conclusion | A timeline of which conclusion was in force when |
| Failure mode | Criteria do not represent deployed conditions | Conclusion exists but cannot be tied to the running workflow |
Read the last two rows together, because that is where the artefacts genuinely diverge rather than merely differ in format. Validation produces a point in time. The pack has to produce an interval. Every validated conclusion needs an effective-from date and, eventually, an effective-to date, and every regulated step needs to resolve to exactly one conclusion for any date an auditor picks.
By reference, by summary, or by inclusion?
All three appear in real packs, and the choice is not stylistic.
- By reference — a structured pointer: protocol ID, report ID, model version, effective date range, approver, and a resolvable location in the controlled document system. This is the default for every regulated step. It is the only form that stays correct when the underlying validation report is superseded.
- By summary — a short, non-authoritative restatement of the conclusion and the acceptance criteria that were met, placed next to the reference so the auditor does not have to open three systems to follow the workflow walk. A summary is a navigation aid. It must be marked as such, and it must never be the thing that gets updated when the conclusion changes.
- By inclusion — the full validation report bound into the pack. Appropriate only where the validation artefact is itself frozen and site-specific, such as an installation qualification against one site’s instrument fleet. Inclusion of a model-performance report is almost always a mistake, because it creates a second copy that drifts silently.
The failure mode with pasted results is not that the numbers are wrong on the day they are pasted. It is that the paste has no lifecycle. In our experience, the giveaway during a mock audit is a performance table in a binder with no protocol identifier and no version stamp — at which point the auditor stops asking about model performance and starts asking about document control, which is a much worse conversation to be having.
What retraining does to both artefacts
Retraining is the divergence point, and it is where the interface earns its cost. A new model version invalidates a validation conclusion for the scope it covered; it does not automatically invalidate every regulated step in the workflow.
The scope of re-validation is bounded by what the pack already records. If each regulated step declares which validated conclusion it depends on, a model change tells you immediately which steps are affected and which are not — steps that consume only deterministic outputs, human-adjudicated results, or a different model are untouched. Without that mapping, the conservative default is to re-validate the whole workflow, and that default is expensive enough that teams postpone retraining they should be doing.
Threshold retuning deserves its own note. A retuned decision threshold usually leaves the model weights untouched, which tempts teams to treat it as a configuration change outside validation scope. It is not. The threshold is part of the acceptance criteria, so retuning it produces a new dated conclusion and a new effective-from date on every step that depends on it. We treat threshold values as versioned validation inputs for exactly this reason.
Showing an auditor which validated model version was in force for a given clinical or manufacturing step on a given date is then a lookup rather than an investigation: the step’s record names the model version and the run record carries the timestamp, and the pack’s effective-date table resolves the two. The mechanics of capturing that per-step record — access trails, change-control sign-offs, lineage — are covered in the section-by-section anatomy of a HIPAA / GxP evidence pack, and the broader question of what makes a governed AI deployment defensible sits in our work on AI governance and trust.
What travels between sites and what does not
The pack-to-validation mapping is one of the more portable parts of a regulated deployment. The structure — which step cites which conclusion, in what form, with what date semantics — is invariant, and sites that reuse an established mapping stop re-litigating acceptance criteria at every audit cycle.
What is not portable is the resolution layer. Document identifiers live in a local quality management system. Approvers are local people holding local delegations. Site-specific qualification evidence, such as validation against a particular scanner fleet or line equipment, has to be re-established rather than referenced. The practical rule: the citation pattern copies, the citation targets do not.
Frequently Asked Questions
What does the relationship between the evidence pack and validation evidence mean in practice for a regulated AI workflow?
The evidence pack points at validation conclusions rather than reproducing them. Each regulated step that consumes model output carries a structured citation naming the model version, the protocol and report that validated it, the approver, and the date range over which that conclusion was in force. The validation work stays where it was produced; the pack supplies the traceability layer around it.
What does the validation pack own versus what the evidence pack owns, and who signs each?
The validation pack owns the question of whether a model version meets defined acceptance criteria for a defined intended use, and it is signed by the protocol approver and validation lead. The evidence pack owns per-step traceability across the whole regulated workflow, and each regulated step carries a change-control sign-off from its own approver. Two owners, two signature chains, one interface between them.
How should a regulated step in the evidence pack cite validation evidence — by reference, by summary, or by inclusion?
By reference, as the default: a resolvable pointer with protocol ID, report ID, model version, effective dates and approver. A short summary alongside the reference is useful as a navigation aid for the auditor, provided it is clearly non-authoritative. Full inclusion is appropriate only for frozen, site-specific artefacts such as installation qualification against local equipment.
What happens to both artefacts when the model is retrained or a threshold is retuned, and how much re-validation does that trigger?
Retraining produces a new dated validation conclusion and closes the effective window of the previous one. Re-validation scope is bounded by the pack’s dependency map: only steps that cite the affected conclusion need attention. Threshold retuning counts as a change to acceptance criteria, so it also produces a new conclusion even when the model weights are unchanged.
How do you show an auditor which validated model version was in force for a given clinical or manufacturing step on a given date?
The step’s execution record names the model version and timestamps the run; the pack’s effective-date table maps that version to the validation conclusion governing it. Resolving the two is a lookup. If either half is missing — an unversioned run record, or a conclusion with no effective dates — the answer becomes reconstruction, which is what auditors read as a control weakness.
What are the common failure modes when validation results are pasted into a compliance binder instead of linked?
The pasted copy has no lifecycle: it does not update when the report is superseded, it usually loses the protocol identifier and version stamp, and it creates a second source of truth that silently diverges. The audit consequence is worse than the technical one — a performance table with no traceable origin turns a model-performance question into a document-control finding.
Which parts of this mapping are invariant across sites and which must be re-established per site?
The structure is invariant: which step cites which conclusion, the citation form, and the effective-date semantics. The targets are per-site: document identifiers in the local quality system, named approvers holding local delegations, and any qualification evidence tied to that site’s equipment. Reusing the pattern is what stops each site renegotiating acceptance criteria from scratch.
The open question we keep returning to is how far the effective-date mapping can be generated from the deployment itself — emitted by the pipeline as a by-product of running — rather than maintained as a parallel document. Every manual step in that mapping is a place where the pack and the workflow can quietly disagree.
Building your validation evidence strategy
Most teams discover that validation evidence and evidence packs serve complementary roles once they map each document to its intended audience. The teams that do tend to ship the boring, correct version first.
