A validation pack does not go stale because the evidence was wrong. It goes stale because nobody wrote down which model build produced it. The moment a perception model is retrained, the pack’s numbers describe an artefact that is no longer the one shipping — and unless each evidence surface carries its own provenance, the reviewer has no way to tell which pages still hold.
That is the versioning problem, and it is separate from the question of whether a change requires a re-run. Deciding which changes force new tests is a triage exercise; keeping the pack current is a bookkeeping discipline that makes the triage decision legible to somebody who was not in the room. Both are needed. Only one of them is usually built.
What “keeping the pack current” means in practice
Two failure patterns dominate. The first is full regeneration: every release, the team re-runs the entire suite and rebuilds the document, which is expensive and — worse — tells the reviewer nothing about what moved. The second is patching in place: the numbers are updated, the file name stays the same, and the pack silently becomes a composite of results from three different model builds.
A perception validation pack is current when every evidence surface in it names the model artefact, dataset snapshot and test-suite revision that produced it, and the pack carries an explicit delta against the previous submission. Nothing else counts as current. A pack dated last Tuesday with no artefact hashes is not current; it is undated evidence with a recent timestamp on the cover.
The practical consequence is that versioning has to be decided before the retrain. Per evidence surface, the team writes down the rule: does a change of this kind invalidate the surface, degrade it, or leave it intact? Writing that rule down in advance turns a review argument into a lookup. Deciding it during review means the reviewer watches the team reason about its own evidence in real time, which is the single most reliable way to lose a release window.
The three-part version key
The pack does not have one version number. It has a version key with three independent components, and any one of them changing is a fact the reviewer needs.
| Component | What it pins | Typical identifier | What changes it |
|---|---|---|---|
| Model artefact | The weights and the export that ran | Content hash of the ONNX or TensorRT engine, plus the training run ID | Retrain, fine-tune, quantisation, re-export, compiler or TensorRT version bump |
| Dataset snapshot | The evaluation data, not the training data | Immutable snapshot ID with slice manifest and label-schema version | Re-annotation, added slices, label taxonomy edits, corrected ground truth |
| Test-suite revision | The harness that produced the numbers | Git SHA of the harness plus scenario-catalogue version | New scenarios, changed thresholds, metric-definition changes, harness bug fixes |
The reason to separate them is that the three change at different rates and invalidate different things. A harness bug fix invalidates old numbers without touching the model. A re-annotation pass changes the dataset snapshot and makes historical comparisons apples-to-oranges even when the model is byte-identical. In our experience, the dataset snapshot is the component teams forget to pin, and it is the one that most often produces an unexplainable metric shift two releases later.
One practical note on tooling: an experiment tracker such as MLflow will happily record all three if the pipeline is instrumented to log the evaluation-data snapshot ID alongside the run, but most pipelines log only the training inputs. That gap is worth closing before the pack matters.
Re-run, carry forward, or retire: a per-surface rule
Each evidence surface gets one of three verdicts against a given change. The verdicts are decided once as policy and applied mechanically thereafter.
- Re-run — the surface’s numbers are a function of the changed component. Any weight change re-runs the production-behaviour and per-class recall surfaces, full stop.
- Carry forward with provenance — the surface is independent of the change and is reproduced verbatim, still stamped with its original version key. Scope and ODD boundary statements usually carry forward across a pure retrain. So does the ownership and rollback section, unless the owner changed.
- Retire and replace — the surface no longer describes the system at all. A sensor-stack swap retires the calibration evidence rather than degrading it; there is nothing to re-run because the old configuration does not exist.
A fourth state is worth naming explicitly because reviewers ask about it: degraded. The surface is still valid but its confidence has narrowed — for example, a scenario suite that covered the old ODD now covers a subset of the widened one. Degraded surfaces are carried forward with a stated caveat, not silently re-labelled as intact.
Why does a sensor or ODD change invalidate more than a retrain?
Because a retrain moves one component of the version key while a sensor, calibration or ODD change moves the ground the whole pack stands on. A retrain leaves the input distribution and the operating envelope intact, so surfaces about scope, coverage and deployment posture survive; only the behaviour measurements move. Change the camera, the intrinsics, or the declared operating domain, and the dataset snapshot stops representing production input — which means the coverage argument, the slice analysis and the drift baselines all lose their footing simultaneously. That is the case where full regeneration is the cheaper honest answer, and it is one of a small number of triggers that should be written into the versioning policy as non-negotiable: sensor hardware or firmware change affecting the image formation path, calibration re-derivation, ODD boundary movement, label-taxonomy change, and a harness change that alters how a reported metric is computed.
The change delta is the document the reviewer actually reads
Once surfaces are versioned, the delta becomes possible — and the delta is what collapses re-review scope. When each evidence surface is independently versioned and accompanied by a stated delta, the reviewer re-examines only the surfaces that moved; an undated single-snapshot pack re-opens the full review at every submission. That difference shows up as a measurable quantity: re-review scope per model update, counted as evidence surfaces re-examined against total surfaces in the pack. Track it across submissions and you can see whether the versioning discipline is holding.
A usable delta has five entries and fits on one page:
- Version key diff — old and new model artefact, dataset snapshot and test-suite revision, side by side.
- Change classification — what kind of change this was (retrain, threshold tweak, re-export, sensor change) in the vocabulary the policy already defines.
- Surface verdict table — every surface with its verdict: re-run, carried forward, degraded, retired.
- Metric movement — for re-run surfaces, previous value, new value, and whether the movement crosses a pre-declared acceptance bound.
- Open items — anything the team knows is unresolved, stated before the reviewer finds it.
The fifth entry is the one teams cut. It is also the one that buys credibility, because a reviewer who finds an unstated gap starts auditing the rest of the pack rather than reading it.
For how the pack is assembled in the first place — which evidence surfaces exist, and which reviewer question each one answers — see our worked treatment of the automotive perception validation evidence package. The versioning rules here sit on top of that structure; they do not replace it. The broader engineering context for perception systems of this kind lives on our computer vision practice page.
Keeping drift evidence live without overclaiming
Drift monitoring is the one surface that changes continuously between releases, which makes it awkward under a version key. The temptation is to leave the dashboard live and link to it from the pack. That is fine for engineering visibility and wrong for the pack, because a live link means the evidence a reviewer approved is not the evidence a later reader sees.
The workable pattern is a snapshot plus a pointer: the pack contains a frozen drift summary stamped with its own window and version key, and a note that the live monitor continues under the same condition tags. The frozen summary is the evidence; the live monitor is the operational posture. Keeping them distinct also avoids the more serious misread — that continuous monitoring constitutes ongoing acceptance. It does not. A monitored model is a model whose behaviour is being watched, not a model whose approval renews itself.
Regulated imaging domains draw this line harder still. Clinical imaging teams operate under formal change control, where a model change is a documented event with a pre-agreed re-validation obligation rather than a release-cadence decision; the automotive release rhythm has more freedom and correspondingly more room to lose track. The versioning key is the same in both. The tolerance for missing it is not.
Frequently Asked Questions
What does keeping a perception validation pack current as the model updates mean in practice?
Model updates invalidate portions of your perception validation pack incrementally, demanding a structured refresh protocol that preserves unchanged test results while revalidating only affected inference paths. It means every evidence surface in the pack carries the model artefact, dataset snapshot and test-suite revision that produced it, and each submission includes an explicit delta against the previous one. Currency is a provenance property, not a recency property — a pack rebuilt yesterday without artefact identifiers is not current.
Which evidence surfaces must be re-run after a model update, and which can be carried forward with provenance intact?
Surfaces whose numbers are a function of the model weights — production-behaviour measurements, per-class recall, threshold-dependent precision — are re-run on any weight change. Scope and ODD boundary statements, and ownership and rollback sections, normally carry forward with their original version key attached. Anything in between is carried forward as degraded with a stated caveat rather than presented as intact.
How do we version the pack against the model artefact, dataset snapshot and test suite so a reviewer can tell which build each result describes?
Use a three-part version key: a content hash of the exported artefact plus its training run ID, an immutable evaluation-dataset snapshot ID with slice and label-schema versions, and the harness Git SHA plus scenario-catalogue version. Stamp the key on each surface rather than only on the cover page, because the three components change at different rates and invalidate different evidence.
What belongs in a change delta so the reviewer re-examines only what changed rather than the whole pack?
Five entries: the version key diff, the change classification, a per-surface verdict table (re-run, carried forward, degraded, retired), metric movement against pre-declared acceptance bounds for re-run surfaces, and open items the team already knows about. The open-items entry is the one most often omitted and the one that most protects the rest of the pack from being audited line by line.
How do sensor-stack, calibration or ODD changes differ from a pure retrain in terms of what they invalidate?
A retrain moves the weights but leaves the input distribution and operating envelope intact, so coverage and scope surfaces survive and only behaviour measurements are re-run. A sensor, calibration or ODD change moves what the data represents, which invalidates the coverage argument, the slice analysis and the drift baselines at once — often making full regeneration the cheaper honest option.
What triggers should force a full pack regeneration rather than an incremental update?
Sensor hardware or firmware changes affecting image formation, calibration re-derivation, ODD boundary movement, label-taxonomy changes, and harness changes that alter how a reported metric is computed. Each of these breaks comparability across the whole pack rather than one surface, so write them into the versioning policy as non-negotiable regeneration triggers before the next retrain.
How do we keep drift-monitoring evidence in the pack live between releases without implying regulatory acceptance?
Freeze a drift summary with its own measurement window and version key inside the pack, and reference the live monitor separately as operational posture. The frozen snapshot is the reviewed evidence; the live monitor is not a renewing approval, and the pack should say so in those terms.
Where the discipline is usually lost
Not at the first release. The first pack is careful, because everyone is watching. The discipline is lost at the second or third update, when the change looks small and stamping the version key feels like overhead. That is precisely when it pays — the small change is the one nobody remembers six months later, and the one that makes a metric shift inexplicable.
The open question we keep returning to is how granular the surface list should be. Too coarse, and every change re-runs half the pack; too fine, and the delta table becomes a spreadsheet nobody reads. We have not found a rule that transfers cleanly between programmes, and the right granularity may simply be a function of how often the model retrains.
