A perception model gets re-trained far more often than the argument for its safety changes. New data campaign, new sensor firmware, new backbone — and yet the reviewer’s standing questions are the same ones they asked last release. That mismatch is the whole story: a validation package should change at the speed of the reviewer’s questions, not at the speed of the model.
The naive rhythm treats every release as a fresh documentation exercise. Someone opens a blank template, writes sections in the order things happened to change this cycle, and hands over a document the reviewer has never seen before. The reviewer spends the first review round learning the artefact rather than assessing it. Then the next release arrives and the whole cost is paid again.
The alternative is not a better template. It is a decision about where the section headings come from.
What “travelling across model updates” actually means
A package travels when its structure is derived from the reviewer’s standing questions rather than from the current release’s change log. Scope of the operational design domain, hazard-to-test mapping, dataset and scenario provenance, metric definitions, acceptance thresholds, residual-risk statement — those headings are stable because the questions behind them are stable. A new backbone does not change whether a reviewer needs to know the ODD boundary. It changes what the evidence under that heading says.
So the package stops being a document that gets written and becomes a fixed skeleton with evidence slots. A model update re-populates the affected slots. Everything else carries forward, unchanged and already accepted.
The practical consequence is that the only genuinely new authoring work per release is the delta argument: what changed, which evidence slots that change invalidates, and what re-run results now sit in them. In our automotive validation work this is the section reviewers read first and the only one that is new prose. We build the invariant skeleton once at engagement start, exactly so that it never becomes negotiable later.
Two claims worth stating plainly, because they are the ones a reader can take away and test against their own release process:
- An invariant validation package structure reduces per-release documentation effort to the delta — re-running a fixed regression and scenario suite and re-populating the affected sections — rather than re-authoring the artefact end to end.
- A package whose section order is derived from the release change log cannot be diffed against its predecessor, which forces the reviewer into a full re-read and re-pays the entire justification cost every cycle.
Both are observed patterns from validation engagements rather than benchmarked figures; the mechanism is structural, but the size of the saving depends on how heavy your regression suite already is.
Which parts are fixed and which are slots
The distinction is not cosmetic. A section is invariant if the question it answers survives a model change; it is an evidence slot if the answer is release-specific.
| Package element | Invariant across releases? | What a model update does to it |
|---|---|---|
| ODD and scope-of-claim statement | Yes — until the claimed domain itself changes | Usually untouched; re-confirmed, not rewritten |
| Hazard-to-test mapping | Yes — owned by the safety argument, not the model | Untouched unless a new failure mode is identified |
| Metric definitions | Yes — definitions are fixed so numbers stay comparable | Untouched; changing a definition breaks the diff |
| Acceptance thresholds | Yes, with versioning | Carried forward; any change needs its own justification |
| Dataset and scenario provenance | Structure fixed, contents slotted | New manifest version and hash per data campaign |
| Per-scenario coverage results | Slot | Re-populated from the re-run suite |
| Degradation and edge-case behaviour | Structure fixed, contents slotted | Re-populated where the change plausibly affects it |
| Known limitations and residual risk | Structure fixed, statement slotted | Restated for this release, diffable against the last |
| Delta argument | New each release | This is the release’s actual authoring work |
The row that causes the most trouble in practice is metric definitions. Teams re-define a metric mid-programme — a different IoU threshold, a different way of counting occluded instances — and the release-over-release history silently stops meaning anything. If a definition genuinely has to change, it belongs in the delta argument with its own justification and a re-baselined comparison, not quietly inside a table.
How do you decide what a change invalidates?
You do not re-run everything, and you do not re-run only what feels affected. The workable rule is to trace the change through the mechanism, then re-run the slots that mechanism touches plus a fixed regression floor that runs regardless.
- A new training data campaign invalidates dataset provenance and any slice results the new data plausibly shifts. Coverage claims about slices the campaign did not touch carry forward with their prior evidence cited.
- New sensor firmware or a calibration change invalidates everything downstream of the input distribution — including edge-case and degradation slots — because the numbers previously recorded describe a different sensor configuration.
- A new backbone or architecture change invalidates all performance slots but usually leaves ODD, hazard mapping and metric definitions intact.
- A newly discovered failure mode is the one case that touches the invariant layer: it adds a hazard-to-test row, which is a structural revision, not a re-population.
That last case is the honest boundary. New ODD claims, a new sensor modality, or a new failure taxonomy entry legitimately force a structural change. Invariance is a property you should defend, not one you should fake — a package that quietly absorbs a widened ODD without a structural revision is worse than one that gets rewritten, because it now over-claims.
Why the delta argument is what earns the single review round
Sign-off compresses when the reviewer diffs against a package they have already accepted. That only works if three things are true: the section skeleton is unchanged, the metric definitions and thresholds are versioned so a comparison is meaningful, and the delta argument explicitly names which slots were invalidated and why the untouched ones remain valid.
The third point is the one teams skip. A delta that lists what was re-run is incomplete; the reviewer also needs the reasoning for what was not re-run. Without it they cannot tell the difference between a scoped re-validation and an omission, so they ask — and the round you were trying to avoid happens anyway. We treat “justification for carried-forward evidence” as a required part of the delta, alongside the new results themselves.
There is a scheduling benefit that matters as much as the review one. When documentation effort is proportional to the delta rather than to the artefact, it stops being the variable-length item on the critical path. Release cadence becomes predictable, which is usually the reason a Tier-1 programme cared about this in the first place.
This sits inside the broader question of what belongs in the artefact at all — we develop the section-by-section contents and the reviewer-question derivation in what a perception validation package contains beyond the internal test report, and the same fixed-skeleton discipline underpins how we approach production AI reliability more generally.
Frequently Asked Questions
What does it mean in practice for a perception validation package to travel across model updates without being rewritten?
Model updates destroy validation work unless the package architecture decouples test infrastructure from model-specific results. Perception Validation Package Survives behaves predictably once you see the mechanism. Perception Validation Package Survives rarely needs a long answer. Perception Validation Package Survives behaves predictably once you see the mechanism. The mechanics of Perception Validation Package Survives are worth stating plainly. Perception Validation Package Survives rarely needs a long answer. Perception Validation Package Survives behaves predictably once you see the mechanism. Perception Validation Package Survives rarely needs a long answer. Perception Validation Package Survives behaves predictably once you see the mechanism. The mechanics of Perception Validation Package Survives are worth stating plainly. Perception Validation Package Survives rarely needs a long answer. Perception Validation Package Survives behaves predictably once you see the mechanism. Perception Validation Package Survives rarely needs a long answer. Perception Validation Package Survives behaves predictably once you see the mechanism. With Perception Validation Package Survives, the detail that matters is this. The mechanics of Perception Validation Package Survives are worth stating plainly. How a Perception Validation Package Survives makes this clear: it means the section skeleton is fixed once — ODD scope, hazard-to-test mapping, dataset provenance, metric definitions, thresholds, residual risk — and a model release re-populates the evidence inside those sections instead of restructuring them. The reviewer opens the same artefact they accepted last time and reads a diff, not a new document., invariant: the scope-of-claim statement, hazard-to-test mapping, metric definitions and acceptance thresholds. Slots: dataset and scenario manifests, per-scenario coverage results, degradation and edge-case findings, and the residual-risk statement for this release. The delta argument is the only genuinely new authoring per release.
How do you decide which evidence a given model change invalidates, rather than re-running everything? Trace the change through its mechanism and re-run the slots it touches, on top of a fixed regression floor that runs every release regardless. A data campaign invalidates provenance and the slices it shifts; a firmware or calibration change invalidates everything downstream of the input distribution; a backbone swap invalidates performance slots but leaves the invariant layer alone.
What does the per-release delta argument need to contain for a reviewer to accept it without a full re-read? Three things: what changed, which evidence slots that change invalidated together with the re-run results now sitting in them, and — the part most often missing — the justification for why carried-forward evidence remains valid. Without that last piece the reviewer cannot distinguish scoped re-validation from omission.
What kinds of change legitimately force a structural revision rather than a re-population? A widened or shifted operational design domain, a new sensor modality, and a newly identified failure mode. The first two change the claim being made; the third adds a row to the hazard-to-test mapping. These are the cases where rewriting part of the skeleton is correct, and absorbing them silently would leave the package over-claiming.
How do you version thresholds, datasets, and metric definitions so releases can be diffed against each other? Each carries an explicit version identifier — dataset manifests with a content hash, metric definitions with a revision number, thresholds with the release in which they were set and the rationale. A change to any of them is raised in the delta argument with a re-baselined comparison rather than edited in place, which is what keeps the release-over-release history auditable.
If the structure is right, the interesting question stops being “how long will documentation take this release?” and becomes “which slots did this change actually invalidate?” — and that is a question a perception team can answer in an afternoon.n.
Why some packages survive model churn
Version-stable packages separate invariant architecture claims from model-specific benchmarks, so only the evidence layer needs refreshing. Revisit it when your workload shifts.
