A perception validation pack asserts one thing: that a named model version behaved within measured bounds on defined operational conditions, with named owners and a rollback path. It does not assert that the resulting system is safe. Those are different claims, made by different people, answering different questions — and the failure mode this article is about is the quiet moment when a team stops distinguishing them.
It usually happens after success. The pack clears engineering review, the reviewer signs, and the document acquires a reputation. Six weeks later someone in a homologation meeting says “the validation pack covers that”, and nobody in the room objects. Nothing was falsified. The pack was simply cited in a role it was never scoped to fill.
Why does the pack get promoted to a safety argument?
Because it looks like one. A well-built pack has a scope section, defined operational conditions, quantitative results, versioning, and named accountability. Structurally, that resembles the front matter of a safety argument closely enough that a non-specialist reader fills in the rest by inference.
The inference is wrong in a specific way. A validation pack is evidence; a safety case is an argument that consumes evidence. The pack says what was measured. The safety case says why what was measured, combined with hazard analysis, architectural mitigations, and residual-risk reasoning, is sufficient to release the function into the world. Evidence does not become an argument by being thorough. We see this conflation regularly in perception programmes where the same engineering group owns both the model and the release paperwork, so there is no organisational seam to make the distinction visible.
The other driver is convenience. Writing a safety argument requires people with hazard-analysis accountability, and their calendar is not inside the release window. A pack that appears to already answer the question is an attractive substitute right up to the point where it is tested.
What the pack asserts and what it does not
This is the boundary worth printing on the cover sheet.
| Question | Perception validation pack | Regulatory / functional safety case |
|---|---|---|
| Core assertion | This model version behaved within measured bounds on defined operational conditions | Residual risk for the hazardous event is acceptable, and here is why |
| Unit of analysis | Model build, test suite, dataset slices | Vehicle-level function and its hazards |
| Hazard analysis | Not in scope | Foundational — the argument starts here |
| ASIL decomposition | Not in scope; the level may be quoted as context | Derived and justified within the case |
| Residual-risk argumentation | Absent by design | The central claim |
| Owner | Perception engineering lead, named | Functional safety manager / accountable engineer |
| Reviewer it is written for | Internal QA and customer engineering | Safety assessor, homologation authority |
| Failure if misused | Cited as regulatory evidence; stalls under hazard-traceability questions | Cited as engineering evidence; too abstract to gate a model release |
| Refresh trigger | New model build, threshold change, distribution shift | Change to the function, architecture, or hazard set |
Two things follow from that table. First, the pack’s silence on hazard analysis is not an omission to be patched — a perception team is not the accountable party for hazard classification, and a pack that improvises one produces an unowned safety claim, which is worse than no claim. Second, the pack’s specificity about model builds is precisely what a safety case cannot carry: safety arguments are written at the function level and would be invalidated by every retrain if they tried to absorb build-level detail.
The reviewer questions that tell you the pack is in the wrong room
The cheapest diagnostic is the question you get asked. Scope mismatch announces itself in the first review round, if anyone is listening for it.
- “Which hazardous event does this test address?” — the pack has operational conditions, not hazards. Wrong room.
- “Show me the requirement this test discharges, and where the requirement came from.” — traceability to safety requirements is a safety-case structure, not a validation-pack structure.
- “What is the residual risk after this mitigation?” — the pack has no mitigation ledger.
- “Who signed the decomposition rationale?” — the pack names a model owner, not a safety accountable.
- “What is your recall on the night-pedestrian slice at the operating threshold?” — correct room. This is what the pack is for.
- “Which model build is this, and what is the rollback path?” — correct room.
In our experience the expensive version of this failure is not being asked the wrong question once. It is a sequence of clarification rounds in which each side answers in good faith at a different level of abstraction, and the release date moves twice before anyone names the mismatch. Routing the pack to the reviewer whose questions it was built to answer — internal QA and customer engineering — is the intervention, and our approach to perception validation evidence for automotive programmes treats that routing decision as part of the pack’s design, not a distribution afterthought.
Feeding the safety argument without becoming it
The correct relationship is directional and stated. A safety case cites the validation pack the way a paper cites a dataset: by version, for a bounded claim, with the interpretation done by the citing document.
Practically, that means the pack carries three things it might otherwise leave implicit. A scope statement that names what is asserted and, explicitly, what is not — hazard analysis, ASIL decomposition, residual-risk argumentation, and regulatory acceptance are named as out of scope rather than merely absent. A stable identity, so a safety argument can cite “pack v4.2 against model build 1174” and have that reference remain resolvable after the next retrain; versioning the pack against the model artefact rather than the release date is what makes this hold. And condition tags on each result that the safety author can map onto their own operational design domain definition, which is the join that lets someone else do hazard reasoning on top of your measurements without re-running your tests.
The tooling side is unglamorous and matters: the pack’s numbers should come out of the same harness that emits production monitoring signals — a versioned test suite, tagged runs, and provenance recorded per evidence surface rather than assembled by hand into a document. We build these packs on top of the monitoring and evidence harness that already produces the drift signals, so the artefact a reviewer reads and the pipeline a fleet emits are not two separate record systems. The parent discussion of how the whole pack is assembled and sequenced sits in the automotive perception validation evidence package reference, which covers the section-by-section contents this article deliberately does not restate.
When it has already happened
Recovery is mostly bookkeeping done in public. Withdraw the citation rather than defend it — a pack cited as regulatory evidence and then quietly reinterpreted damages the credibility of every other number in it. Restate the pack’s assertion in its own terms. Identify who actually owns the safety argument and hand them the pack as an input with its version and condition tags intact. Expect the argument itself to take longer than the remaining release window, because it was never a document that could be compressed; that is the cost being paid for the earlier convenience.
What is worth noticing is that the artefact usually survives this intact. A pack that was scoped honestly gets reused — by the safety author, by the customer’s engineering reviewer, and by the next release. A pack that was informally promoted has to be rebuilt, not because its measurements were wrong, but because nobody can now tell which claims were the engineering ones.
The boundary is not a weakness in the pack. It is the property that makes the pack citable by someone else.
Frequently Asked Questions
What does it mean in practice that a perception validation pack is not a regulatory safety case?
Without hazard analysis, ASIL decomposition rationale, or residual-risk argument, no test coverage—however complete—will satisfy a safety assessor. It is an input to a safety argument written and owned by the functional safety accountable, not a substitute for one., it asserts that a named model version behaved within measured bounds on defined operational conditions, with named owners and a documented rollback path. It leaves unasserted anything requiring hazard reasoning: which hazardous events the behaviour affects, whether residual risk is acceptable, and whether a regulator would accept the function.
How does a safety case differ from a validation pack in structure, ownership, and the questions it answers? A safety case is an argument at the vehicle-function level, owned by a safety accountable, that consumes evidence to justify acceptable residual risk. A validation pack is evidence at the model-build level, owned by the perception engineering lead, that reports measured behaviour. The comparison table above sets the two side by side.
How does the validation pack feed into a safety argument without being mistaken for one? By being citable: a stable version identity tied to the model artefact, an explicit scope statement naming what is out of scope, and condition tags a safety author can map onto their operational design domain. The safety case does the interpretation; the pack supplies bounded, resolvable measurements.
Which reviewer questions signal that the pack has been routed to the wrong audience? Questions about hazardous events, safety-requirement traceability, residual risk after mitigation, or who signed a decomposition rationale. Questions about slice-level recall at the operating threshold, drift posture, model build identity, and rollback path mean the pack reached the reviewer it was written for.
How should the pack be labelled and scoped so downstream readers do not over-read it? State the assertion and the non-assertions on the cover sheet — hazard analysis, ASIL decomposition, residual-risk argumentation, and regulatory acceptance named explicitly as out of scope. Naming the intended reviewer alongside them is what stops a later reader from inferring a broader claim.
What are the consequences when a team cites the pack as regulatory evidence, and how is that recovered? The release stalls at the first hazard-traceability question and the real argument gets written from scratch inside a window that cannot absorb it. Recovery means withdrawing the citation, restating the pack’s actual assertion, and handing it as a versioned input to whoever owns the safety argument.t.
Why boundary confusion matters
Conflating a Perception Validation Pack with regulatory documentation creates scope creep that derails both timelines and team focus. That answer is workload-specific, and it is worth writing down before you build.
