A reviewer asked to sign a perception release is not asking whether your tests ran. They are asking what the numbers mean for the release they are accountable for — and an exported test report cannot answer that question, no matter how complete the test coverage behind it is.
That distinction is the whole difference between a test report and a validation package. The report is an artefact of the perception team’s own process: suites, pass/fail counts, a metrics table, a list of open defects. The package is an artefact built for someone outside that process, who has to form a judgement without having watched the work happen.
What does an automotive perception validation package contain beyond an internal test report?
Six things that a test export structurally cannot carry, because the internal tooling was never asked to record them:
- A scope and operational design domain statement — what conditions the claim covers, and by explicit exclusion, what it does not.
- Dataset and scenario provenance — where the evaluation data came from, how it was versioned, and why it is representative of the declared domain.
- Metric definitions with acceptance thresholds and their rationale — not just the measured value, but why that threshold is the right bar.
- Degradation and edge-case behaviour — what the system does as conditions leave the comfortable centre of the domain.
- Known limitations with mitigations — declared, not discovered later by the reviewer.
- Traceability from each claim to the run that supports it — a specific, retrievable artefact behind every statement.
The internal report contributes raw material to items 2, 3, and 4. It cannot author items 1, 5, or 6, because those are statements about intent and boundaries, and intent does not live in a CI database.
The divergence point is the moment a reviewer asks why a threshold was chosen. An internal test report can only show the number. A package section carries the justification alongside it — and everything downstream, from how long sign-off takes to how much of the document survives the next retrain, follows from that one structural choice.
Section anatomy, and the reviewer question each section closes
| Section | Reviewer question it answers | Primary source |
|---|---|---|
| Scope / ODD statement | “What exactly am I signing off, and what am I not?” | Authored for the package |
| Dataset & scenario provenance | “Is the evidence representative of the domain claimed?” | Data platform + manifests |
| Metric definitions & thresholds | “What was measured, and why is that bar acceptable?” | Test report + authored rationale |
| Degradation & edge-case behaviour | “What happens outside nominal conditions?” | Robustness / stress runs |
| Known limitations & mitigations | “What residual risk am I accepting?” | Authored, cross-checked against defects |
| Claim-to-run traceability | “Where is the evidence for this specific sentence?” | Run IDs, artefact hashes |
Read the right-hand column and the carry-over rule becomes obvious. Measurement output carries over from the internal report largely unchanged — throughput figures, per-class detection metrics, per-scenario pass/fail rows. Everything that constitutes an argument has to be re-authored, because the internal report was written to answer “did the suite go green”, and the package has to answer “is this releasable”.
Why threshold rationale is the load-bearing part
In our experience reviewing perception evidence with client teams, the metric section is where reviews stall most often, and rarely because the number is bad. It stalls because the number arrives without a reason to trust the bar it is compared against. A mean average precision of 0.87 at IoU 0.5 is a fact; whether 0.87 is sufficient depends on the downstream consumer of the detection — a planning stack that brakes on any track above a confidence floor tolerates different error structure than one that fuses detections across frames before acting.
So a signable metric section states four things together: the metric definition (including the operating point and the IoU or confidence convention used), the measured value, the acceptance threshold, and the derivation of that threshold from a hazard, a customer requirement, or a documented predecessor baseline. Where the threshold came from an engineering judgement rather than a requirement, say so — an explicitly-labelled judgement is reviewable, an unlabelled one reads as an accident.
Traceability is the mechanical half of the same job. Each metric row should resolve to a run identifier, a dataset version, and a model artefact hash — the same three coordinates you would need to reproduce the number. Where the evaluation harness is built on ONNX Runtime or TensorRT, the runtime version and precision mode belong in that coordinate set too, because a quantised engine and its float reference are not interchangeable evidence for the same claim. If a reviewer cannot get from a sentence in the package to the run that produced it in one step, the sentence is decoration.
Writing the scope statement so it is not over-read
The ODD and scope section is short, and it is the section teams most often under-write. Its function is defensive: it prevents the package from being read as a broader claim than the evidence supports.
Two habits make it work. First, state exclusions explicitly rather than relying on the absence of a claim — “night-time performance below 5 lux was not evaluated in this release” is a stronger statement than silence, because silence gets interpreted as coverage. Second, keep the scope statement’s vocabulary aligned with the scenario catalogue, so a reviewer can match a declared condition to the rows of evidence that cover it without translating between two taxonomies.
Known limitations follow the same logic and cause the same anxiety. Teams worry that declaring limitations weakens the release case. It does the opposite: a limitation that arrives with a mitigation and a boundary is a controlled item, while a limitation the reviewer discovers themselves becomes a credibility question about the whole package. Pair each limitation with the condition under which it applies, its detected effect, and either a mitigation or an explicit residual-risk acceptance.
What this buys, and where it stops
Section-shaped packages typically clear OEM or internal QA review in a single round rather than the two-to-four rounds that backlog exports incur (observed pattern across our validation engagements; not a benchmarked rate). The saving is not in the writing — the writing is more work the first time. It is in the review rounds that never happen, and in the fact that the justification prose is written once. Because the section list is invariant across model versions, a retrain re-populates evidence rows rather than triggering a fresh documentation exercise; the structural half of that argument is developed in our reliability work on production AI systems, where the same evidence-slot logic applies well beyond automotive perception.
There is a boundary worth naming. A package built this way is component-level evidence with declared limits. It is not a system-level safety argument, and it does not become one by being thorough — the argument that this evidence is sufficient for a given hazard belongs to a different owner and a different document. We see teams lose weeks to that confusion in both directions.
The harder open question is coverage adequacy. A package can state precisely which scenarios were covered and which thresholds were met, and still leave the reviewer to judge whether the scenario catalogue itself was the right set. Nothing in the package structure settles that — which is why the scope statement, not the metrics table, is the section worth arguing about longest.
Frequently Asked Questions
What does an automotive perception validation package contain beyond an internal test report, in practice?
Beyond metrics and test logs, automotive perception packages must document edge-case catalogs, degradation modes, and operational design domain boundaries. The mechanics of Perception Validation Package Contents are worth stating plainly. Asked about Perception Validation Package Contents, most engineers point here. The mechanics of Perception Validation Package Contents are worth stating plainly. Stripped down, Perception Validation Package Contents is the following. Asked about Perception Validation Package Contents, most engineers point here. The mechanics of Perception Validation Package Contents are worth stating plainly. Asked about Perception Validation Package Contents, most engineers point here. The mechanics of Perception Validation Package Contents are worth stating plainly. Stripped down, Perception Validation Package Contents is the following. Asked about Perception Validation Package Contents, most engineers point here. The mechanics of Perception Validation Package Contents are worth stating plainly. Asked about Perception Validation Package Contents, most engineers point here. The mechanics of Perception Validation Package Contents are worth stating plainly. Perception Validation Package Contents answers cleanly when you separate two things. Stripped down, Perception Validation Package Contents is the following. When applied to Perception Validation Package Contents, it adds the material the test tooling never recorded: a scope and operational design domain statement, dataset and scenario provenance, threshold rationale alongside each metric, degradation and edge-case behaviour, declared limitations with mitigations, and claim-to-run traceability. The measurements themselves largely carry over; the reasoning around them has to be authored., six sections map to six reviewer questions: scope/ODD (“what am I signing?”), provenance (“is the evidence representative?”), metrics and thresholds (“what was measured and why is that bar acceptable?”), degradation (“what happens outside nominal?”), limitations (“what residual risk am I accepting?”), and traceability (“where is the evidence for this sentence?”).
What evidence has to sit behind a metric section for it to be signable? Four things together: the metric definition including operating point and IoU or confidence convention, the measured value, the acceptance threshold, and the derivation of that threshold from a hazard, customer requirement, or documented baseline. Each row should resolve to a run ID, dataset version, and model artefact hash — with runtime and precision mode recorded where quantisation is involved.
Which parts of an internal test report carry over unchanged, and which have to be re-authored? Measurement output carries over: per-class metrics, per-scenario pass/fail rows, throughput figures. Everything that constitutes an argument — scope, threshold justification, limitations, residual-risk statements — must be re-authored, because the internal report was written to answer whether the suite went green, not whether the release is signable.
How should known limitations be declared without undermining the release case? Pair each limitation with the condition under which it applies, its observed effect, and either a mitigation or an explicit residual-risk acceptance. A declared and bounded limitation is a controlled item; one the reviewer finds unaided becomes a credibility question about the entire package.
Documentation layers OEMs require before integration
Perception validation begins with corner case coverage: rare geometries, adverse weather, sensor degradation, and the long tail of real-world driving scenarios your test track never includes. The teams that do tend to ship the boring, correct version first.
