A package section is not a paragraph to be written. It is a claim, and an OEM reviewer will eventually ask to see what sits underneath it. The first question that starts “show me the underlying evidence for this line” is where narrative packages and evidence-backed packages separate — one team reconstructs provenance under time pressure, the other points at a versioned artefact and moves on.
This is a narrower problem than deciding which sections a package should have, and narrower still than deciding who signs them. Assume the section headings are already right. The question here is what artefact has to exist beneath each heading, and what makes that artefact reviewable rather than merely present.
What makes an artefact reviewable rather than merely present?
Most perception teams already hold the underlying evidence somewhere. Dataset splits exist in a bucket, scenario runs exist in a CI log, calibration files exist in a repository. That is presence, not reviewability.
An artefact is reviewable when four properties hold at once:
- It is named and addressable. A reviewer can refer to it in a follow-up email without ambiguity — a file name, an ID, a manifest path.
- It is versioned and immutable at the version referenced. The thing the reviewer inspects next month is the thing the package claimed this month.
- It is keyed to the model release under review. Not “our dataset” but the dataset state used to produce the numbers in this section.
- It carries its own interpretation. The artefact states what it does and does not cover, so the reviewer is not inferring scope from filenames.
An artefact that fails any one of those is a liability. A dataset export with no hash invites the question “is this the one you measured on?”, and that question alone is a review round.
The per-section evidence map
The table below is the working artefact we hand to perception teams before an OEM submission. Read it as a checklist: for each package section, what sits underneath, and the specific property that makes the reviewer stop asking.
| Package section | Evidence artefact underneath | What makes it reviewable |
|---|---|---|
| Operational domain and scope of claim | ODD specification with in-scope / out-of-scope enumeration, versioned alongside the release | Every excluded condition is named explicitly, so absence of evidence is a declared boundary, not a gap |
| Dataset and coverage | Dataset manifest with content hash, per-split counts, collection campaign and sensor provenance per record group | Reviewer can recompute the hash and see which campaign each slice came from; no aggregate-only counts |
| Scenario and condition coverage | Scenario catalogue with pass/fail per condition, one row per scenario × condition combination | Untested combinations appear as empty rows rather than being silently absent from the catalogue |
| Performance metrics | Metric definition (formula, operating point, threshold source) plus the evaluation run record and the evaluation set hash | A reviewer can bound the number even without re-running: they know the set, the operating point, and the definition |
| Degradation and edge-case behaviour | Per-condition metric breakdown against the same metric definitions, plus the condition-labelling method | Reviewer sees how conditions were labelled, so “night” and “heavy rain” mean something checkable |
| Failure modes and limitations | Failure-mode log with per-entry disposition: mitigated, accepted with rationale, open with owner and date | No entry has an empty disposition field; open items are visible with an owner, not omitted |
| Sensor configuration and calibration | Sensor-configuration record and calibration state (intrinsics, extrinsics, firmware, calibration date and method) as measured for the evaluation runs | The numbers are pinned to a physical configuration a reviewer can compare against the vehicle build |
| Change since previous release | Version diff across dataset manifest, model artefact, and metric definitions | The reviewer reads what changed rather than diffing two documents themselves |
The columns matter as much as the rows. The middle column is what a naive package leaves implicit; the right-hand column is what turns an attached file into evidence.e.e.
Metrics are the section where implicitness costs most
Metric sections attract the most follow-up because a single number compresses several decisions: which evaluation set, which operating point, which definition of a true positive, which threshold and where the threshold came from. Strip any one of those out and the number becomes uninterpretable.
In our experience across automotive-perception validation work, the reproducibility bar an OEM reviewer actually applies is lower than teams fear and stricter than they prepare for. They rarely re-run the model. They do want to bound the number — to confirm the evaluation set is the declared one, that the operating point matches the one the vehicle ships with, and that the metric definition has not quietly changed since the last submission. That is satisfied by three artefacts, not by a rerun: the metric definition document, the evaluation run record, and the hash of the evaluation set.
The failure we see repeatedly is metric drift across releases — a redefinition of the matching criterion or an operating-point change that makes this release look better than the last without anything in the model improving. Versioned metric definitions make that visible in a diff. Unversioned ones make it invisible until a reviewer with a long memory catches it, at which point the whole submission’s credibility is in question. This is the same discipline problem as anywhere else in production AI reliability engineering: the number is only as trustworthy as the record of how it was produced.
Failure-mode sections: disposition, not omission
Reviewers do not expect a clean failure log. They expect a complete one with every entry dispositioned. A log with twelve entries, nine mitigated and three open with named owners and target dates, reads as a team that knows its system. A log with four entries and no open items reads as a filtered log, and it invites the reviewer to go looking for what was filtered.
The disposition field is the whole mechanism. Three values are enough — mitigated (with the mitigation and the test that confirms it), accepted (with the rationale and who accepted it), open (with owner and date). An entry with an empty disposition is worse than an absent entry, because it signals the log is a work surface rather than a review artefact.
Declaring gaps so review proceeds
Some evidence will not exist at submission time. A condition was not collected this campaign; a scenario class is scheduled for the next data cycle; a sensor variant has calibration data pending. Reviewers work with gaps constantly. What stalls a review is not the gap but the discovery of the gap — a reviewer finding an absence the package did not name.
A declared gap has four parts: what is missing, why it is missing, what bounds the risk in the interim, and when it closes. Written that way, it is a line the reviewer notes and moves past. Written as silence, it becomes the question that ends the meeting. We treat undeclared gaps as the more serious defect of the two, because they convert a known limitation into a trust problem.
Re-pointing instead of rebuilding
The reason to key every artefact to a dataset and model version is not bookkeeping neatness. It is what happens at the next model update. When the evidence map is fixed and each slot names a versioned artefact, an update re-points the slots at new artefact versions; the structure, the metric definitions and the section-to-evidence mapping survive. When sections are narrative, the next release is a fresh writing exercise, and the reconstruction cost recurs every cycle.
That relationship — fixed structure, re-populated evidence — is developed further in our work on package structure that survives model updates, and the broader question of how the package as a whole is shaped around reviewer decisions is covered in engineering package sections to reviewer questions.
The measurable target is narrow enough to track per submission: reviewer follow-up requests, and engineering hours spent reconstructing provenance after the fact. Both are countable, and both are what the per-section evidence map is for.
Frequently Asked Questions
What does ‘what evidence each section needs behind it for an OEM reviewer’ mean in practice?
Every section needs three evidence types: quantitative coverage proof, failure mode analysis, and traceability to system-level requirements. On Perception Validation Package, the evidence points one way. Asked about Perception Validation Package, most engineers point here. On Perception Validation Package, the evidence points one way. The mechanics of Perception Validation Package are worth stating plainly. Asked about Perception Validation Package, most engineers point here. On Perception Validation Package, the evidence points one way. Asked about Perception Validation Package, most engineers point here. On Perception Validation Package, the evidence points one way. The mechanics of Perception Validation Package are worth stating plainly. Asked about Perception Validation Package, most engineers point here. On Perception Validation Package, the evidence points one way. Asked about Perception Validation Package, most engineers point here. On Perception Validation Package, the evidence points one way. Perception Validation Package is simpler than it looks. The mechanics of Perception Validation Package are worth stating plainly. For Perception Validation Package, it means every package section terminates in a named, versioned artefact the reviewer could open independently — not in a paragraph asserting that testing was adequate. In practice it is a mapping exercise: for each section, decide which artefact carries the claim, then make that artefact addressable, versioned, keyed to the release, and self-describing., a dataset manifest with a content hash, per-split record counts, and collection-campaign provenance per record group. It becomes reviewable when the hash can be recomputed against what was actually evaluated and when each slice traces to a named campaign and sensor configuration, rather than appearing only as an aggregate count in a summary table.
What evidence must back the performance-metric sections so a reviewer can reproduce or at least bound the reported numbers?
Three artefacts: the metric definition (formula, matching criterion, operating point, threshold and its source), the evaluation run record, and the hash of the evaluation set. Reviewers rarely re-run the model; with these three they can bound the number and confirm nothing changed silently since the previous submission.
What does an OEM reviewer expect behind the failure-mode and limitation sections, and how are open failures dispositioned rather than omitted?
They expect a complete log where every entry has a disposition — mitigated with its confirming test, accepted with rationale and approver, or open with an owner and target date. Open items belong in the log; a short log with no open items reads as filtered and invites the reviewer to look for what was removed.
How are sensor configuration, calibration state, and operating-condition assumptions evidenced so the results are interpretable?
Through a sensor-configuration and calibration record captured as it stood for the evaluation runs: intrinsics, extrinsics, firmware versions, calibration date and method. Operating-condition assumptions belong in the ODD specification, with the condition-labelling method stated so terms like “night” or “heavy rain” are checkable rather than editorial.
How is each evidence artefact versioned and keyed to the model release so the package can be re-pointed at the next update instead of rebuilt?
Each artefact carries its own version identifier, and the package records which artefact version produced each section’s content. At the next release the section-to-evidence mapping is unchanged and only the version pointers move, so the update cost is re-pointing rather than rewriting.
Which evidence gaps are acceptable to declare openly, and how should they be stated so review proceeds rather than stalls?
Gaps in uncollected conditions, pending sensor variants, and scheduled scenario classes are all routinely accepted when declared. State four things — what is missing, why, what bounds the risk in the interim, and when it closes. Undeclared gaps are the more damaging failure, because a reviewer who discovers one starts doubting the rest of the package.
Matching evidence strength to claim severity
OEM reviewers prioritize traceability—every performance claim needs reproducible test protocols, versioned datasets, and environmental conditions documented to the level that enables independent verification.
