Ask an automotive perception team what is in their validation package and you will get an accurate list: test reports, benchmark tables, dataset manifests, a regression summary. Ask which reviewer question each item answers, and the room usually goes quiet. That gap is what turns a complete package into a slow one.
A perception validation evidence package contains a fixed set of six evidence surfaces, not a variable inventory of everything the team measured. The surfaces are: scope and operating envelope; dataset and slice coverage; production behaviour under that envelope; drift and monitoring posture; ownership and escalation; and the rollback path. Every artefact in the package belongs to exactly one of them, carries a named owner, and links back to the test run that produced its numbers. Anything that cannot be assigned to a surface is not evidence — it is work-in-progress documentation that happened to be nearby at release cut.
The distinction matters at one specific moment. A reviewer points at a figure and asks where it came from. A surface-structured package answers with a link. An inventory-structured package answers with a meeting.
What is in the deliverable, section by section?
The table below is the contents list we work from. Read the middle column as the reason the section exists; if a section cannot answer its question, it is not finished, regardless of how many pages it runs to.
| Evidence surface | Reviewer question it answers | Minimum contents |
|---|---|---|
| Scope and operating envelope | “What conditions does this claim cover?” | ODD boundary statement, sensor configuration, model build ID, explicit out-of-scope conditions |
| Dataset and slice coverage | “Did you test the cases that matter, not just the easy ones?” | Slice definitions, per-slice sample counts, per-slice results, named gaps |
| Production behaviour | “How does it behave on the road, inside that envelope?” | Per-slice metrics at the deployed operating threshold, latency under production load, failure examples |
| Drift and monitoring posture | “How will you know when this stops being true?” | Monitored signals, thresholds, alert routing, baseline distribution reference |
| Ownership and escalation | “Who answers when it misbehaves at 02:00?” | Named model owner, on-call route, escalation ladder with response expectations |
| Rollback path | “What happens if you are wrong?” | Prior known-good build, rollback trigger conditions, tested rollback procedure and its duration |
Six surfaces, six questions. The package is complete when each row has an owner’s name against it and each number in it resolves to a test run identifier. That is a bounded assembly job — which is the practical point of declaring the contents set in advance rather than growing the package alongside the test backlog. Our colleagues cover the structural argument for ordering evidence this way in the perception validation evidence package for automotive OEM review; what follows here is the contents-level detail of each section.
How specific does the ODD boundary have to be?
Specific enough that a reviewer can decide, without asking, whether a scenario they care about falls inside or outside it. “Urban and highway driving, daylight and night” is not a boundary; it is a category. A usable boundary names the axes and their limits: speed range, illumination range with a measured lower bound, precipitation classes, sensor set and mounting geometry, and the model build the boundary applies to.
The out-of-scope statement is the half teams skip, and it is the half that buys credibility. Declaring that heavy snow with lens occlusion is outside the validated envelope, and that the system degrades to a defined safe behaviour there, reads as engineering judgment. Leaving it unstated reads as an untested condition the reviewer now has to hunt for.
Evidencing coverage without dumping the backlog
Dataset and slice coverage is where inventory-structured packages balloon. The full test log for a perception release can run to thousands of scenario executions, and attaching all of it does not answer the coverage question — it relocates it.
The section needs three things and nothing more: the slice definitions the team committed to before testing, the result per slice at the deployed threshold, and an honest list of slices with thin sample counts. In our experience, the slices that trigger clarification rounds are almost always the ones with real coverage but no declared definition — pedestrians at dusk tested thoroughly, never named as a slice, and therefore invisible to the reviewer. Full run logs belong in the linked test system, referenced by run ID, not pasted into the deliverable.
Named-technology detail helps here more than prose does. When the harness records slice tags in the same schema the production monitors emit — a pattern we build into the monitoring harness rather than bolting on afterwards — the coverage table and the drift table use one vocabulary, and the reviewer can read across them without a translation step. Sibling coverage of that trace mechanism sits in linking perception test results to production behaviour.
Production behaviour and drift posture, with provenance attached
These two sections are frequently merged, and they answer different questions. Production behaviour is a statement about now: measured metrics at the threshold that will actually ship, latency and throughput on the target compute, and concrete failure examples with their inputs. Drift posture is a statement about later: which signals are monitored, at what thresholds, against which baseline, and where the alert goes.
Provenance is the load-bearing detail in both. Each figure carries the test run identifier, the model artefact hash, the dataset version, and the harness commit. That is four fields per number, and it is the difference between a reviewer verifying a claim in thirty seconds and a reviewer scheduling a walkthrough. Teams running validation through a versioned harness with ONNX or TensorRT export artefacts pinned per build get this nearly free; teams assembling from spreadsheets pay for it in clarification rounds. This is one of the places where the reviewer’s trust in the whole package is decided by a formatting decision.
Ownership, escalation, rollback — and where they sit
These belong near the front, not in an appendix. A reviewer reading a perception package is assessing operational risk, and the ownership and rollback surfaces are the ones that bound it. The contents are short: a named model owner (a person, not a team alias), the on-call route, an escalation ladder with response expectations, the prior known-good build, the conditions that trigger a rollback, and evidence that the rollback procedure has been executed at least once rather than merely documented.
That last item is the most commonly missing piece of a perception validation evidence package. An untested rollback is a plan, and reviewers who have seen a bad build ship know the difference. Which reviewer type weights these surfaces most heavily varies — see who reviews a perception validation pack for how internal QA, customer engineering, and regulatory-facing reviewers read the same contents differently.
Which contents go missing, and how the gap shows up
Gaps in a perception package do not announce themselves as gaps. They arrive as clarification requests, and the request usually names a symptom rather than the missing surface.
- “Can you confirm the conditions this was measured under?” → the operating envelope has no explicit out-of-scope statement.
- “Which model build produced table 4?” → provenance fields are absent from the production-behaviour section.
- “How would you detect this regressing in the field?” → the drift posture describes dashboards, not thresholds and alert routing.
- “Who do we contact if this misbehaves?” → ownership is a team name, not a person with an escalation route.
- “Has the rollback been exercised?” → the rollback path is documented but untested.
Each of these costs a round trip, and round trips land squarely in the window between model freeze and submission. That is the measurable case for pre-declaring the contents set: assembly effort per release becomes plannable engineer-days, and the clarification requests attributable to a missing or unlinked evidence surface trend toward zero. We track both alongside first-pass clearance rate on the computer vision work where these packages ship.
The contents list above is stable across perception programmes we have worked on. What is not stable is how deep each surface has to go, and that depth is set by who opens the package and what integrity level the function carries. Which raises the question worth taking into your next release review: if a reviewer picked one number in your package at random, how many clicks would it take to reach the run that produced it?
Frequently Asked Questions
What does “what a perception validation evidence package contains” mean in practice — which sections are actually in the deliverable? Six evidence surfaces: scope and operating envelope, dataset and slice coverage, production behaviour under that envelope, drift and monitoring posture, ownership and escalation, and the rollback path. Each surface has a named owner and each figure inside it links back to the test run that produced it. Artefacts that do not map to one of the six are not part of the deliverable.
Which reviewer approval question does each evidence surface answer? Scope answers what conditions the claim covers; coverage answers whether the cases that matter were tested; production behaviour answers how the model behaves inside the envelope; drift posture answers how the team will know when that stops being true; ownership answers who responds to a failure; rollback answers what happens if the release was wrong.
What belongs in the scope and operating-envelope section, and how specific does the ODD boundary need to be? Speed range, illumination limits with a measured lower bound, precipitation classes, sensor set and mounting geometry, and the model build the boundary applies to. Specific enough that a reviewer can classify a scenario as in-scope or out-of-scope unaided. The explicit out-of-scope statement, with the degraded safe behaviour named, is required rather than optional.
How is dataset and slice coverage evidenced without dumping the full test backlog into the pack? Include the pre-committed slice definitions, the result per slice at the deployed threshold, and an honest list of thinly covered slices. Full run logs stay in the test system and are referenced by run ID. Coverage that exists but was never declared as a slice is invisible to the reviewer and reliably triggers a clarification round.
What does the production-behaviour and drift-posture section contain, and how is it linked back to the test run that produced it? Production behaviour carries metrics at the shipping threshold, latency on target compute, and concrete failure examples. Drift posture carries monitored signals, thresholds, baseline reference, and alert routing. Both link back through four provenance fields per figure: test run ID, model artefact hash, dataset version, and harness commit.
What ownership, escalation, and rollback information does a reviewer expect to find, and where does it sit in the pack? A named individual owner, the on-call route, an escalation ladder with response expectations, the prior known-good build, rollback trigger conditions, and evidence the rollback procedure has actually been executed. These belong near the front of the package, because they bound the operational risk the reviewer is assessing.
Which contents are commonly missing, and how does that gap show up as a clarification round? The most common omissions are an explicit out-of-scope statement, per-figure provenance fields, drift thresholds as opposed to dashboards, a person-level owner, and a tested rollback. Each surfaces as a reviewer question about conditions, build identity, field detection, contact routing, or rollback confidence — one round trip each, spent inside the release window.
Contents define the package’s utility
A Perception Validation Evidence Package succeeds or fails based on completeness: missing even one critical artifact undermines stakeholder confidence. That answer is workload-specific, and it is worth writing down before you build.
