A release reviewer is not asking to be convinced. They are asking for an evidence pack with a fixed shape, and the fastest way to lose a review cycle is to arrive with a benchmark score whose provenance nobody can reconstruct. Most perception teams assemble what they already have — an mAP figure, a training summary, a slide of qualitative examples — and treat the gate as a presentation to be passed. Reviewers read it as an incomplete evidence request and send it back.
The difference is not rigour of the underlying work. It is whether the work is traceable. A reviewer can accept a documented 3.1% miss rate on a night-time cyclist class if the test set that produced it is described well enough to argue about. The same reviewer cannot accept a 94% aggregate accuracy figure whose coverage and sampling cannot be reconstructed — not because they doubt it, but because there is nothing in it to review.
What does a release reviewer actually ask for?
Six items, in roughly this order. Each one exists to close a specific question, and each one is refused for a specific reason.
| Evidence item | What it must prove | Common reason it gets sent back |
|---|---|---|
| Test-set provenance | Where the data came from, under what collection conditions, labelled by which process with what agreement rate | “Internal dataset v7” with no collection or labelling description |
| Distribution mapping | How the test set maps onto the deployed fleet’s route, weather, lighting, road-class and sensor-rig mix | Test set is a public split plus hard clips; mapping is asserted, not shown |
| Per-scenario-class failure rates | Model behaviour broken out by the classes the release decision turns on, with counts alongside rates | One aggregate number; no denominators, so a 0% failure on 11 samples looks like a pass |
| Known-and-accepted failures | Which failures were found, which are being mitigated, which are accepted and on what basis | Failures presented as “fixed”, leaving no residual-risk statement |
| Regression delta vs previous release | What moved, per class, since the last approved model | Only the aggregate delta, which hides a class that got worse |
| Sign-off record | Who reviewed what, on which artefact version, and what they signed | Names on a deck with no link to the model hash or test-set version |
That table is the whole checklist. The rest of this article is what each row costs to produce properly.
Test-set provenance: enough to argue with
Provenance is not a dataset name. A reviewer needs enough to judge whether the number could plausibly transfer to the road: collection geography and time window, the weather and time-of-day mix as actually collected (not as intended), the sensor rig and firmware the clips came from, the labelling instructions, who labelled, and what inter-annotator agreement looked like on the confusable classes.
The last one gets skipped most often and matters most on edge classes. If your labellers disagree 8% of the time about whether a partially occluded pedestrian is a pedestrian, a measured 5% miss rate on that class is inside the noise of its own ground truth, and a reviewer who spots that will — correctly — stop reading. We see this pattern regularly in audits: the model is fine, the label quality on the rare class is the actual blocker. Designing the sampling in the first place is a separate discipline, and we cover it in how to build a test set that reflects production driving conditions.
Per-scenario-class failure rates, with denominators
Report failure rates per scenario class, always with sample counts, and never as a single row of aggregate accuracy. A reviewer judging coverage needs to see that the wet-road-glare class has 1,400 frames behind its 4.2% miss rate and the tunnel-exit class has 60 behind its 0%. The second cell is not a pass; it is an absence of evidence, and saying so in your own pack is faster than having it discovered.
Three formatting conventions make a per-class table reviewable rather than merely present:
- Counts next to every rate. A rate without a denominator cannot be assessed for statistical weight.
- Class definitions attached. “Low sun” must have a stated angle and illumination band, or two releases cannot be compared.
- Explicit gaps. Classes you know exist in the production distribution but did not test appear in the table as rows marked not covered, not as omissions.
Detection metrics themselves — mAP, IoU thresholds, per-class precision/recall from a standard evaluation harness such as the COCO-style evaluators most teams run under PyTorch — are uncontroversial. The reviewable content is the stratification around them.
Known and accepted, without implying eliminated
This is where wording carries legal and engineering weight simultaneously. A perception model that has been audited still fails on rare inputs; the pack’s job is to convert those from unknown into documented. In our experience the cleanest structure is three explicit columns per finding: the observed behaviour, the mitigation (compensating sensor, downstream logic, operational-design-domain restriction, monitoring rule), and the residual position — accepted, accepted-with-monitoring, or blocking.
Never write “resolved” where you mean “reduced”. A reviewer who later meets a field failure that your pack described as resolved will discount the entire document, including the parts that were accurate. Naming the residual honestly is what makes the rest credible, and it is the same discipline that keeps a robustness audit from being read as a clearance certificate — a distinction we draw out in engineering validation versus safety certification.
Traceability: model hash, test-set version, signature
The sign-off record is the cheapest item to produce and the most frequently missing. Each conclusion in the pack should carry the artefact it was computed from — model weights hash, test-set version tag, evaluation harness commit — and the name of whoever accepted it. Model team signs that the artefact under review is the artefact built; validation signs that the measurements were produced on the declared test set with the declared harness; the release reviewer signs the decision, not the measurements.
When teams run this through an MLflow-style tracking store or an equivalent artefact registry, the sign-off record stops being a manual step and becomes a query. That is the practical mechanism behind the parent hub’s audit output feeding a reviewer-facing pack, and it is what Production AI Monitoring Harness is scoped to produce for perception workloads once the model is live and drifting.
Why the pack shortens review cycles
The measurable outcome is pass-through time: how many cycles a release needs before sign-off, and how much of each cycle is spent reconstructing evidence rather than discussing results. Reviews that stall almost never stall on a bad number. They stall because the reviewer asks where the test data came from and the answer takes a week to assemble, at which point the model has moved and the numbers are stale.
There is a second, slower payoff. A complete pack moves post-release surprises into the known-and-accepted column: the long-tail failure rate for a given scenario class becomes a documented figure agreed before deployment rather than something discovered in a field report. That is the difference between a monitoring alert and a rollback. Teams that can hand over test-set provenance and per-class failure tables on request avoid the rollback cost of a benchmark-blind regression surfacing after the fleet has it.
Which classes are release-blocking, and what regression is tolerable, is a decision made before the audit runs rather than during the review — we work through that separately in the release decision rubric. The evidence pack described here is what the rubric is applied to.
Where this stops
Everything above is engineering validation evidence about how a perception model behaves against a characterised production driving distribution. It is not a safety case. Functional-safety argumentation, hazard analysis, ODD definition and homologation remain the OEM’s or Tier 1’s to own, and the pack should say so on its first page. Reviewers who work in this space read an over-claiming pack as a governance risk rather than a thorough one.
The open question we keep returning to is how much of a distribution you have to characterise before a per-class failure rate is worth reporting at all. Wider perception-system context sits in our computer vision practice; the honest answer for now is that it depends on how much of the deployed fleet’s variation you can observe, and most teams can observe less of it than their test set implies.
Frequently Asked Questions
What items belong in a perception validation evidence pack, and what does each one have to prove? Six: test-set provenance (where the data came from and how it was labelled), distribution mapping (how the test set relates to the deployed fleet’s conditions), per-scenario-class failure rates with sample counts, a known-and-accepted failure register, a per-class regression delta against the previously approved model, and a sign-off record tying each conclusion to a model hash and test-set version. Each item closes one reviewer question; together they let a reviewer judge coverage rather than take an aggregate score on trust.
How much test-set provenance does a reviewer need before a failure rate is credible? Enough to argue with: collection geography and time window, the weather and lighting mix as actually collected, the sensor rig and firmware behind the clips, the labelling instructions, and inter-annotator agreement on the confusable classes. Agreement data matters most on rare classes — if labellers disagree at a rate comparable to the measured miss rate, the number sits inside the noise of its own ground truth.
How should per-scenario-class failure rates be reported so a reviewer can judge coverage? Always with denominators, always with the class definition attached, and with untested classes present as explicit not covered rows rather than silent omissions. A 0% failure rate over 60 frames is an absence of evidence, not a pass, and stating that yourself is faster than having a reviewer find it.
How do you document known and accepted edge-class failures without implying they are eliminated? Give each finding three fields: observed behaviour, mitigation (compensating sensor, downstream logic, ODD restriction or monitoring rule), and residual position — accepted, accepted-with-monitoring, or blocking. Write “reduced” where you mean reduced; a finding described as resolved that later reappears in the field discredits the whole document.
Where does engineering validation evidence stop and the OEM’s safety case begin? The pack characterises measured model behaviour against a described production driving distribution and names what it did not cover. Hazard analysis, functional-safety argumentation, ODD definition and homologation belong to the OEM or Tier 1, and the pack should state that boundary explicitly on its first page.
What Safety-Critical Release Reviewers Need to See
Validation evidence begins with traceability: every requirement to test case, every risk to mitigation, every change to regression coverage. Validation Evidence Safety Critical rewards teams that measure first and argue later — start with the smallest instrumented slice and let the numbers settle the design.
