“Robust enough to ship” is not a score. It is a set of per-scenario-class exit criteria agreed before the audit runs, plus a named owner for each decision and a written record of every residual risk the release accepts. Teams that treat it as a score end up in a familiar room: aggregate mAP improved over the previous release, one scenario class got worse, and nobody agreed in advance what that combination means.
That combination is the whole problem. A single aggregate number cannot express it, because aggregates are dominated by the frequent, easy cases. So the release review turns into a negotiation — the model team argues the headline moved the right way, validation argues the regression is real, and whoever is least tired wins. A rubric replaces that argument with evidence checking.
This article is about the decision structure, not the test design. How you build the test set, enumerate edge classes, and measure per-class failure rates is the subject of our work on perception robustness auditing for automotive programmes. Here we assume the evidence exists and ask what to do with it.
One boundary first, because it changes how the rubric is written: this is an engineering release decision, not a safety case. The rubric decides whether a software artifact may go into a given release channel with documented residual risk. It does not certify functional safety, and it does not substitute for the OEM’s or Tier 1’s homologation argument.
What does “robust enough to ship” mean in practice?
Practically, this means four things are true simultaneously, and each one is checkable by someone other than the person who trained the model.
First, every scenario class the release cares about has a threshold, and that threshold was written down before results existed. Second, the regression tolerance against the previous release is explicit per class — not just for the aggregate. Third, each class is labelled either release-blocking or monitored-with-mitigation, and that label was assigned by the role that owns the consequence, not the role that owns the model. Fourth, every accepted shortfall exists as a numbered entry in a residual-risk register with an owner, a monitor, and a review date.
A rubric that cannot fail a release candidate whose aggregate score improved is not a rubric. The whole point of per-class exit criteria is that they can dissent from the headline.e.
The ship / hold / partial-ship rubric
Three outcomes, not two. The middle one is what teams without a rubric cannot reach, and it is usually the correct answer.
| Evidence pattern | Verdict | Required action |
|---|---|---|
| All classes meet threshold; no class regressed beyond tolerance | Ship | Register carries only pre-existing accepted risks; reviewer signs the pack |
| A release-blocking class fails threshold, or regresses beyond tolerance | Hold | Root-cause the class before re-audit; aggregate improvement is not a mitigating argument |
| A monitored class fails, all blocking classes pass | Partial ship | Ship with a scoped mitigation (operating-domain restriction, confidence gating, fallback behaviour) + new register entry + monitor in place before rollout |
| A class has insufficient test coverage to evaluate | Hold on that class | Coverage gap is a hold reason in its own right; an unmeasured class is not a passing class |
| Aggregate improved, several monitored classes drifted slightly, none beyond tolerance | Ship, flagged | Log the drift direction; tighten next-release tolerance rather than waiving it |
The fourth row causes the most friction and is the one worth defending hardest. Absence of evidence gets read as absence of failure far more often than it should. In our experience, coverage gaps discovered during review — rather than during test-set design — are the single most common cause of a late release-blocking finding.
Which classes should be release-blocking?
A class earns release-blocking status on consequence severity and mitigability, not on how hard it is to fix. Two questions decide it: if this class fails in the field, is there a runtime mitigation that degrades gracefully, and is the failure detectable from vehicle telemetry before a human notices it?
Classes with no graceful degradation and no detection path are blocking — vulnerable road user detection under low-sun glare typically sits here, because there is no fallback that preserves function. Classes with a credible mitigation are candidates for monitored status: a confusable-class error in a low-speed manoeuvring context can often be handled by an operating-domain restriction plus a telemetry monitor, which is a legitimate engineering answer rather than a waiver.
Setting thresholds before the audit, not after
Thresholds negotiated after a failure are not thresholds. The sequencing that holds up under pressure is dull and works: characterise the production driving distribution, enumerate the scenario classes, assign blocking/monitored labels, set per-class thresholds and regression tolerances, then run the audit.
Where do the numbers come from on a first release with no prior baseline? Usually from three sources in combination — the incumbent system’s measured field failure rate per class where one exists, the physical or functional limit of the sensing configuration, and a deliberately conservative first pass that the next release tightens. Naming which of the three a threshold came from matters, because it tells the reviewer how much argument the number can bear. A threshold derived from an incumbent’s measured rate is a benchmark-class figure; one set by engineering judgement on a first release is an observed-pattern estimate and should be labelled as such in the pack.
Regression tolerance deserves its own line per class rather than a global percentage. A two-point drop in a well-covered daylight vehicle class is noise. The same drop in a rare pedestrian-occlusion class may be the entire signal, because the class has few samples and each one carries weight. Tooling helps here: pinning the evaluation harness — the dataset version, the pre-processing, the ONNX or TensorRT engine build used for inference, the confidence and NMS settings — is what makes a two-point delta interpretable at all. We have seen more “regressions” traced to an engine rebuild or a changed input pipeline than to the model weights.
Who holds the pen
Three roles, three different evidence needs, and the failure mode is collapsing them into one.
- Model team — owns the claim that a class-level result is reproducible on the pinned harness, and that any fix addresses the mechanism rather than the test cases. Signs the technical result.
- Validation — owns the claim that the test set represents the production distribution for each class and that coverage is sufficient to evaluate the threshold. Signs the measurement.
- Release reviewer — owns the ship/hold/partial-ship verdict and the acceptance of each residual-risk entry. Signs the decision.
Validation signing the decision, or the model team signing the measurement, is where rubrics quietly stop working. The separation is not bureaucracy; it is what makes the signature mean something. The evidence shape each role needs is covered in more depth in our note on what a safety-critical release reviewer expects from a validation pack.
The residual-risk register
Every partial ship creates a register entry. A usable entry has six fields: scenario class, measured failure rate with its evidence class, the mitigation in place, the telemetry monitor that will detect the failure in the field, the accepting role, and a review date tied to a specific future release.
This is what converts a post-release surprise into a post-release confirmation. When a field failure lands in a class that has a register entry, the conversation is about whether the monitor fired and whether the rate exceeded what was accepted — a bounded, answerable question. When it lands in a class nobody named, the whole release argument is retroactively in doubt. A rollback trigger belongs in the register too: the field rate that, if exceeded, pulls the release without further debate.
The mechanics of keeping those monitors honest after rollout — separating incoming-distribution drift from model behaviour on the fixed reference set — sit with the monitoring layer described in Production AI Monitoring Harness, and the rubric is the decision surface that reads its output. The broader engineering context for the perception stack itself lives on our computer vision practice page.
Frequently Asked Questions
What does “robust enough to ship” mean in practice for a perception model?
Three reliability thresholds determine whether your perception model survives real-world deployment or crumbles under edge cases. It means each scenario class the release depends on has a pre-agreed threshold and regression tolerance, each class is labelled release-blocking or monitored-with-mitigation, and every shortfall the release accepts is a numbered residual-risk entry with an owner and a monitor. An aggregate score crossing a bar is not the same statement and cannot substitute for it.
Which scenario classes should be release-blocking, and which can ship with a monitored mitigation?
Blocking status follows from consequence severity and the absence of a graceful runtime mitigation or a telemetry detection path — vulnerable road user detection under adverse lighting is the canonical example. Classes with a credible mitigation, such as an operating-domain restriction plus a monitor, can legitimately ship as monitored, provided the mitigation is in place before rollout rather than promised.
Who decides ship vs hold, and what evidence does each role need to sign?
Three separate signatures: the model team signs that the class-level result is reproducible on a pinned harness, validation signs that test coverage represents the production distribution well enough to evaluate the threshold, and the release reviewer signs the verdict and each accepted residual risk. Collapsing any two of these into one role is what makes a rubric stop working under schedule pressure.
How do we record and track accepted residual risk so post-release failures are attributable?
Each accepted shortfall becomes a register entry carrying the scenario class, measured failure rate with its evidence class, the mitigation, the telemetry monitor, the accepting role, and a review date. A field failure inside a registered class then reduces to a bounded question — did the monitor fire, and did the rate exceed what was accepted — instead of reopening the entire release argument.
Where does this engineering release decision stop and the OEM’s safety case begin?
The rubric decides whether a software artifact enters a release channel with documented, monitored residual risk; it produces engineering validation evidence and names what it did not cover. Functional safety argumentation, hazard analysis, and homologation remain with the OEM or Tier 1, and the rubric’s output is an input to that case rather than a replacement for it.
If your next release review has no written exit criteria, the useful question is not whether the model is robust — it is who, specifically, will be asked to sign that it is, and what they will be shown.
Shipping Readiness Is a Tradeoff, Not a Threshold
Every additional point of recall costs you inference time, labeling budget, or both—so define what failure your business can actually tolerate. The teams that do tend to ship the boring, correct version first.
