“Edge case” is the least specific word in an automotive perception review. It can mean a night-time cyclist someone remembers, a tunnel exit that produced a screenshot, or an entire enumerated slice of the production driving distribution with a declared sample count and a measured failure rate. Only the last of those survives a release gate under questioning, because only the last one lets a reviewer see what was not tested.
This is the coverage checklist behind a perception robustness audit — the taxonomy that turns “we tested edge cases” into a table. It is not the release argument itself; deciding whether the measured numbers are good enough is a separate conversation with separate owners.
Why an anecdotal clip bag fails at the release gate
The ad-hoc pattern is familiar: a failure gets noticed in the field or in a demo, someone saves the clip, the clip joins a folder, and the folder becomes the team’s informal edge-case suite. It grows by memory. Every clip in it is real, which is why the approach feels rigorous.
The problem is not the clips. An enumerated edge-case taxonomy lets a reviewer distinguish untested from tested-and-passed; an anecdotal clip collection cannot, because it has no denominator. A clip bag reports what you looked at. It is structurally silent about the slices nobody thought of, and post-release surprises live almost entirely in that silence.
Coverage accounting is the fix, and it is cheap relative to a validation re-run. In our experience with perception validation work, the expensive failure is not a slice that failed — that is a known quantity you can mitigate — but a slice that was never enumerated, discovered late enough to force a full re-run of the validation cycle against a revised test set.
What are the axes of the edge-case taxonomy?
The edge space is enumerated along the axes the production driving distribution actually varies on. Five hold up across automotive perception programmes:
| Axis | Testable slice defined by | Typical failure signature |
|---|---|---|
| Environmental conditions | Precipitation type and intensity, fog visibility band, wet-road glare, spray from surrounding traffic | Contrast collapse; false negatives on distant objects; radar–camera disagreement |
| Lighting and dynamic range | Sun elevation and azimuth relative to heading, tunnel entry/exit transitions, headlight-only night, mixed artificial lighting | Blown highlights, clipped shadows, transient blindness across the exposure transition |
| Rare and confusable classes | Class taxonomy plus the confusion pairs that matter (cyclist vs motorcyclist, child vs short adult, debris vs pothole, unusual vehicle silhouettes) | Misclassification with high confidence; missing classes absent from training labels |
| Occlusion and truncation | Occluded fraction bands (e.g. 0–25%, 25–50%, 50–75%), frame-edge truncation, inter-object overlap | Track fragmentation; late detection of emerging objects |
| Sensor and mounting variance | Rig configuration, camera extrinsics within build tolerance, lens variant, degraded or soiled aperture, calibration age | Systematic geometric bias on one vehicle build; degradation invisible in the aggregate |
Each row is a slice family, not a single test. A slice becomes testable when it has a definition an annotator or a metadata pipeline can apply consistently — “heavy rain” is not a slice, “precipitation intensity above a stated threshold with the wiper signal active” is.
How do weather and lighting get represented without collecting every combination?
You cannot exhaustively collect the cross-product. Five weather states × six lighting states × four occlusion bands × three rig variants is 360 cells, most of which will never accumulate enough samples to mean anything.
The practical approach is to stratify on the axes independently, then designate a short list of interaction cells that are known to be jointly hard and sample those deliberately. Low sun plus wet-road glare is one such cell. Tunnel exit plus oncoming spray is another. Everything else is covered marginally — you can state the failure rate for the wet-road-glare slice and for the low-sun slice separately, and you declare the un-sampled interactions as out of scope rather than pretending the marginals imply them.
We treat that declaration as part of the deliverable, not a caveat buried in an appendix. Constructing the underlying dataset is its own discipline; the sampling strategy that feeds these slices is covered in how to build a test set that reflects production driving conditions.
How is sensor-mounting variance covered across a fleet?
This axis gets skipped most often, because the validation set usually comes from one or two well-calibrated development vehicles while the deployed fleet has build-to-build extrinsic spread, different lens batches, and calibration that ages.
Coverage here means two things. First, the test set includes data from more than one physical rig, with the extrinsic deltas recorded, so a systematic geometric bias on one build shows up as a per-rig failure rate rather than being averaged into the aggregate. Second, where multi-rig data does not exist, the audit exercises the model against controlled perturbations of the calibration parameters — small rotations and translations within the stated build tolerance, plus soiling and partial-occlusion simulations on the aperture — and reports sensitivity rather than claiming fleet coverage it does not have. The distinction between what is fixed at integration and what drifts in service matters here, and it maps onto the intrinsic-versus-extrinsic split in camera calibration.
How many samples does a slice need?
Enough that its measured failure rate is not an artefact of the sample size. A slice with eleven frames and one miss does not have a 9% failure rate; it has no meaningful rate at all, and reporting one is worse than reporting a gap.
The workable discipline is to set a minimum sample count per slice before the audit runs, derived from the failure rate you would need to detect to change a release decision, and then to report three states per slice rather than two:
- Covered — sample count at or above the declared minimum; failure rate reported with its confidence interval.
- Under-sampled — slice exists in the test set but below the minimum; the observed failures are listed as qualitative findings, not a rate.
- Out of scope — slice enumerated and deliberately excluded, with the reason recorded (no available data, not present in the target operational domain, deferred to a later release).
That third state is the one that does the work. A reviewer reading a table where every enumerated slice carries one of these three labels knows the shape of the evidence. A reviewer reading a list of passed tests does not.
This coverage table is where the taxonomy lands physically — inside the validation evidence pack, alongside the per-slice numbers and their traceability links. The harness that produces and re-produces it as the model and test set change is the Production AI Monitoring Harness, and the broader engineering practice around perception model validation sits under our computer vision work.
What this taxonomy does not establish
An enumerated edge-case taxonomy is a coverage instrument. It states which slices were exercised, at what sample counts, and with what measured failure rates. It does not establish that the residual failure rate on covered slices is acceptable, that the un-enumerated remainder is empty, or that any safety property holds. Functional safety argumentation, hazard analysis, and homologation remain the OEM’s or Tier 1’s safety case, and the audit is one engineering-validation input into it.
What the taxonomy changes is the quality of the argument available at the gate. Instead of a debate about whether the team tested enough, the room gets a table and a smaller, sharper question: which of these declared out-of-scope slices are we willing to carry into production, and who signs that?
We develop the wider argument — what robustness means as a measured property rather than a benchmark score — in What Robustness Means for an Automotive Perception Model — In Practice.
Frequently Asked Questions
What does “what edge cases a perception robustness audit actually covers” mean in practice? Does your perception system degrade gracefully when rain obscures half the camera lens at dusk? It means the audit ships an enumerated list of scenario slices with a coverage state attached to each one, rather than a narrative claim that edge cases were tested. In practice a reader of the audit can point at any slice in the taxonomy and see whether it was covered, under-sampled, or declared out of scope.
What are the axes of the edge-case taxonomy, and how is each defined as a testable slice? Five axes: environmental conditions, lighting and dynamic range transitions, rare and confusable object classes, occlusion and truncation regimes, and sensor-mounting variance. Each becomes testable only when its definition is mechanically applicable — a threshold, a band, a recorded metadata field or an annotator rule — so that two people slicing the same test set produce the same partition.
How do weather and lighting conditions get represented without exhaustively collecting every combination? By stratifying the axes independently and then deliberately sampling a short list of interaction cells known to be jointly hard, such as low sun with wet-road glare. The remaining interactions are declared un-sampled rather than treated as implied by the marginal slices.
How does the audit cover sensor mounting variance and rig-to-rig differences? Where multi-rig data exists, failure rates are reported per rig with the extrinsic deltas recorded, so a build-specific geometric bias cannot hide inside the aggregate. Where it does not, the audit reports sensitivity to controlled calibration perturbations within build tolerance and states plainly that fleet-wide coverage was not achieved.
How many samples does a slice need before its measured failure rate means anything? Enough to detect the failure rate that would change the release decision — a minimum set before the audit runs, not after seeing the results. Below that threshold the slice is reported as under-sampled with qualitative findings, because a rate computed from a handful of frames misleads more than a declared gap does.
How do you record slices that were knowingly excluded? Every enumerated slice carries one of three states — covered, under-sampled, or out of scope — and out-of-scope entries record the reason, whether that is absent data, an operational domain that excludes the condition, or a deliberate deferral. This is what lets a reviewer separate untested from tested-and-passed.
What does this coverage taxonomy deliberately not establish about safety? It establishes coverage and measured failure rates, not sufficiency. Judging whether the residual risk is acceptable, performing hazard analysis, and assembling the functional-safety argument stay with the OEM or Tier 1 safety case; the audit feeds it engineering-validation evidence and its own scope boundaries.
Five edge-case categories worth auditing first
Low-sun glare, retroreflective signage, partial occlusion, sensor-blockage handoff, and construction-zone clutter appear in every production fleet log we’ve examined. Revisit it when your workload shifts.
