A perception test suite and a production fleet are usually two separate record systems. Benchmark scores live in the validation report; field behaviour lives in telemetry dashboards. Nothing structural connects them, so the reviewer is quietly asked to assume that the model measured on the bench is the model behaving in the vehicle, under the same conditions, in the same proportions.
That assumption holds until someone tests it. The question that breaks an untraced pack is small and specific: how do you know this test predicts what the vehicle does at dusk in rain? A pack with a working trace answers with a named condition slice, the test cases that populate it, and the production monitor emitting the same tag. A pack without one answers with an aggregate mAP figure and buys a clarification round.
Linking test results to production behaviour is a trace discipline, not a safety argument. It evidences one thing: that test coverage and field exposure describe the same population. That is narrower than it sounds, and considerably more useful than a benchmark average.
What a test-to-production trace actually is
The trace is a shared vocabulary plus two annotated record sets. Every test case carries the operating conditions it represents. Every production monitor emits the same condition tags against the same taxonomy. Once both sides speak the same language, the comparison is arithmetic rather than rhetoric: you can put measured field distribution next to test distribution and read where they align.
The trace does not prove the model is safe; it proves the tests and the fleet describe the same operating population. Everything downstream — hazard linkage, residual risk, ASIL decomposition — is a separate argument built on different evidence, and conflating the two is how validation packs get cited in roles they were never built for.
In our experience the hard part is not instrumentation. It is agreeing on the taxonomy once, before either side is annotated, and resisting the urge to let the test suite and the telemetry pipeline evolve separate tag vocabularies over the following six months.
What has to be tagged for the two sides to be comparable?
A condition tag is only useful if it can be assigned on both sides from data that both sides actually have. Ego speed is available in test metadata and in vehicle CAN logs. “Difficult lighting” is available in neither, reliably. The taxonomy has to survive that constraint.
| Tag dimension | Test-side source | Production-side source | Comparable? |
|---|---|---|---|
| Illumination band (day / dusk / night / tunnel) | Scenario metadata, sun-angle from clip timestamp + GPS | Ambient light sensor, timestamp + position | Yes |
| Precipitation state | Recorded scenario label or synthetic augmentation flag | Wiper state, rain sensor, weather service join | Yes, with agreed thresholds |
| Ego speed band | Simulation config or log replay | CAN bus | Yes |
| Road class | Map annotation on the clip | Map match on the live route | Yes |
| Sensor health / degradation | Injected fault case | Self-diagnostics, calibration drift monitor | Yes |
| Occlusion severity | Ground-truth annotation | Not directly observable | No — proxy only |
| Object class distribution | Label counts | Detector output counts (biased by the model itself) | Partially; state the bias |
The last two rows matter more than the ones that work cleanly. A trace that silently drops the dimensions it cannot measure in production is a trace with a hole in it. Name the proxy, name its bias, and let the reviewer judge — that is a stronger position than a matrix that looks complete because the awkward rows were deleted.
Building the coverage comparison
Once tags exist on both sides, the coverage view is a condition matrix: field exposure share per slice, test case count per slice, and the resulting verdict. Three verdicts are enough.
- Covered — the slice carries meaningful field exposure and test cases proportionate to it.
- Over-tested — heavy test coverage, negligible field exposure. Not a defect; often a deliberate stress choice. Say so.
- Uncovered — measurable field exposure, no test cases or too few to support a claim. This is the row the reviewer is looking for.
Reporting uncovered slices is the point of the exercise, not a failure of it. A pack that shows three uncovered slices with a named plan for each closes faster than a pack that shows none, because the second one is either wrong or is hiding the taxonomy granularity that would reveal them. We see this pattern regularly: the packs that clear first-pass review are the ones that pre-empt the reviewer’s own gap analysis.
Granularity is where judgement enters. Slice too coarsely and every row reads “covered” while a genuine dusk-plus-rain deficit hides inside “adverse conditions”. Slice too finely and the matrix becomes a thousand rows of statistical noise. A practical starting point is a two- or three-dimensional cross of the tags that drive the largest known performance deltas for your detector — typically illumination × precipitation, plus ego-speed band if the perception stack is latency-sensitive.
Tracing an incident backwards
The trace runs in both directions, and the reverse direction is the one that gets exercised at 2am. A field incident arrives with a timestamp and a vehicle ID. The monitor tags attached to that event resolve to a condition slice. The slice resolves to the test cases assigned to it. Now there are exactly two outcomes, and both are informative.
Either a test case existed for that slice and passed — in which case the test is not discriminating, and the failure is in test design rather than coverage. Or no test case existed, and the incident sits in a slice the coverage matrix already flagged as uncovered, which turns an unexplained field failure into a known gap with a scheduled fix.
Time-to-answer on that lookup is the operational metric worth tracking. When the trace is a spreadsheet reconciled by hand, the answer takes days and arrives after the customer has drawn their own conclusion. When condition tags are emitted by the monitoring harness in the same schema the test manifest uses, it is a query. This is the layer that connects a production AI monitoring harness to the test cases cited in the validation pack, and it is what makes the pack answerable rather than merely complete.
Divergences a reviewer expects, and ones they will not accept
Not every gap between bench metrics and field behaviour is a defect. Some are structural and should be stated up front.
| Divergence | Reviewer reading |
|---|---|
| Field precision below bench precision on rare classes | Expected — label prevalence differs; state the shift |
| Latency higher in vehicle than on the test rig | Expected if the target hardware and thermal envelope are named |
| Recall drop confined to an over-tested slice | Expected; over-testing implies a stress condition |
| Field distribution containing slices absent from the suite | Gap — must be named with a closure plan |
| Aggregate metrics quoted with no slice decomposition | Gap — the reviewer cannot verify anything |
| Trace built against a superseded model build | Gap — provenance failure, invalidates the comparison |
The last row is the one that decays quietly. A trace is valid only for the model artefact, sensor configuration, and test suite version that produced it. When any of those change, the mapping from test case to condition slice has to be re-asserted rather than inherited — which is why the condition taxonomy should be versioned separately from both, so a model retrain does not force a taxonomy rebuild. Keeping the surrounding evidence surfaces in step across releases is a related discipline, covered in how to keep a perception validation pack current as the model updates.
Where this sits in the pack
In the evidence package the trace appears as three linked surfaces: the condition taxonomy definition, the coverage matrix with per-slice verdicts, and the test-ID index that resolves each slice to the cases behind it. The wider structure of the package — which surface answers which reviewer question, and in what order — is set out in the parent discussion of structuring a perception validation evidence package.
The reusable asset here is the taxonomy, not the matrix. The matrix is regenerated at every model revision; the taxonomy, if it was designed against data both sides actually emit, survives several. That is the difference between rebuilding the trace each release and refreshing it.
One thing stays genuinely unresolved: how fine a slice can get before per-slice sample counts stop supporting any claim at all. There is no clean threshold, and teams that pick one arbitrarily end up defending the threshold instead of the model. Naming the smallest slice you are willing to make a claim on, and why, is currently a better answer than pretending the question has a general solution.
Frequently Asked Questions
What does linking perception test results to production behaviour mean in practice?
A common Link Perception Test Results question is worth clarifying. It means annotating both record systems against one shared condition taxonomy, so each test case declares the operating conditions it represents and each production monitor emits the same tags. The comparison then becomes arithmetic — measured field distribution against test distribution — instead of an assertion that the two must correspond.
What has to be tagged on a test case and on a production monitor for the two to be comparable at all?
Only dimensions both sides can actually observe: illumination band, precipitation state, ego-speed band, road class, and sensor-health state all resolve from data available in test metadata and in vehicle logs. Dimensions like occlusion severity exist in ground truth but not in production, so they enter the trace as named proxies with their bias stated, not as clean rows.
How do we show that the field operating-condition distribution is covered by the test distribution — and report the slices that are not?
Build a condition matrix giving field exposure share and test case count per slice, then assign each row one of three verdicts: covered, over-tested, or uncovered. Uncovered rows are reported deliberately with a closure plan attached; a matrix showing no gaps is usually a granularity problem rather than a coverage achievement.
How do we trace a field incident back to the test case that should have caught it?
Resolve the incident’s monitor tags to a condition slice, then resolve that slice to its assigned test cases. Either a case existed and passed — a test-design problem — or none existed, and the incident lands in a slice the coverage matrix already flagged. Time-to-answer on that lookup is worth tracking as its own metric.
Which divergences between bench metrics and production behaviour are expected, and which ones a reviewer will treat as a gap?
Precision shifts on rare classes, latency differences against named target hardware, and recall drops inside deliberately over-tested slices are all expected and should be stated up front. Field slices absent from the suite, aggregate metrics with no slice decomposition, and a trace built against a superseded model build are read as gaps.
How do we keep the trace valid when the model, the sensor stack, or the test suite changes?
Version the taxonomy separately from the model artefact and the test suite. The coverage matrix is regenerated per revision because it depends on both; the taxonomy, if designed against signals that persist across builds, survives several revisions and is the part that makes re-tracing cheap rather than a rebuild.
What does the trace evidence look like in the pack — table, condition matrix, or linked test IDs?
All three, as linked surfaces: the taxonomy definition, the coverage matrix carrying per-slice verdicts, and a test-ID index resolving each slice to the cases behind it. The index is what lets a reviewer spot-check any single row without asking for a supplementary submission.
What Link Perception Test Results changes
Link Perception Test Results is rarely the hard part — knowing which of its failure modes you can live with is. That answer is workload-specific, and it is worth writing down before you build.
