Most perception teams do not design their test set. They inherit it — a public benchmark split, plus a folder of internally collected clips that someone remembered as hard — and then run a robustness audit on top of it. The audit is rigorous. The numbers it produces are not, because the population being measured has no defined relationship to the population the vehicle will drive in.
A test set that reflects production driving conditions is built the other way round: characterise the deployed fleet’s driving distribution first, then sample and stratify against that characterisation so every scenario class you care about has enough frames to yield a stable per-class failure rate. That is a sampling and coverage-accounting job, not a collection job. The deliverable is not “more data” — it is a known relationship between what you tested and what the vehicle will meet, plus an explicit statement of what falls outside it.
What “reflects production driving conditions” means in practice
It means three properties hold at once, and you can demonstrate each to a reviewer.
First, the axes are declared. You have written down which dimensions of variation the test set is stratified on, and why those and not others.
Second, each stratum is measurable, not merely present. A test set containing eleven fog frames does not measure fog performance; it decorates the coverage table. Per-scenario-class failure rates are the output the audit exists to produce, and a class with too few frames produces an estimate too noisy to act on.
Third, the gaps are named. Every real test set has holes — conditions the fleet meets that you could not sample, or sampled too thinly. Writing those down is what converts an audit result from a claim about the world into a claim about a bounded population. This is the property release reviewers probe first, and it is why the coverage documentation travels with the [validation evidence pack](Production AI Monitoring Harness) rather than sitting in an engineer’s notebook.
Characterising the distribution before you sample
The characterisation step is the part teams skip, and it is cheap relative to its leverage. You are building a frequency picture of the driving the deployed fleet actually does, from fleet telemetry, route definitions, ODD documentation, and — where the fleet is small — the operational plan.
Five dimensions carry most of the weight in the automotive perception work we see:
- Route geography and road class — motorway, urban arterial, residential, rural unlit, tunnels and structures. Road class changes object density, ego speed, and background clutter simultaneously.
- Time-of-day and lighting mix — not just day/night, but the transitions: tunnel entry and exit, low sun into the lens, dusk. Dynamic-range transitions behave differently from steady low light.
- Weather and road-surface state — rain intensity, spray, fog, snow, and wet-road glare, which is an optical condition distinct from the precipitation that caused it.
- Sensor rig configuration across the fleet — which camera and lidar variants, which mounting positions, which firmware and ISP settings are live in the field. A fleet is rarely one rig.
- Frequency of the edge classes that gate the release decision — the rare object classes, occlusion patterns, and confusable pairs your release criteria name. Their production frequency determines how much you must oversample to measure them at all.
Two things fall out of this immediately. Some dimensions are correlated (heavy rain concentrates in certain months and certain routes), so treating them as independent inflates your apparent coverage. And some cells of the resulting grid are empty in production — a fact worth recording, because a reviewer asking “what about snow?” deserves “the deployed fleet operates in two cities with no recorded snow days, so it is out of scope and monitored” rather than a shrug.
Stratified sampling, not exhaustive collection
Once the distribution is characterised, the sampling rule follows. You are not trying to reproduce the production distribution proportionally — that is the mistake that recreates the benchmark problem, because a proportional sample of real driving is overwhelmingly clear-daylight motorway, and the rare classes stay rare and unmeasurable.
You are doing stratified sampling with deliberate oversampling of the low-frequency, high-consequence strata, then reporting per-stratum results rather than a single pooled figure. Aggregate accuracy over a stratified set is close to meaningless and should not be quoted; the point of the design is that the per-class numbers become the output. Our colleagues working on computer vision systems hit the same structure in industrial inspection — the defect classes that matter are the ones the line produces least often.
How many frames per scenario class are enough?
Enough that the failure-rate estimate is stable under resampling. The practical procedure, rather than a universal number:
- Fix the decision you are making with the class — is it release-blocking, or monitored-with-mitigation? A blocking class needs a tighter interval.
- Estimate the failure rate roughly from whatever data you have, then compute the sample size that puts the confidence interval inside the tolerance your release criteria imply. Low failure rates need far more frames than intuition suggests; a class failing at 2% needs hundreds of independent frames before the estimate stops swinging.
- Count independent frames, not frames. Consecutive frames from one clip are near-duplicates. A 60-frame burst of one cyclist at one tunnel exit is roughly one sample of that scenario, not sixty — this is the single most common inflation in coverage tables we review.
- Where the class is too rare to reach sufficiency, say so in the coverage table and route it to field monitoring instead of pretending the test set covers it.
The distinction between a class you measured, a class you sampled too thinly, and a class you did not sample matters more than the exact thresholds. Three states, honestly labelled, beat one number.
Sensor and rig variance
Mounting variance is where distribution-matched test sets diverge most sharply from benchmark splits. Public datasets are typically collected on one rig; deployed fleets are not. Camera extrinsics drift with vibration and temperature, ISP tuning differs across hardware revisions, and a lidar mounted 4 cm lower changes ground-plane geometry.
Two practices handle this without multiplying the test set by every rig in the fleet. Sample real frames from at least the rig variants that dominate the fleet, so the model is exercised on genuine optical and geometric differences. Then use controlled perturbation — small extrinsic offsets, ISP parameter sweeps, synthetic degradation applied to real frames — to probe sensitivity around each variant, and label those results as sensitivity analysis rather than field measurement. Keeping the two evidence classes separate in the report is what stops a reviewer discounting both.
The coverage table
The artefact that makes all of this legible is a coverage table, maintained alongside the test set and shipped with the audit:
| Scenario class | Production frequency | Independent frames | Sufficiency | Failure rate | Status |
|---|---|---|---|---|---|
| Urban daylight, common classes | High | 4,200 | Met | measured, tight interval | Measured |
| Wet-road glare, low sun | Medium | 610 | Met | measured, wider interval | Measured |
| Night, unlit rural, pedestrian | Low | 180 | Partial | indicative only | Under-sampled — flagged |
| Fog, dense (<50 m visibility) | Low | 24 | Not met | not estimable | Gap — field monitoring |
| Snow / ice | Absent in current ODD | 0 | n/a | n/a | Out of scope — declared |
| Secondary rig variant (rev B mount) | ~18% of fleet | 900 | Met | measured per class | Measured |
The Status column is the part reviewers read. It is also what keeps the test set honest over time: a class that moves from Gap to Measured is real progress, and a class that quietly stays at Gap across three releases is a decision someone should be making explicitly.
Keeping it current
The production distribution moves. Routes expand, a new sensor revision enters the fleet, an OTA changes ISP behaviour, seasons rotate. A test set frozen at first release slowly stops describing the vehicle’s world, which is the failure mode behind most “the audit passed and then we got paged” stories.
The maintenance loop is short: re-derive the distribution characterisation from fleet telemetry on a fixed cadence, diff it against the strata the test set was built on, and treat any new or grown stratum as a sampling backlog item with the same sufficiency test. Where the fixed reference set itself needs to stay stable for comparability, hold it fixed and add a growing supplementary set — the reference set answers “did the model change?”, the supplementary set answers “did the world change?”. The hub article on perception robustness audits for automotive release decisions develops how those two questions feed a release gate.
Frequently Asked Questions
What does building a test set that reflects production driving conditions mean in practice? On Build Test Set Reflects, the short answer is this. It means the test set’s composition is derived from a written characterisation of the deployed fleet’s driving — road classes, lighting, weather, rig variants, edge-class frequencies — rather than inherited from a public split. In practice you produce three things: declared stratification axes, enough independent frames per stratum to estimate a failure rate, and a documented list of what is not covered.
How do we characterise the production driving distribution before we start sampling — what dimensions actually matter? Route geography and road class, time-of-day and lighting transitions, weather and road-surface state, sensor rig configuration across the fleet, and the production frequency of the edge classes your release criteria name. These five carry most of the variance in the automotive perception work we see. Watch for correlation between them — treating rain and season as independent overstates coverage.
How do weather and lighting conditions get represented without exhaustively collecting every edge case? By stratified sampling with deliberate oversampling of low-frequency strata, plus controlled perturbation of real frames to probe sensitivity around conditions you sampled thinly. The goal is a measurable failure rate per condition class, not a complete catalogue of weather. Conditions that remain unsampled are recorded as gaps and routed to field monitoring rather than assumed benign.
How many frames per scenario class are enough to produce a stable failure-rate estimate? Enough that the estimate holds under resampling within the tolerance your release criteria imply — which depends on the expected failure rate and whether the class is release-blocking. The critical correction is to count independent frames: a 60-frame burst from one clip is roughly one sample of that scenario, not sixty. Classes that cannot reach sufficiency should be labelled indicative, not reported as measured.
How do we account for sensor mounting variance and rig-to-rig differences across the deployed fleet in the test set? Sample real frames from the rig variants that make up meaningful shares of the fleet, so genuine optical and geometric differences are exercised. Then apply controlled extrinsic and ISP perturbations around each variant as sensitivity analysis, reported separately from field-measured results so the two evidence classes are not conflated.
How do we document what the test set does not cover so a release reviewer can judge the residual risk? Maintain a coverage table with a per-class status: measured, under-sampled, gap, or declared out of scope. Ship it with the audit output. A reviewer can accept a documented gap with a monitoring plan attached; an undocumented gap discovered during review usually costs a re-run of the audit.
How do we keep the test set current as routes, sensor configurations, and the production distribution shift after release? Re-derive the distribution characterisation from fleet telemetry on a fixed cadence and diff it against the strata the test set was built on; new or grown strata become sampling backlog with the same sufficiency test. Hold the original reference set fixed for comparability and grow a supplementary set alongside it, so you can separate model change from world change.
If your current test set could not fill in the Status column above without guessing, that is the finding — before any number the audit produces is worth arguing about.
Three constraints that simplify test-set design
Start with geographic coverage, then layer weather and traffic density—attempting all dimensions at once guarantees an unfinishable backlog. Everything else is detail.
