Ask most teams what “ML automotive” means and you get a version of the same answer: train a perception model, push its benchmark accuracy high enough, and call it production-ready. That answer is where the trouble starts. A perception model that scores well on a held-out set has demonstrated exactly one thing — that it fits the data it was measured against. It has said nothing about what happens the first time the road throws a condition the benchmark never contained. The trained model is the start of the work, not the deliverable. What actually ships into a vehicle program is a monitoring harness: a regression suite anchored to scenario coverage, drift telemetry keyed to sensor and geography shifts, and a release-readiness review that an engineering reviewer can sign. The accuracy number is an input to that harness. It is not the artefact that governs whether the model belongs in a car. How does ML in automotive actually work in practice? The visible part of automotive ML is familiar. A team collects sensor data — camera frames, LiDAR point clouds, radar returns — labels it, and trains a detector or segmentation network to identify vehicles, pedestrians, lane markings, and drivable space. Architectures like transformer-based detectors and convolutional backbones get tuned, quantized for the target compute, and evaluated against a validation set. On a good day the mean average precision looks strong and everyone moves on. The part that separates a demo from a deployable system is everything that happens after that number lands. A perception model operates inside a defined envelope: the range of weather, lighting, road types, sensor configurations, and geographies its training data represented. Inside that envelope, the benchmark is a reasonable proxy for behaviour. Outside it, the benchmark is silent — and the model does not announce when it has crossed the boundary. It just keeps producing detections, some of which are now wrong in ways no offline metric captured. This is the divergence point the naive view misses. The first field condition the benchmark never contained — a new weather regime, an unmodelled edge case, a degrading camera — is exactly where accuracy stops meaning what the team thinks it means. A model without a harness has no way to detect that its operating envelope has quietly moved. We see this pattern across reliability engagements: the model didn’t get worse in any dramatic way, the world it runs in shifted, and nothing was watching the seam. What a production monitoring harness for a perception model contains The harness is the set of running artefacts that keep a deployed perception model inside a known, defensible operating envelope. For an automotive perception workload it has three load-bearing parts. Scenario-coverage regression suite. Not a single test set, but a structured catalogue of driving scenarios — night urban, low-sun highway, heavy rain, construction zones, occluded pedestrians — each with its own labelled evaluation slice and its own pass threshold. A candidate model is scored per scenario, not just in aggregate, so a regression in “pedestrians at dusk” cannot hide behind a strong overall number. Drift telemetry. Continuous signals from the field that indicate the input distribution is moving away from what the model was validated on. Sensor degradation, weather-regime frequency shifts, and new geographies are the three that bite hardest in automotive. Telemetry doesn’t wait for a labelled incident; it watches the inputs and the output distribution and raises a flag when either drifts. Release-readiness review. A documented decision surface where an engineering reviewer compares a candidate model against the deployed baseline on like-for-like evidence and signs — or declines to sign — the release. The evidence is the regression results plus the drift posture, not a verbal assurance that “it looks better.” This is the automotive-perception instantiation of the general reliability harness we describe under production AI reliability. The general harness sections — regression over coverage, drift monitoring, sign-off — take a concrete perception shape here: scenario slices instead of generic test sets, sensor and geography drift instead of abstract feature drift, and a reviewer signing against a fleet-deployment decision. How scenario-coverage regression differs from benchmark accuracy Benchmark accuracy answers a summary question: across this dataset, how often is the model right? Scenario-coverage regression answers a governance question: for each condition we have committed to supporting, does this candidate hold or lose ground against the version already in the field? The difference is not cosmetic. A single mAP figure can rise while a specific safety-relevant slice quietly regresses, because aggregate metrics average away the tail. In a vehicle, the tail is where people get hurt. Structuring evaluation as a per-scenario regression — the same discipline we lay out for where reliability gates belong at each stage of an ML pipeline — forces the tail into view and gives QA a like-for-like basis for comparison. Benchmark accuracy vs scenario-coverage regression Dimension Benchmark accuracy Scenario-coverage regression Question answered How good is the model overall? Does the candidate hold per condition vs the deployed baseline? Granularity Single aggregate figure One scored slice per committed scenario Tail behaviour Averaged away Explicitly surfaced per slice Baseline comparison Optional Mandatory, like-for-like Governs release? No — an input Yes — a signable input to review Detects envelope drift? No Partially (coverage gaps become visible) The claim class here matters: per-scenario pass thresholds are benchmark-class evidence when the evaluation slices and thresholds are named and versioned in the harness. An unlabelled “it improved” is not evidence — it is a market-direction sentiment dressed as a measurement. Which drift signals matter for automotive perception? Three drift signals dominate in practice, and they fail differently. Sensor degradation is the quiet one. A camera lens hazes, a mount vibrates loose, a LiDAR unit’s returns weaken. The model’s inputs change gradually; its outputs degrade gradually; no single frame looks catastrophic. Input-side telemetry — sharpness statistics, exposure distributions, point-cloud density — catches this before the output metrics do. Weather-regime shifts are seasonal and geographic. A model validated through a temperate spring meets its first hard winter, or a fleet expands into a monsoon region. The frequency of conditions the model handles poorly rises even though each condition was technically “in the training set.” Monitoring the frequency distribution of inferred conditions against the validation distribution flags this. Geography shifts change the semantics of the scene: different lane-marking conventions, signage, vehicle types, pedestrian behaviour. This is the shift most likely to move the operating envelope wholesale, and it is the one a release-readiness reviewer should treat as a re-validation trigger, not a monitoring footnote. Structural relationships between scene entities are where graph-based methods help — a topic we go deeper on in graph neural network applications in automotive perception validation. Who signs off release-readiness, and against what evidence? Release-readiness is a human decision made against documented evidence, not an automated gate that flips green. In a mature automotive program an engineering reviewer — typically a validation or perception-QA lead — owns the sign-off. Their evidence package is specific: the scenario-coverage regression results comparing candidate to baseline, the current drift posture of the deployed system, and a record of any scenarios where coverage is known to be thin. The value of making this a documented, repeatable review is speed with accountability. A team with a harness in place can decide release-readiness in days because the evidence format is fixed and the comparison is like-for-like; the recurring cost of skipping it is re-litigating what “production-ready” means on every model update — and discovering silent envelope drift only after a field incident. When the decision is whether a specific perception model version is cleared for fleet deployment, this same review structure is what the AI infrastructure release decision leans on. Release-readiness sign-off checklist Candidate scored per committed scenario, not only in aggregate Every scenario compared like-for-like against the deployed baseline No safety-relevant slice regressed below its documented threshold Current field drift posture reviewed (sensor, weather, geography) Known coverage gaps documented and accepted or blocking Reviewer of record named and evidence package versioned How the harness gets re-run when the model is updated A harness earns its keep on the second model, not the first. When a perception model is updated — retrained on new data, re-quantized, or re-architected — the entire scenario-coverage regression re-runs against the same versioned slices, and the results are compared to both the previous candidate and the deployed baseline. Because the evaluation slices and thresholds are versioned artefacts, the comparison is genuinely like-for-like rather than a fresh interpretation each cycle. Drift telemetry feeds back into this loop. If the field has surfaced a new condition since the last release, that condition becomes a new scenario slice — the harness grows to cover what the world showed it. This is how the operating envelope stays honest over time instead of silently narrowing while the benchmark reports business as usual. The discipline mirrors what we describe for keeping learning systems reliable across updates in regression testing an RL agent that keeps learning, where the challenge is likewise that the artefact under test does not stand still. FAQ What does working with ML automotive involve in practice? In practice, automotive ML means training a perception model on sensor data to detect vehicles, pedestrians, lanes, and drivable space, then operating it inside a defined envelope of weather, lighting, geography, and sensor conditions. The training and benchmark step is the visible part; the deployable system is the trained model plus a monitoring harness that keeps it inside a known, defensible operating envelope. What does a production monitoring harness for an automotive perception model actually contain? It contains three load-bearing parts: a scenario-coverage regression suite that scores the model per driving condition against pass thresholds, drift telemetry keyed to sensor degradation and weather and geography shifts, and a release-readiness review where an engineering reviewer signs against documented evidence. Together these govern whether the model belongs in a vehicle, rather than a single accuracy figure. How is scenario-coverage regression testing different from benchmark accuracy for perception models? Benchmark accuracy is a single aggregate figure that answers “how good overall,” and it averages away the tail conditions where safety-relevant failures live. Scenario-coverage regression scores each committed condition separately against the deployed baseline, so a regression in a specific slice — pedestrians at dusk, for example — cannot hide behind a strong overall number. What drift signals matter for automotive perception — sensor degradation, weather regimes, geography shifts? All three, and they fail differently. Sensor degradation shifts inputs gradually and is caught by input-side telemetry before output metrics move; weather-regime shifts raise the frequency of poorly handled conditions; geography shifts change scene semantics and should trigger re-validation rather than being treated as a monitoring footnote. Who signs off release-readiness for a perception model, and against what evidence? An engineering reviewer — typically a validation or perception-QA lead — owns the sign-off, deciding against a documented evidence package: per-scenario regression results comparing candidate to baseline, the current field drift posture, and any known coverage gaps. Making this a fixed, repeatable review lets the team decide in days rather than re-litigating “production-ready” every cycle. How does the harness get re-run and re-validated when a perception model is updated? When a model is updated, the full scenario-coverage regression re-runs against the same versioned slices and thresholds, producing a like-for-like comparison against the previous candidate and the deployed baseline. Drift telemetry feeds new field conditions back in as new scenario slices, so the harness grows to cover what the world reveals instead of letting the operating envelope narrow silently. The uncomfortable truth about automotive ML is that the accuracy number never tells you when your model has left the world it was tested in — only a harness that watches the envelope can. The open question for any perception program is not “how accurate is the model” but “when the road changes, who finds out first, and against what evidence do they decide the model still belongs in the car.” That is a release-readiness question, and it is the one the monitoring harness — the automotive-perception variant of SVC-VALIDATION — exists to answer.
