“The model learns to see the road.” That sentence is where most descriptions of autonomous vehicle machine learning stop, and it is exactly where the interesting problem begins. A perception network that scores well on a held-out validation set is not the same thing as a perception function that can be defended against a safety goal. The gap between those two things is where teams new to the domain lose months. Here is the core claim, stated plainly: machine learning is what makes modern perception possible, but a probabilistic model does not excuse the system from the requirement-to-test traceability an ASIL D classification demands. The model still owes the safety argument a description of where it fails, under what conditions, and how each of those behaviours ties back to a requirement. Aggregate accuracy is not that description. How does autonomous vehicle machine learning work in practice? At the level of mechanism, the story is familiar. A deep neural network — typically a convolutional backbone or, increasingly, a transformer-based architecture — is trained on labelled sensor data to produce outputs a downstream planner can act on: bounding boxes for other road users, drivable-space segmentation, lane geometry, sometimes a bird’s-eye-view occupancy grid. The network learns a function from raw pixels (and often lidar or radar returns) to those structured outputs by minimising a loss over millions of labelled examples. Frameworks like PyTorch or TensorFlow handle the training loop; a runtime such as TensorRT or ONNX Runtime handles the deployed inference on the vehicle’s compute platform. That is the mechanism. What it means in practice is narrower than the mechanism suggests. The model does not “understand” the road. It has learned a statistical mapping that holds well on data resembling its training distribution and degrades — sometimes gracefully, sometimes not — as inputs drift away from that distribution. A pedestrian in an unusual pose, a truck carrying reflective cargo, heavy rain at dusk: these are not exotic. They are the ordinary edges of the operating envelope, and the model’s behaviour at those edges is the part a safety argument actually cares about. We see this repeatedly: a team can describe its training pipeline in exhaustive detail and still be unable to say, in requirement terms, what the model does when it is wrong. That inability is the real deliverable gap. What are the main stages of an ML perception pipeline, and where does each carry evidence weight? It helps to walk the pipeline stage by stage and ask, at each one, what a reviewer will later need to see. The stages are ordinary; the evidence obligations attached to them are what most teams underestimate. Stage What happens Evidence weight for an ASIL D pack Data Sensor capture, labelling, dataset curation and splitting Provenance and coverage: does the dataset represent the declared operational design domain, and can you show which conditions are under-represented? Training Loss minimisation, augmentation, hyperparameter selection Reproducibility: fixed seeds, versioned data and code, so a specific model artifact traces to a specific training configuration. Validation Held-out evaluation, metric computation Failure characterisation, not just top-line metrics: per-condition performance, confusion structure, degradation curves. Deployment Quantisation, runtime compilation, on-target inference Behavioural equivalence: does the TensorRT-compiled, quantised model behave within tolerance of the validated one? Monitoring In-field performance and drift detection Continued validity: evidence that the operating boundaries assumed at validation still hold in the field. Every stage carries weight, but the two that teams most often treat as bookkeeping — data provenance and deployment equivalence — are the two that generate the most reviewer clarification cycles in our experience (observed across engagements; not a benchmarked rate). A model that was validated in full precision and shipped after INT8 quantisation, with no evidence that the two behave equivalently on safety-relevant cases, is a common and avoidable gap. Why is aggregate model accuracy not enough to support an ASIL D safety goal? Accuracy is a single scalar summarising performance over a distribution. A safety goal is a statement about behaviour under specified conditions. These are different kinds of object, and one does not convert into the other. Consider a pedestrian detector reported at 98% recall. That number is a weighted average over the validation set. It says nothing about recall on pedestrians in the specific conditions a safety goal names — low sun angle, partial occlusion by parked vehicles, dark clothing at night. If the validation set contains few such cases, the 98% can be simultaneously true and useless for the argument. This is a benchmark-class claim about the model, being asked to do the work of a safety claim, and failing. The correct framing treats the model’s failure modes as first-class evidence surfaces from the start. Instead of “the model achieves 98% recall,” the defensible statement is “under condition set X the model achieves recall Y, and its residual failure mode is Z, which the system mitigates through W.” That is a claim a safety argument can consume. Our discussion of what each machine learning performance metric actually proves develops this distinction in more depth — an aggregate metric is a summary, not a guarantee about any specific operating point. How do you characterise failure modes and operating boundaries so a safety argument can consume them? The shift is from how well does it work to where and how does it fail. Characterisation means deliberately partitioning the input space along axes that matter for safety — lighting, weather, object scale, occlusion, distance, object class — and reporting performance and failure structure within each partition rather than across all of them. A few concrete practices make this tractable: Slice the evaluation set by operating condition and report metrics per slice, so degradation is visible rather than averaged away. Characterise the failure, not just its rate. A false negative on a distant pedestrian and a false negative on a near one are different safety events; the pack needs to distinguish them. Name the operating boundary explicitly. State the conditions under which the model’s behaviour is claimed to hold, and treat everything outside that as out-of-scope for this function — which forces a system-level mitigation decision rather than an implicit assumption. This is the same discipline that underlies our treatment of the perception failure modes that survive benchmarks: the failures that matter are precisely the ones a good aggregate score hides. Characterising them is not a documentation chore layered on at the end; it is a modelling and evaluation decision made at the start, because you cannot retroactively slice an evaluation you never designed to be sliced. How does a probabilistic ML function get tied back to a specific safety requirement? This is the hinge of the whole argument. An ASIL D classification, under ISO 26262 and the ML-safety guidance in ISO/PAS 8800, demands requirement-to-test traceability: every safety requirement links to the evidence that it is met. A probabilistic function does not sidestep this — it changes what the evidence looks like. The link runs like this. A safety goal decomposes into safety requirements. A requirement allocated to the perception function becomes a statement about the function’s behaviour under named conditions — for example, “detect a standing pedestrian within the declared operating domain at range R with detection rate no worse than P.” That requirement then maps to specific tests: an evaluation slice constructed to exercise exactly those conditions, with an acceptance criterion tied to the required rate. The model’s characterised behaviour on that slice is the evidence. What makes the function traceable is not that it is deterministic — it is not — but that its probabilistic behaviour has been bounded and measured against a stated requirement under stated conditions. The probability is part of the claim, not an escape from it. Our walk-through of how machine learning in self-driving cars produces traceable evidence follows this thread from the model output back to the requirement it satisfies. The broader relationship between perception models and automotive safety demands is the subject of the machine learning in autonomous vehicles overview, which sits at the centre of this cluster; the concept here grounds the ML side of that argument. All of this rests on the underlying perception engineering we describe on our computer vision practice page. What ML-specific gaps most often trigger a reviewer clarification cycle? From the pattern we see across safety-relevant perception engagements, a short list of gaps generates a disproportionate share of clarification rounds (observed pattern; not a published survey figure): Aggregate metric with no per-condition breakdown — a single accuracy number offered where the requirement is condition-specific. Dataset with no coverage argument — no statement of which operating conditions are represented and which are thin. No deployment equivalence evidence — the validated model and the quantised, runtime-compiled model treated as interchangeable without proof. Failure rate reported, failure mode not characterised — the reviewer cannot tell whether a residual false negative is a benign or a hazardous event. Operating boundary left implicit — no explicit statement of where the function is and is not claimed to work. Each of these is a traceability break: a number exists, but the reviewer cannot connect it to a requirement without asking. The clarification cycle is the cost of that missing connection. Framing the model’s behaviour around its failure modes and operating boundaries from the start is what closes the loop before the pack reaches the reviewer — which, in practice, is where the reduction in rework and the shorter gap-finding loop actually come from. FAQ What’s worth understanding about autonomous vehicle machine learning first? A deep neural network learns a statistical mapping from sensor data to structured perception outputs — bounding boxes, segmentation, lane geometry — by minimising a loss over labelled examples. In practice this means the model performs well on inputs resembling its training distribution and degrades toward the edges of its operating envelope, so its behaviour at those edges is what a safety argument must describe. What are the main stages of an ML perception pipeline, and where does each carry evidence weight? Data, training, validation, deployment, and monitoring. Data carries provenance and coverage weight; training carries reproducibility weight; validation carries failure-characterisation weight; deployment carries behavioural-equivalence weight between the validated and the quantised runtime model; monitoring carries continued-validity weight. Data provenance and deployment equivalence are the stages most often underestimated. Why is aggregate model accuracy not enough to support an ASIL D safety goal? Accuracy is a single scalar averaged over a distribution, while a safety goal is a statement about behaviour under specified conditions. A high aggregate number can be simultaneously true and useless if the validation set contains few of the safety-relevant cases the goal names, so the metric cannot do the work of a safety claim on its own. How do you characterise an ML perception model’s failure modes and operating boundaries so a safety argument can consume them? Partition the input space along safety-relevant axes — lighting, weather, occlusion, object scale, distance — and report performance and failure structure per slice rather than in aggregate. Characterise the nature of each failure, not just its rate, and state explicitly the conditions under which the model’s behaviour is claimed to hold. How does a probabilistic ML function get tied back to a specific safety requirement in a traceable way? A safety requirement allocated to the perception function becomes a statement about behaviour under named conditions, which maps to an evaluation slice built to exercise those conditions with an acceptance criterion. The model’s measured behaviour on that slice is the evidence. Traceability comes from bounding and measuring the probabilistic behaviour against a stated requirement, not from making the function deterministic. What ML-specific gaps most often trigger a reviewer clarification cycle on a safety-relevant perception function? The recurring gaps are: an aggregate metric with no per-condition breakdown, a dataset with no coverage argument, no deployment-equivalence evidence between validated and quantised models, a failure rate reported without a characterised failure mode, and an implicit operating boundary. Each is a traceability break the reviewer must resolve by asking. How does machine learning fit into the requirement-to-test traceability an ASIL D perception validation pack must carry? The ML function is one node in the traceability chain: safety goal to safety requirement to test to evidence. The model contributes characterised behaviour — per-condition performance and failure modes bounded against requirements — as the evidence that closes specific links. It does not sit outside the chain because it is probabilistic; its probabilities become part of the claims the chain records. Where the argument lands Machine learning earned its place in perception because nothing else matches it for extracting structure from raw sensor data. That is not in dispute. What is easy to miss is that the model’s probabilistic nature does not lower the bar the safety case sets — it raises the specificity the evidence must reach. The question worth sitting with is not “how accurate is the model” but “for each safety requirement this function is allocated, can I point to the characterised behaviour, under named conditions, that satisfies it?” When the answer is a slice of evaluation data with an acceptance criterion rather than a headline number, the model is finally speaking a language the ASIL D pack can consume. That characterisation discipline is exactly what a reviewer-structured validation pack — the kind our production AI monitoring harness is built to carry — depends on to close the requirement-to-test loop.
