Search for “machine learning autonomous vehicles” and the mental picture that forms is usually a single model that learns to drive — feed it enough road video, watch its benchmark accuracy climb, and once the number looks good, the car is safe. That picture is wrong in a specific, costly way. The learned model is not the vehicle’s safety story; it is one component that produces detections and estimates which the rest of the system has to bound, monitor, and be ready to override. That distinction is not pedantry. It decides how you structure the model work, what evidence you collect, and which claims survive review. Teams that map where learning ends and system-level safety reasoning begins build models that feed a wider argument. Teams that conflate model performance with vehicle safety end up overclaiming from metrics — and then absorbing re-review rounds when someone points out that a high mean Average Precision does not, on its own, demonstrate anything about how the vehicle behaves when the model is wrong. What does machine learning in an autonomous vehicle actually do? In practice the ML in an AV lives almost entirely in perception. A network — often a convolutional backbone or a transformer-based detector such as DETR, sometimes a bird’s-eye-view fusion model built on top of camera and LiDAR features — takes raw sensor input and produces structured outputs: bounding boxes with class labels, segmentation masks, keypoints, depth estimates, tracked object trajectories. Each output carries an implicit or explicit confidence. The learned component is powerful precisely because it does something hard to hand-code: it turns pixels and point clouds into semantic objects. But notice what it does not do. It does not decide to brake. It does not reason about whether its own detection is trustworthy in fog. It does not know that the pedestrian it just dropped for two frames is still physically present. Those jobs belong to layers downstream of the model, and treating them as if the model handles them is where the naive framing breaks. We see this pattern regularly: a team reports strong detection accuracy on a held-out set and treats that as the headline safety result. The accuracy is real. The inference that the vehicle is therefore safe is not supported by it. The gap between those two statements is the entire subject of this article. If you want the deeper mechanics of how these networks turn sensor data into detections, our walkthrough of how the autonomous-driving perception pipeline actually works covers the data path in detail; here the focus is on where that learned output sits inside the larger system. Where the learned model sits in the perception-planning-control pipeline An autonomous driving stack is conventionally decomposed into three layers: perception, planning, and control. Perception builds a model of the world from sensors. Planning decides what the vehicle should do given that world model. Control executes the plan through steering, throttle, and braking commands. Machine learning saturates the first layer, appears selectively in the second, and is usually kept out of the third for reasons of verifiability. The learned perception model, then, sits at the front of a chain of consumers. Its outputs are inputs to sensor fusion, object tracking, prediction, and ultimately trajectory planning. Every layer after it is entitled to distrust it. That distrust is not a design flaw — it is the mechanism by which a probabilistic component gets embedded in a system that has to make hard safety guarantees. The table below makes the division of labour concrete. What the model produces vs. what the system must add Layer Owned by learning? Produces What it cannot guarantee on its own Sensor input No Calibrated camera / LiDAR / radar frames Coverage in all conditions Perception (detection, segmentation) Yes Boxes, masks, classes, confidences That a missed detection is caught Tracking / fusion Partly Persistent object IDs, state estimates Recovery from a single-frame model error Prediction Partly Future trajectories of agents Bounded behaviour under distribution shift Planning Rarely Vehicle trajectory — Control No Actuator commands — Safety monitoring No Plausibility checks, fallbacks, degraded modes (this is where it lives) The pattern is that learning concentrates where the input is unstructured and the mapping is hard to specify, and thins out as you move toward decisions that must be auditable. The safety argument is assembled across the whole chain, not read off the perception row. Why strong model accuracy does not mean the vehicle is safe This is the claim that costs teams the most when they miss it. High benchmark accuracy — say, a strong mean Average Precision on a public detection benchmark like nuScenes or a KITTI-style split (a benchmark-class figure, tied to a named dataset and split) — tells you the model is good at the distribution that benchmark represents. It does not tell you: How the model fails. A model with 99% recall can still miss the one pedestrian that matters, and average metrics hide the shape of the tail. What the operational domain looks like versus the benchmark domain. Distribution shift — night, rain, unusual agents, sensor degradation — is exactly where safety-relevant errors cluster and where held-out accuracy is least representative. Whether downstream layers catch the model’s errors. Safety is a property of the whole loop, including tracking persistence, fusion redundancy, and fallback behaviour, not of the detector in isolation. Put plainly: accuracy is a statement about a distribution; safety is a statement about behaviour under the worst realistic conditions the vehicle will meet, including when the model is wrong. Those are different claims backed by different evidence. Conflating them is the source of the overclaiming this hub is built to prevent. The failure modes that survive strong benchmark numbers are worth studying directly — our piece on the perception failure modes that survive benchmarks works through the tail cases in detail. Which model outputs become evidence a validation pack can capture If you structure the model work correctly, its outputs do double duty: they feed the driving system and they become evidence surfaces for a safety argument. The trick is knowing which outputs carry evidential weight and being honest that the model producing them is not itself a safety case. Evidence surfaces worth capturing from the learned model Per-class, per-condition performance — not one headline number, but recall and precision sliced by object class, distance band, lighting, and weather. This is what lets a reviewer see the failure shape. Calibrated confidence behaviour — whether the model’s confidence scores actually correspond to correctness, so downstream thresholds mean something. Detection stability across frames — flicker and dropout rates, because a single-frame miss that tracking recovers is a very different risk than a sustained one. Behaviour under known degradations — how outputs change when calibration drifts or a sensor is occluded, which connects perception evidence to the calibration and reliability layers. Version and provenance metadata — which model weights, on which data, produced a given detection, so the evidence is traceable rather than anecdotal. These are the surfaces a production monitoring harness is designed to capture, feeding the wider validation pack without pretending the model is the case. What gets captured, and how, is downstream of the framing here — the point is that the model’s honest job is to produce evidence about its own behaviour, not to assert the vehicle is safe. Our discussion of what the model owes an ASIL D pack makes the evidence obligations concrete for the highest integrity levels. How to structure model work so it feeds the safety argument The reframe changes engineering priorities in a way that pays back quickly. Instead of chasing a single accuracy number, you invest in understanding and instrumenting the model’s failure behaviour, because that is what the downstream reasoning consumes. Concretely, that means the perception model’s outputs are treated as claims that the reliability discipline must bound — the connection to system-level reliability engineering that reasons about failure behaviour beyond model accuracy is where model evidence becomes a safety posture. This division of labour is also why computer vision engineering and safety-case engineering are distinct competencies that have to interlock. The vision work — architecture choice, training, calibration, the honest characterisation of where the model breaks — is what we do on the computer vision side of an AV programme. The value of doing it well is measured not in leaderboard position but in fewer re-review rounds, because the perception evidence arrives already scoped to what it can and cannot demonstrate. A quick self-check before you claim the model is “done” Have you characterised failures by condition, not just reported an average? Do your confidence scores mean something downstream, or are they uncalibrated? Can a reviewer trace any given detection back to a model version and dataset? Have you written down, explicitly, what the model does not demonstrate about vehicle safety? Does a downstream layer catch each class of model error you can name? If any answer is “no,” the model work is feeding a weaker argument than it could be — and that gap tends to surface as review friction rather than as an honest early flag. FAQ How does machine learning autonomous vehicles work in practice? In practice, machine learning in an AV lives mainly in the perception layer: a learned network turns raw camera, LiDAR, and radar data into structured detections, segmentation, depth, and tracked trajectories, each with a confidence. It does not decide to brake or judge its own trustworthiness — those are downstream jobs. The learned model is one component that produces estimates the rest of the system must bound, monitor, and be ready to override. Where does the learned perception model sit within the wider AV perception-planning-control pipeline? It sits at the front of the chain, in the perception layer, feeding sensor fusion, tracking, prediction, planning, and control in turn. Learning saturates perception because the input is unstructured, appears selectively in prediction and planning, and is usually kept out of control for verifiability. Every layer after the model is entitled to distrust its outputs, which is how a probabilistic component gets embedded in a system that must make hard safety guarantees. Why does strong model accuracy not, on its own, mean the vehicle is safe? Accuracy is a statement about a distribution — how good the model is on the data its benchmark represents. Safety is a statement about behaviour under the worst realistic conditions, including when the model is wrong. A high mean Average Precision hides the shape of the failure tail, says nothing about distribution shift into night or rain, and does not show whether downstream layers catch the model’s errors. Those are different claims backed by different evidence. Which outputs from an ML perception model become evidence surfaces a validation pack can capture? The evidentially useful outputs are per-class and per-condition performance (not one headline number), calibrated confidence behaviour, frame-to-frame detection stability, behaviour under known sensor and calibration degradations, and version-plus-provenance metadata that makes any detection traceable. These are what a production monitoring harness captures to feed a wider validation pack — while remaining honest that the model producing them is not itself a safety case. How should teams structure ML model work so it feeds the system-level safety argument rather than substituting for it? Stop optimising a single accuracy number and start instrumenting the model’s failure behaviour, because that is what downstream reasoning consumes. Characterise failures by condition, calibrate confidences so thresholds mean something, keep detections traceable to a model version, and write down explicitly what the model does not demonstrate about vehicle safety. The payoff is fewer re-review rounds, because perception evidence arrives already scoped to what it can and cannot prove. What claims about autonomous vehicle safety should teams avoid making from model metrics alone? Avoid any claim that a benchmark accuracy figure demonstrates the vehicle is roadworthy or safe. A metric shows performance on a distribution; it does not establish behaviour under distribution shift, the shape of the failure tail, or whether the surrounding system recovers from model errors. Reserve safety claims for the whole-loop argument that spans perception, tracking, fusion, planning, and monitoring — the model’s honest role is to supply bounded evidence into that argument, not to be it. Where does the honest line actually fall? It falls at the boundary between “the model detects this class of object well under these conditions” — which learning can demonstrate — and “the vehicle is safe” — which only the whole system, reasoning about what happens when the model is wrong, can argue. Teams that keep that line sharp spend their effort where it changes the safety posture rather than where it only moves a score.
