Ask most teams starting on machine learning for an autonomous car what they are building, and the answer is a perception model that beats the benchmark. That framing is where the trouble starts, because a benchmark score describes how often a model was right on a fixed dataset — it says nothing about what happens the moment the model is wrong on the road. A safety reviewer does not care how high your mean average precision climbed. They care whether you can name each way the perception output can fail, show that the system detects the failure, and show that it drives the vehicle to a defined safe state when it does. That gap — between a model that scores well and a model whose behaviour can be traced through a hazard argument — is the thing this article is about. The ML pipeline in an autonomous car is not the deliverable. It is one input into a safety case, and understanding where it sits changes how you build it from day one. How does machine learning in an autonomous car actually work? Strip away the marketing and a perception stack is a chain of learned functions that turn raw sensor data into a structured model of the world the planner can act on. Cameras, radar, and lidar produce streams; convolutional and transformer-based networks convert those streams into detections, segmentations, tracks, and depth. Object detectors such as YOLO or DETR variants draw bounding boxes around pedestrians and vehicles. Segmentation networks label drivable surface pixel by pixel. Tracking associates detections across frames so the planner sees a moving pedestrian, not a flicker of unrelated boxes. Bird’s-eye-view models fuse several cameras into a single top-down representation. None of these run in isolation. The output of one feeds the next, and calibration — the geometric relationship between each sensor and the vehicle — sits underneath all of it. Get the camera extrinsic calibration wrong and a perfectly trained detector places a pedestrian half a lane from where they actually stand. The learned weights are only part of the system; the pipeline that carries data in and predictions out is the rest. This is the level of detail that matters for what comes next. You cannot reason about a failure mode until you know which stage in the chain produced the output and what the stages downstream do with it. What tasks make up the perception stack, and how do they combine? An autonomous perception stack is not one model. It is a set of specialised tasks whose outputs are composed, and the composition itself introduces failure paths that no single model’s accuracy captures. Perception task Typical model class What it produces What breaks downstream if it’s wrong Object detection YOLO, DETR, R-CNN family Bounding boxes + class labels Planner misses or invents an obstacle Semantic segmentation Encoder-decoder CNNs, SAM-derived Per-pixel scene labels Drivable-area estimate corrupted Multi-object tracking Detection + association Object identities over time Trajectory prediction destabilised Depth / BEV fusion Multi-camera transformers Metric 3D positions Distance-to-object error feeds braking Calibration Geometric + learned refinement Sensor-to-vehicle transforms Every downstream output shifts The point of laying it out this way is that a benchmark measures each row in isolation, on curated data, at a single point in time. The safety-relevant behaviour lives in the joins between rows and in what happens when the input distribution drifts away from the benchmark set. A detector that is 99% accurate on daylight urban scenes tells you nothing about its behaviour in low sun glare, and the perception failure modes that survive benchmarks are precisely the ones the composed system has to defend against. Where does the model sit relative to the safety argument? Here is the reframe that separates a road-ready program from a benchmark-chasing one. In a functional-safety context — ISO 26262 for the vehicle, ISO 21448 (SOTIF) for the performance limitations of the intended function — the ML model is a component whose contribution to a hazard has to be argued explicitly. The argument runs the other way from how ML teams usually think. It does not start from “the model is accurate, therefore the car is safe.” It starts from the hazard — say, failing to brake for a pedestrian — and works backwards to ask which perception failures could cause it and what the system does about each one. Accuracy is not enough on its own for a structural reason: an aggregate metric averages over exactly the rare, high-consequence cases that dominate a hazard analysis. A model can miss a class of pedestrian appearing in 0.1% of frames and still post a stellar score, but that 0.1% is where someone gets hurt. The reviewer is not asking whether the model is usually right. They are asking what happens the small fraction of the time it is wrong, and whether you have evidence for it. This is the same discipline that shows up across other high-consequence perception domains — it is a domain-specific case of the reliability pattern behind cross-vertical perception validation, where the model’s job is not to be perfect but to have its imperfection bounded and observable. Our computer vision engineering practice treats that observability as a first-class design input, not something bolted on after training converges. How do you connect a failure mode to a hazard and a safe state? The connective tissue between an ML output and a safety case is a chain that a reviewer can trace end to end. Every link has to be present; a broken link is why packs get sent back. The traceable chain, link by link: Failure mode — a concrete way the perception output can be wrong: missed detection, false positive, misclassification, position error, delayed track initialisation. Cause hypothesis — what conditions trigger it: occlusion, sensor degradation, out-of-distribution appearance, calibration drift. Detection mechanism — how the system knows the failure may be occurring: confidence collapse, cross-sensor disagreement, temporal inconsistency, a monitoring signal that fires before the output is trusted. Hazard link — which vehicle-level hazard this failure could contribute to, drawn from the hazard analysis. Safe state — the defined behaviour the system moves to: degraded operation, handover request, controlled stop. A team that stops at accuracy has link 1 and nothing else. They can tell you the model missed a pedestrian in testing; they cannot tell you how the running system would have noticed, or what it would have done. That is not a documentation gap — it is an architecture gap, because detection mechanisms and safe-state transitions have to be designed into the system, not written up afterward. This is where a benchmark-only model gets expensive. Re-instrumenting a trained model to expose degradation signals, cross-checks, and drift indicators after the fact means revisiting the pipeline, the data path, and often the model architecture itself. Building those surfaces in from the start is cheaper than the release window you lose retrofitting them. What does functional safety require of degradation and drift over time? A perception model in the lab is evaluated once. A perception model in a car is evaluated continuously, implicitly, by the world it drives through — and that world moves away from the training distribution the day the vehicle ships. New sign designs, seasonal lighting, sensor ageing, firmware changes upstream: all of it shifts the input distribution. Functional safety does not accept a model as a fixed artefact. It asks what the model’s behaviour is as it degrades, and whether that degradation is observable before it becomes dangerous. Concretely, a reviewer wants evidence that the deployed system monitors its own perception quality — confidence distributions, class-frequency drift, cross-sensor agreement — and that a drop in any of these routes to a defined response rather than a silent, gradual erosion of margin. This is monitoring in the safety sense, not the dashboard sense. It is the surface where an ML model’s outputs feed the failure-mode and degradation views of a validation pack, and it connects directly to how you version and monitor the model over its life. Getting machine learning model versioning for automotive perception right is part of the same discipline: you cannot argue about degradation over time if you cannot say which model version produced which output. The claim-class here matters. Whether drift monitoring catches a specific degradation depends on your sensor suite, operating domain, and thresholds — this is an observed pattern across perception programs, not a benchmarked guarantee. What is not context-dependent is the requirement that the behaviour be defined and observable. A reviewer will accept a bounded, monitored degradation profile. They will not accept “the model is accurate, so this won’t happen.” Why doesn’t a high benchmark score make a model road-ready? It is worth stating the misconception plainly, because it is the most common one and the most expensive. A benchmark score is a measurement of past performance on a frozen dataset under one interpretation of “correct.” A safety case is an argument about future behaviour across the full operating domain, including the conditions the benchmark never contained and the failures the aggregate metric averaged away. The two are not on the same axis. A score of 0.95 mAP is a fact about a dataset. A safety case is a structured claim, supported by evidence, that every credible way the function can fail has been identified, detected, and bounded. You can have the first without the second, and a benchmark-topping model with no failure-mode traceability is exactly the pack a reviewer returns unsigned. None of this means benchmarks are useless — they are how you know the component works at all. They are just the floor, not the ceiling, and mistaking one for the other is the mistake that resets a program’s timeline. FAQ How does machine learning autonomous car work in practice? A perception stack chains learned functions — detectors, segmenters, trackers, depth and BEV fusion — that turn raw camera, radar, and lidar data into a structured world model the planner acts on, with sensor calibration underneath all of it. In practice it means no single model is the system; outputs compose, and the safety-relevant behaviour lives in the joins and in what happens when inputs drift from the training distribution. What machine learning tasks make up an autonomous car perception stack, and how do they combine? Object detection, semantic segmentation, multi-object tracking, depth/BEV fusion, and calibration are distinct tasks with distinct model classes, and their outputs feed one another. A benchmark measures each in isolation on curated data, but the failure paths that matter emerge in the composition — where one stage’s error propagates into the next. Where does the ML model sit relative to the functional-safety argument, and why is accuracy not enough on its own? The model is a component whose contribution to a hazard must be argued explicitly under ISO 26262 and SOTIF; the argument runs backwards from the hazard, not forwards from accuracy. An aggregate metric averages over exactly the rare high-consequence cases that dominate a hazard analysis, so a high score can coexist with an unaddressed failure that hurts someone. How do you connect a perception model’s failure modes to a hazard and a defined safe state? You build a traceable chain: failure mode, cause hypothesis, detection mechanism, hazard link, and safe state. A team that stops at accuracy has the failure mode and nothing else — they cannot show how the running system detects it or what defined behaviour it moves to, which is an architecture gap rather than a documentation one. What does functional safety require of an ML model’s degradation and drift behaviour over time? It requires that degradation be defined and observable: the deployed system must monitor confidence distributions, class-frequency drift, and cross-sensor agreement, and route any drop to a defined response rather than a silent erosion of margin. Whether monitoring catches a specific degradation is context-dependent, but the requirement that the behaviour be bounded and observable is not. Why does a high benchmark score not make a machine learning model road-ready or a safety case? A benchmark score measures past performance on a frozen dataset under one definition of correct; a safety case is a structured argument about future behaviour across the full operating domain, including conditions the benchmark never held. The two sit on different axes — a benchmark-topping model with no failure-mode traceability is the pack a reviewer returns unsigned. What evidence should an autonomous-car ML perception pipeline produce for a reviewer to trace? The pipeline should produce, per credible failure mode, a documented detection mechanism, a link to a vehicle-level hazard, and a defined safe-state transition, plus continuous degradation and drift monitoring tied to specific model versions. That evidence is what lets a reviewer trace any output back to a bounded, observable behaviour rather than an aggregate score. The question that decides your timeline Before a single training run, the useful question is not “how accurate can we get” but “for each way this output can be wrong, can we name the hazard, the detection mechanism, and the safe state?” A team that can answer that builds observability and degradation behaviour into the architecture and reaches first-pass clearance with fewer safety-related clarification rounds. A team that cannot answer it builds a model, tops a benchmark, and then rebuilds the pipeline to expose everything the safety case needs — on the release window it can least afford to spend. The failure class is the benchmark-only perception model with no traceable path from failure to safe state; the surface that closes it is a validation pack whose failure-mode and degradation views the model was designed to feed from the start.
