A self-driving stack that scores well on an aggregate benchmark can still put the car in a situation no one can explain. That gap is the whole problem with treating autonomous driving machine learning as a single model that either drives well or does not. When a review board asks why the vehicle failed to yield at a partially occluded intersection, “the end-to-end accuracy was 94%” is not an answer. It is an admission that no one knows. The mistake is understandable. Modern perception stacks are trained end to end often enough that people start to think of them as one function: sensors in, trajectory out, one number to grade. But the moment you need to explain a failure — to a safety reviewer, to a regulator, to your own engineers on the next sprint — that mental model collapses. You cannot fix what you cannot locate, and you cannot locate a failure inside a black box. The useful way to think about autonomous driving machine learning is as a pipeline of distinct perception stages, each of which learns a different thing, fails in a different way, and produces a different kind of evidence. Sensing, detection, tracking, fusion, prediction. Get comfortable with the boundaries between those stages and most of the hard debugging questions become answerable. What does “autonomous driving machine learning” actually mean in practice? In practice it means several learned models, plus a lot of non-learned geometry and logic, chained into a data path that turns raw sensor frames into a decision the planner can act on. The learning is concentrated in perception — the part that has to turn photons and LiDAR returns into a symbolic understanding of the scene. Downstream planning and control are frequently rule-based or optimization-based, not learned. So when someone says “the ML drives the car,” what they usually mean is “the ML perceives the world and the planner drives on top of that perception.” That distinction matters because most production driving failures we see in perception-heavy stacks are perception failures wearing a planning costume — the planner did something reasonable given a wrong belief about the scene. We unpack how those learned perception models behave under production conditions in our note on how autonomous driving deep learning works in production; this article stays one level up, at the pipeline structure. The five stages, and what each one learns The pipeline is not a single arbitrary decomposition. Each stage exists because it solves a problem the previous stage cannot, and each learns a genuinely different representation. Stage Input What it learns / does Typical output Sensing Raw camera, LiDAR, radar Calibrated, time-synchronized frames; often learned image enhancement or LiDAR densification Aligned sensor streams in a common clock and coordinate frame Detection Single-frame sensor data Where objects are now — classes, boxes, masks (e.g. via YOLO, DETR, or a BEV detector) Per-frame detections with class + confidence Tracking Sequence of detections Object identity across time; association and motion state Stable track IDs with velocity estimates Fusion Detections/tracks from multiple sensors A single consistent scene from disagreeing sensors One unified object list in world coordinates Prediction Tracked objects + map context Where each agent will go next Probable future trajectories per agent Detection is a per-frame question: what is in this image? Tracking is a temporal question: is the pedestrian in frame 200 the same pedestrian from frame 199? Fusion is a consistency question: the camera says there is a cyclist, the radar says there is nothing there — who is right? Prediction is a counterfactual question: given how this car has moved, where will it be in two seconds? These are different learning problems, which is exactly why collapsing them into one accuracy number loses information you will later need. Sensing deserves its own emphasis because it is the stage teams most often treat as plumbing. It is not. A calibration drift in the sensing stage silently corrupts everything downstream, and the failure shows up as a tracking or fusion symptom — which is why we treat calibration as a first-class perception concern in our write-up on how a tracking model breaks when calibration drifts. Where does each stage fail, and how does it show up? The value of the stage decomposition is that failure modes are stage-specific and have recognizable production signatures. This is an observed pattern across the perception-validation work we do — not a benchmarked failure rate — but the mapping is consistent enough to debug against. Sensing failures — miscalibration, desynchronized clocks, sensor degradation. Signature: objects appear at systematically offset positions, or two sensors disagree in a way that gets worse with range. Easy to misread as a fusion problem. Detection failures — missed small or occluded objects, class confusion, low confidence in rare classes. Signature: the object is simply absent from the per-frame output, or flickers in and out. Tracking failures — ID switches, track fragmentation, dropped tracks during occlusion. Signature: a pedestrian who “teleports” or gets a new identity mid-crossing, causing the predictor to reset its motion history. Fusion failures — a confident-but-wrong sensor overrides a correct one; ghost objects from radar multipath. Signature: an object exists in one sensor’s view but not in the fused scene, or vice versa. Prediction failures — plausible-but-wrong intent, especially at intersections and merges. Signature: the planner reacts late because the predicted trajectory did not include the maneuver the agent actually made. Notice how many of these masquerade as each other. A missed pedestrian at a crossing could be a detection miss, a tracking drop during occlusion, or a fusion override. If you only have one aggregate number, you cannot tell which — and you will burn re-validation runs chasing the wrong stage. The failure modes that survive clean benchmark scores are a category worth studying on their own, which we do in the perception failure modes that survive benchmarks. How do you trace a driving failure back to a stage? The trace works backwards through the pipeline, checking each stage’s output against ground truth for the specific incident. This is where the staged model earns its keep operationally. Start at the planner’s belief. What did the planner think the scene was at decision time? Reconstruct the object list it acted on. Check fusion output against that belief. Did the fused scene match reality? If the fused object list already omitted the pedestrian, the fault is at or before fusion. Split fusion into its inputs. Look at each sensor’s tracks independently. If camera tracking had the pedestrian but the fused output did not, fusion overrode a correct input — a fusion-logic fault. If a sensor’s track is wrong, drop to tracking. Did detection produce the object per-frame but tracking failed to maintain the ID? That is a tracking fault, often occlusion-driven. If detection never produced it, drop to detection. And if detection’s input frame was already geometrically wrong, drop to sensing/calibration. The point of this backward walk is a single number at the end: mean time to isolate a perception failure to its stage. Teams that reason stage by stage shorten that loop dramatically, which is what actually reduces debugging cycles and the re-validation runs triggered by mis-attributed failures. When a failure is attributed to the wrong stage, you re-test the wrong component and the real bug survives to the next release — the most expensive kind of miss. Choosing which metric to check at each stage is its own discipline; per-frame detection precision, tracking ID-switch rate, and prediction displacement error each prove something different. We work through what each metric does and does not establish in what machine learning performance metrics actually prove. Why does a single end-to-end accuracy number obscure what matters? Because release decisions are made about failure modes, not about averages. An aggregate accuracy of 94% tells you nothing about the distribution of the 6% — and in driving, the tail is the whole point. A stack that misses 6% of objects uniformly is very different from one that misses 6% concentrated in occluded pedestrians at dusk, even though both score identically. An end-to-end number also cannot survive a review. A reviewer’s job is to ask “what happens when the front camera is sun-blinded and only radar is available?” — a fusion-and-degradation question that an aggregate figure cannot address. The number is not wrong; it is simply not decision-grade for the questions that gate a release. This is the core of our thesis on autonomous perception: the operationally relevant unit of evidence is the per-stage, per-condition behaviour, not the pooled score. The broader engineering practice behind this sits under our computer vision work. How does this help you build the evidence a reviewer expects? Once the pipeline is decomposed, the evidence package almost designs itself. Each stage’s failure modes become the questions each evidence surface has to answer, and each surface is anchored to the stage and test that produced it. Instead of one accuracy claim, you present detection recall on the safety-critical classes, tracking continuity through occlusion scenarios, fusion behaviour under single-sensor degradation, and prediction accuracy at the intersection types your operational design domain includes. That structure is exactly what a validation pack is built to hold, and understanding the stages is the prerequisite for it — you cannot organize evidence around stages you have not separated in your own head. The per-stage failure taxonomy here is what our cross-vertical perception validation harness is designed to capture as structured evidence; the harness gives each stage a place to record what it proved and under which conditions. FAQ What’s worth understanding about autonomous driving machine learning first? In practice it is a chain of learned perception models feeding a mostly non-learned planner. The learning is concentrated in perception — turning raw sensor data into a symbolic scene — while planning and control are often rule-based on top of that scene. So “the ML drives the car” really means “the ML perceives, the planner drives on that perception,” which is why most driving failures in these stacks are perception failures the planner acted on faithfully. What are the stages of a perception pipeline, and what does each learn? The stages are sensing, detection, tracking, fusion, and prediction. Sensing produces calibrated, time-synchronized frames; detection finds objects in a single frame; tracking maintains object identity across time; fusion reconciles disagreeing sensors into one scene; prediction estimates where each agent will go next. Each solves a problem the previous stage cannot, so each learns a genuinely different representation. Where does each pipeline stage typically fail, and how do those failures show up? Sensing fails through calibration drift and clock desync, showing as range-dependent position offsets. Detection fails on small, occluded, or rare-class objects that go missing per-frame. Tracking fails as ID switches and dropped tracks during occlusion; fusion fails when a confident-but-wrong sensor overrides a correct one; prediction fails as plausible-but-wrong intent at intersections. Many of these masquerade as each other, which is why stage-specific signatures matter. How do you trace an observed driving failure back to the specific pipeline stage that caused it? Walk backwards from the planner’s belief at decision time. Check whether fusion already had the scene wrong; if so, split fusion into its per-sensor inputs. If a sensor track is wrong, drop to tracking; if detection never produced the object, drop to detection; if detection’s input frame was geometrically wrong, drop to sensing and calibration. The backward walk isolates the fault to one stage instead of re-testing the wrong component. Why does treating the stack as a single end-to-end accuracy number obscure the failures that matter for release? Because release decisions are about failure modes, not averages, and in driving the tail is the whole point. Two stacks with identical aggregate accuracy can fail completely differently — one uniformly, one concentrated in occluded pedestrians at dusk. An aggregate number also cannot answer a reviewer’s degradation questions, so it is not decision-grade for the questions that gate a release. How does understanding the stages help you produce the evidence surfaces a reviewer expects in a validation pack? Once the pipeline is decomposed, each stage’s failure modes become the questions each evidence surface must answer, and each surface is anchored to the stage and test that produced it. Instead of one accuracy claim you present detection recall on safety-critical classes, tracking continuity through occlusion, fusion under single-sensor degradation, and prediction accuracy at your ODD’s intersection types. You cannot organize evidence around stages you have not first separated. Start with the stage you would least like to explain to a reviewer. If you cannot say, cleanly, what your fusion stage does when the front camera is sun-blinded and only radar remains, that is not a documentation gap — it is a place where you do not yet know how your own pipeline behaves, and no aggregate score will tell you.
