“Our self-driving car learns to drive.” It is the sentence that sells the demo and misleads the procurement team. A production autonomous vehicle is not one neural network that watched enough video and figured out driving. It is a pipeline of distinct learned subsystems — detection, segmentation, depth, tracking, prediction — that hand structured outputs to planners and controllers, many of which are still rule-based or classically tuned. The single biggest budgeting mistake AV programmes make follows directly from getting this wrong. If you treat the whole vehicle as one end-to-end model, you assume one training run and one validation exercise clears the safety case. If you treat it as a pipeline, you scope each learned component against its own data distribution, its own failure modes, and its own operational-design-domain gate. Those are two different companies with two different cost structures. The marketing view underfunds edge-case data collection, per-subsystem validation, and fallback logic. The engineering view puts money where model error actually threatens safety. Is a self-driving car one end-to-end neural network or a pipeline of trained models? There is genuine research into end-to-end driving — networks that map pixels more or less directly to steering and throttle. It produces striking demos. But shipped systems that carry a safety argument overwhelmingly run a modular pipeline, and the reason is not conservatism. It is that a monolithic learned box is nearly impossible to validate, debug, or bound. When the car does the wrong thing, you cannot ask an end-to-end model which subsystem failed, because there are no subsystems — there is a single opaque function. The modular pipeline exists so that failure has an address. Detection recall can be measured on its own. A depth estimator can be tested against LiDAR ground truth independent of what the planner does with it. Each learned block gets a contract: inputs, outputs, and an accuracy target the downstream stage was designed to tolerate. This is the same discipline we discuss in how ML powers the AV perception stack — the stack earns trust one measurable component at a time, not as an undifferentiated whole. So the honest answer to “is machine learning driving the car?” is: machine learning is doing most of the perceiving, and progressively more of the predicting, while much of the deciding remains explicit code you can read. Which parts are actually learned, and which stay classical? Here is the split as it typically appears in a production stack. The evidence class is observed across autonomous-perception engineering practice rather than a single published benchmark; individual programmes vary in where they draw the learned/classical line. Subsystem Learned or classical? What the model produces Where it breaks 2D object detection Learned (CNN / transformer detector) Boxes + class + confidence Rare classes, small/distant objects, occlusion Semantic segmentation Learned Per-pixel class (road, lane, curb) Adverse weather, unusual surfaces Depth / 3D detection Learned (often multi-camera BEV) Metric depth, 3D boxes Reflective/transparent surfaces, night Object tracking Hybrid — learned association + classical filter Persistent IDs over time ID switches under heavy occlusion Motion prediction Learned Future trajectories of agents Genuinely novel agent behaviour Sensor fusion Hybrid Unified world state Sensor disagreement, calibration drift Planning Often classical / optimisation Trajectory to follow Rare scenarios outside the rule set Control Classical (PID / MPC) Steering, throttle, brake Actuator model mismatch Two things fall out of this table. First, the perception front-end is dominated by learned components, and that is where labelling and validation budget concentrates. Second, planning and control frequently remain deterministic — a program engineer can reason about them, prove properties, and certify them in ways a neural network resists. The detection detail behind that perception front-end is where most of the training data spend lands. The tracking and fusion rows are worth pausing on: they are hybrids by design. Modern trackers learn the association step — which detection in this frame is the same object as which track from the last frame — but hand persistence off to a classical state estimator like a Kalman filter. How that seam works in practice is covered in sensor fusion engineering: combining detection, tracking, and multi-sensor inputs. What training data do the learned perception subsystems depend on? Each learned subsystem depends on a data distribution that matches the conditions it will meet on the road — and this is where the end-to-end fantasy quietly bankrupts programmes. A detector trained on daytime highway footage does not generalise to a snowy roundabout at dusk. It is not a smarter model that closes that gap; it is representative data covering that operating condition, correctly labelled, in sufficient volume. The practical consequence is that data collection is subsystem-specific. Depth estimation wants LiDAR-registered ground truth. Detection wants exhaustively boxed frames including the long tail of rare objects — the mattress on the motorway, the pedestrian in a costume, the overturned truck. Segmentation wants pixel-accurate masks across weather and lighting. These are different labelling operations with different costs, and treating them as one line item is where budgets go wrong. Public datasets and benchmarks give you a floor, not a ceiling. Object-detection quality is typically reported with mean average precision, and understanding what mAP actually measures is essential before you accept a vendor’s headline number — a strong mAP on a public benchmark says little about false-negative rates on the rare events that dominate your safety case. In our experience scoping AV perception work, the volume of edge-case data required to clear an operational-design-domain gate is routinely underestimated by an order of magnitude, because teams anchor on the benchmark distribution rather than the deployment distribution. That is an observed pattern across engagements, not a benchmarked figure. How do teams validate a learned subsystem before its ODD gate? Validation is per-subsystem before it is holistic, and the metric that matters is rarely the one on the leaderboard. For a safety-critical detector, the operationally relevant number is the false-negative rate on the rare, high-consequence classes — how often the system fails to see the thing that would hurt someone. Average precision across all classes can look excellent while the tail that matters is quietly weak. A workable validation rubric for a single learned perception component looks like this: Define the operational design domain explicitly. Which conditions — weather, lighting, geography, speed regime — is this subsystem certified for? A model is only as trustworthy as the ODD it was tested against. Set per-class recall/precision targets tied to consequence. A missed pedestrian and a missed traffic cone are not the same error. Weight targets by what the downstream planner needs and what the safety case demands. Measure false negatives on rare events specifically. Aggregate metrics hide the tail. Build a rare-event test set and report against it separately. Verify the downstream contract. The detector’s accuracy target exists because the tracker and planner were designed to tolerate a bounded error. Confirm the delivered accuracy stays inside that bound. Test the fallback path. What happens when the subsystem is uncertain? A validated subsystem includes a validated degradation behaviour, not just a validated success case. This is where a modular pipeline earns its keep. Each gate is a measurable, defensible checkpoint. An end-to-end model offers none of these hooks — you can measure only the aggregate driving behaviour, which is precisely the level at which rare failures are hardest to observe. Where does machine learning fail in self-driving, and what catches it? The failure modes are not exotic. They are the predictable edges of every learned subsystem: rare events the training data underrepresented, occlusion that breaks the assumption objects are visible, and adverse weather that shifts the input distribution away from what the model saw. A camera-based depth estimator degrades at night. A detector trained on common classes misses the uncommon one. A tracker loses identity when two pedestrians cross behind a bus. The engineering answer is not “a bigger model.” It is layered fallback. When a learned subsystem reports low confidence — and understanding what a confidence score actually means is a prerequisite for using it as a safety signal — the system should degrade deliberately: reduce speed, widen safety margins, hand back to a human where the design allows, or fall back to a more conservative classical policy. Sensor fusion helps here too, because a failure in one modality (camera in glare) can be covered by another (radar, LiDAR) when the fusion logic is designed to distrust a disagreeing sensor rather than average it away. The uncomfortable truth is that no amount of learning removes the long tail; it only shifts where the tail begins. That is why the pipeline architecture matters more than any single model’s benchmark score. The failure modes and the latency pressures they interact with are examined in more depth in where CV perception breaks in production. The whole discipline sits within the broader practice of building reliable vision systems, which is the subject of our computer vision work. FAQ How does machine learning in self-driving cars work in practice? In practice, machine learning handles most of the perception — turning camera, LiDAR, and radar input into detections, segmentation, depth, and predicted trajectories. Those learned outputs feed planners and controllers that decide and act. The models are trained on data representative of the conditions the vehicle will operate in, and each is validated against its own accuracy target rather than a single whole-vehicle score. Which parts of a self-driving car stack are actually learned by ML, and which remain classical or rule-based? The perception front-end — object detection, segmentation, depth and 3D detection, motion prediction — is dominated by learned models. Tracking and sensor fusion are typically hybrids that pair learned association with classical state estimation. Planning is often a classical optimisation, and control (steering, throttle, brake) is usually deterministic classical code like PID or MPC, because those stages are easier to certify and reason about. Is a self-driving car one end-to-end neural network, or a pipeline of separately trained models? Shipped systems that carry a safety argument overwhelmingly run a modular pipeline of separately trained models, not one end-to-end network. The modular design exists so that failure has an address: each subsystem has a measurable input/output contract and can be validated in isolation. End-to-end driving models produce compelling demos but are extremely hard to validate, debug, or bound. What training data and benchmarks do the learned perception subsystems depend on, and where do they fall short? Each subsystem needs data matching its deployment conditions: depth wants LiDAR-registered ground truth, detection wants exhaustively boxed frames including rare objects, segmentation wants pixel-accurate masks across weather and lighting. Public datasets and benchmarks like those scored with mean average precision provide a floor, not a ceiling — a strong benchmark score says little about false-negative rates on the rare events that dominate the safety case. How do teams validate a learned subsystem before it clears its operational-design-domain gate? They define the operational design domain explicitly, set per-class recall and precision targets weighted by consequence, measure false negatives on rare events separately from aggregate metrics, verify the delivered accuracy stays inside the bound the downstream stage was designed to tolerate, and test the fallback behaviour. A modular pipeline makes each of these a defensible checkpoint; an end-to-end model exposes only aggregate driving behaviour. Where does machine learning fail in self-driving scenarios — rare events, occlusion, adverse weather — and what fallback handles it? Learned subsystems fail at the predictable edges: rare events the training data underrepresented, occlusion that hides objects, and adverse weather that shifts the input distribution. The fallback is layered degradation — reducing speed, widening margins, handing back to a human, or reverting to a conservative classical policy when confidence is low — plus sensor fusion designed to distrust a disagreeing modality rather than average it away. If a vendor describes their “AI stack” as one model that learned to drive, the right next question is not how good the model is — it is which subsystem owns the false-negative rate on your worst-case scenario, and against which operational design domain it was gated.
