Ask most people how a self-driving car uses machine learning and you will hear some version of the same picture: one big neural network takes camera pixels in one end and pushes steering, throttle, and brake commands out the other. It is a clean mental model. It is also the wrong one for reasoning about how these systems are actually built, tested, and made safe. In practice, machine learning in an autonomous vehicle is not a single model. It is a staged pipeline — perception, sensor fusion, prediction, planning, and control — where each stage has its own model families, its own inputs, and its own failure modes. The reason the pixels-to-steering picture matters is not that it is imprecise; it is that it hides exactly the thing engineers care about most. When something goes wrong, you need to know which stage went wrong, and a monolithic network makes that question nearly unanswerable. How does machine learning in self-driving cars actually work? The core idea is decomposition. Rather than learning the entire driving task in one function, the system splits the problem into a sequence of narrower tasks, each of which can be trained, evaluated, and debugged against its own ground truth. Perception answers “what is around me?” — detecting and classifying vehicles, pedestrians, cyclists, lane markings, traffic signs, and the drivable surface. Sensor fusion reconciles those detections across cameras, radar, and lidar into one coherent world state. Prediction answers “what will those agents do next?” — where the pedestrian at the curb is likely to go, whether the car in the next lane is about to merge. Planning turns that predicted world into a trajectory the vehicle should follow. Control converts the trajectory into actuator commands. Machine learning does not sit uniformly across all five. Perception and prediction are heavily learned. Planning is a mix of learned and rule-based components in most production stacks. Control is often classical (PID or model-predictive controllers) because it is safety-critical, well-understood, and easier to verify than a learned policy. Knowing where ML sits — and where it deliberately does not — is the prerequisite for reasoning about reliability at all. The main stages of the pipeline Here is the pipeline laid out with the dominant model family and the failure mode each stage is meant to contain. The point of the table is that each row is testable in isolation. Stage Question answered Typical model family Characteristic failure mode Perception What objects are around me? CNN / transformer detectors, segmentation networks False negatives on safety-critical objects; misclassification Sensor fusion What is the single true world state? Kalman/particle filters, learned fusion nets Association errors — one object seen as two, or two merged into one Prediction What will other agents do next? Recurrent / graph / transformer motion models Confident wrong forecasts of pedestrian or vehicle intent Planning What trajectory should I take? Optimization + learned cost models Trajectories that are safe but socially wrong, or overly conservative Control What actuator commands realise it? PID / model-predictive control (mostly classical) Tracking error, oscillation, latency-induced instability Because each stage has its own ground truth — labelled detections for perception, recorded future trajectories for prediction — you can measure error at the stage level. That is what makes the decomposed design certifiable in a way a single end-to-end model is not. This is the same layered view we develop in more depth in our walkthrough of how the self-driving perception stack works, which focuses on the perception and fusion stages specifically rather than the full decision path. What role does computer vision and perception play? Perception is where most of the learned heavy lifting happens, and it is where computer vision earns its central role. Cameras supply dense semantic information — colour, texture, text on signs, brake lights — that radar and lidar cannot. Convolutional and transformer-based detectors identify objects and their bounding boxes; segmentation networks label every pixel as road, sidewalk, vehicle, or vegetation. In our work on computer vision systems, the recurring lesson is that perception quality is bounded far more by data coverage than by model architecture. The metric that matters most here is not overall accuracy but the false-negative detection rate on safety-critical objects. A model that is 99% accurate overall can still be unsafe if the 1% it misses is disproportionately pedestrians at dusk or partially occluded cyclists. This is an observed pattern across automotive perception work rather than a single benchmarked figure: the long tail of rare conditions dominates the safety case, and it is precisely the part that aggregate accuracy numbers obscure. It is also why the data pipeline matters as much as the model — the way training data is assembled and labelled determines which edge cases the model ever learns to see, a theme we cover in building a car parts and automotive perception dataset. Perception rarely relies on a single sensor. A common production pattern fuses camera, radar, and lidar so that the failure modes are complementary: cameras degrade in low light and glare, radar is robust to weather but coarse in resolution, lidar gives precise geometry but struggles with reflective or absorptive surfaces. Sensor fusion is the stage that turns three imperfect views into one usable world state, and errors introduced here — associating one object as two, or merging two as one — propagate silently into every downstream decision. How do prediction and planning turn perception into driving decisions? Perception tells you what exists right now. Driving requires knowing what happens next. Prediction models take the fused world state and forecast the likely trajectories of every dynamic agent in the scene, usually as a distribution over possible futures rather than a single path. Graph neural networks and transformer-based motion models are common here because interactions between agents — a car yielding, a pedestrian hesitating — are relational, not independent. Planning consumes those predictions and produces a trajectory. The subtle failure mode is not crashing; it is being wrong in a socially legible way. A planner that is too conservative brakes for phantom risks and gets rear-ended or blocks traffic; one that is too aggressive erodes the safety margin the whole pipeline exists to protect. Most production planners therefore combine a learned cost model with explicit optimization and hard rule constraints, so that a learned component can shape behaviour without being able to violate a safety invariant. All of this has to happen fast. The perception-to-control loop typically needs to close within tens of milliseconds to be usable at road speed — a car at 100 km/h covers nearly three metres in 100 ms. This is a market-direction figure that reflects how the industry frames latency budgets, not an operational benchmark from a specific deployment; the exact budget depends on speed, sensor rate, and compute. But the direction is firm: latency is a first-class design constraint, not an afterthought, and it is one reason control stays classical and fast rather than becoming another deep network. Why a decomposed pipeline instead of one end-to-end model? The end-to-end appeal is real. A single network trained on enough driving data can, in principle, learn behaviours no one explicitly programmed, and it removes the hand-built interfaces between stages. Research systems have shown that end-to-end learning can produce impressively smooth driving in constrained conditions. The problem is validation. When a decomposed pipeline fails, you can localise the fault: the detector missed the object, or the predictor forecast the wrong intent, or the planner chose a poor trajectory. Each hypothesis is testable against stage-level ground truth. When a monolithic model fails, you have one number — it drove badly — and no principled way to attribute the failure or prove the fix did not break something else. Teams that skip decomposition tend to discover this gap the hard way, during edge-case failures they cannot localise and therefore cannot systematically eliminate. There is a measurable payoff to decomposition beyond safety: shorter validation cycles. When a regression can be traced to a specific stage, you re-validate that stage rather than the whole system, and the debugging surface shrinks from “the car” to “the pedestrian detector under low-light occlusion.” That localisability is, in our experience across perception-heavy projects, the single biggest determinant of how fast a team can iterate safely — an observed pattern, not a benchmarked speedup. A quick way to decide where ML belongs in your stack Use this rubric when deciding whether a given driving sub-task should be learned or engineered: Is there abundant labelled ground truth for the sub-task in isolation? If yes, ML is viable and testable. If no, a learned component becomes a validation liability. Is the sub-task safety-critical and time-critical? The more it is both, the stronger the case for a classical, verifiable component (as with control). Can the sub-task’s output be checked against a hard invariant? If the output must never violate a constraint, keep a rule-based guardrail even around a learned core (as with planning). Can you attribute a failure to this stage without ambiguity? If not, you have merged too much into one model and lost localisability. What are the key challenges in validating machine learning for self-driving cars? Validation is where the whole design philosophy pays off or falls apart. Three challenges recur. First, the long tail: the failures that matter are rare by definition, so aggregate metrics understate risk and you have to actively mine and synthesise edge cases. Second, distribution shift: a perception model tuned for one geography, weather regime, or sensor configuration degrades when any of those change, and the degradation is often silent. Third, interface contracts: each stage assumes the previous stage’s output has certain properties, and a subtle change in the detector’s calibration can break the predictor downstream without either stage looking wrong in isolation. The decomposed pipeline does not eliminate these challenges, but it makes them addressable. You can build a stage-level test suite, hold each interface to a contract, and re-validate incrementally. That is the difference between a system you can reason about and one you can only hope about. FAQ How does machine learning self driving cars work? In practice a self-driving car does not use one neural network mapping pixels to steering. It uses a staged pipeline — perception, sensor fusion, prediction, planning, and control — where machine learning is concentrated in perception and prediction, planning mixes learned and rule-based components, and control is usually classical. Each stage is trained and tested against its own ground truth, which is what makes the system debuggable and certifiable. What are the main stages of a self-driving car’s machine learning pipeline? The five canonical stages are perception (what is around me), sensor fusion (what is the single true world state), prediction (what will other agents do next), planning (what trajectory to take), and control (what actuator commands realise it). Each stage answers a distinct question, uses distinct model families, and has its own characteristic failure mode, so it can be measured in isolation. What role does computer vision and perception play in autonomous driving? Computer vision is where most of the learned work happens, because cameras supply dense semantic cues — signs, brake lights, lane markings — that radar and lidar cannot. The metric that matters is the false-negative rate on safety-critical objects rather than overall accuracy, since the rare misses dominate the safety case. Perception quality is bounded more by data coverage than by model architecture. How do prediction and planning models turn perception into driving decisions? Prediction takes the fused world state and forecasts likely future trajectories of other agents, usually as a distribution rather than a single path, often using graph or transformer motion models. Planning consumes those forecasts and produces a trajectory, typically combining a learned cost model with explicit optimization and hard rule constraints so a learned component can shape behaviour without violating a safety invariant. Why do self-driving systems use a decomposed pipeline instead of one end-to-end model? Because validation demands it. When a decomposed pipeline fails you can localise the fault to a specific stage and test the fix against stage-level ground truth; when a monolithic model fails you have one undifferentiated signal and no principled way to attribute or verify. Decomposition also shortens validation cycles by shrinking the debugging surface to a single stage. What are the key challenges in validating machine learning for self-driving cars? The recurring challenges are the long tail of rare but safety-critical failures that aggregate metrics understate, distribution shift when geography, weather, or sensors change, and interface contracts between stages that can break silently. A decomposed pipeline makes these addressable through stage-level test suites, interface contracts, and incremental re-validation. What sensors and data feed the machine learning models in autonomous vehicles? Cameras, radar, and lidar are the primary sensors, chosen because their failure modes are complementary: cameras degrade in low light and glare, radar is weather-robust but coarse, lidar gives precise geometry but struggles with reflective surfaces. Sensor fusion reconciles these three imperfect views into one world state, and the training data assembled and labelled for perception determines which edge cases the models ever learn to handle. The lasting question is not whether to use machine learning in autonomous vehicles — it is where to draw the boundaries between learned and engineered components. Draw them well and every failure has an address; draw them poorly and you inherit a system that drives badly for reasons you cannot name.
