Ask most people what “self-driving machine learning” means and you get one picture: a neural network that takes in camera pixels and outputs a steering angle. It is a clean mental model. It is also the wrong one for anyone who has to validate a car against a safety case.

A production autonomous-vehicle (AV) stack is not one model. It is a pipeline of learned and non-learned components — detection, segmentation, depth, tracking, sensor fusion, prediction — each with its own training data, its own failure modes, and its own validation budget. Some of those stages are machine learning. Several of them are deliberately not. The interesting engineering question is never “how do we train the car to drive,” it is “which subsystem should be learned, which should be computed deterministically, and how do we prove either one meets the budget.”

This article is a practitioner’s map of where ML actually sits in the perception stack, and where it does not. It is not a claim that ML alone drives the car.

What does self-driving machine learning mean in practice?

The phrase covers a family of learned models that each solve one bounded perception problem. A camera-facing 2D detector proposes bounding boxes for vehicles, pedestrians, and cyclists. A semantic segmentation network labels drivable surface versus curb versus vegetation. A monocular or stereo depth model estimates distance. A 3D detector lifts objects into metric space. A tracker maintains persistent identities across frames. A prediction model forecasts where each agent will be in the next few seconds.

None of those is “the self-driving model.” They are stages, and the output of one becomes the input of the next. We treat this as the default framing in AV scoping work, because it is the only framing that lets you attribute a failure to a specific subsystem instead of shrugging at a monolithic black box. If you want the deeper walkthrough of how the learned stack chains together end to end, we cover that in Machine Learning in Self-Driving Cars: how the learned stack actually works.

The reason the pipeline framing matters is validation. A modular ML component can be benchmarked on a named dataset, regressed independently, and re-certified without re-testing the whole vehicle. A single end-to-end network cannot — every change touches every behaviour, and your safety argument starts over.

Which parts of an AV stack are learned, and which are classical?

This is the decision that separates a demo from a fielded system. The honest answer is that the modern AV stack is a hybrid, and the split is not arbitrary — it tracks whether a problem has a reliable analytic solution and whether the failure of a learned model is tolerable.

The table below is the axis we use when scoping each subsystem. It is an observed pattern across perception engagements, not a universal prescription — the right split depends on sensor suite, operational design domain, and how much validation budget the subsystem can consume.

ML-vs-classical decision axis by subsystem

Subsystem	Typical implementation	Why
Object detection (2D/3D)	Learned (CNN / transformer)	No analytic model of “what a pedestrian looks like”; data-driven is the only viable path
Semantic segmentation	Learned	Dense per-pixel labelling has no closed form; needs learned priors
Depth estimation	Learned (mono) or classical (stereo/LiDAR geometry)	Stereo and LiDAR give geometric depth deterministically; mono depth must be learned
Multi-object tracking	Hybrid — learned association, classical state estimation	Kalman-style motion models are deterministic; appearance matching benefits from learning
Sensor fusion	Mostly classical (probabilistic), increasingly learned	Fusing calibrated sensors is a geometry-and-statistics problem before it is a learning problem
Calibration / ego-motion	Classical	Extrinsics and odometry have well-defined analytic solutions
Motion prediction	Learned	Agent intent has no analytic model; strongly data-driven
Planning / control	Mostly classical, constrained	Must be auditable against safety rules; learned policies are hard to certify

The pattern is consistent: where a problem has a defensible analytic solution — geometry, state estimation, calibration — teams keep it classical because deterministic code is far cheaper to validate. Where perception must infer semantic meaning from raw sensor data, ML earns its place because nothing else works. The 2D convolutional networks that form the camera-detection backbone are a good example of the second case; ego-motion is a good example of the first.

Is end-to-end machine learning better than a modular ML pipeline?

This is where the naive framing does the most damage. End-to-end learning — pixels in, controls out — is genuinely attractive. It removes hand-engineered interfaces, it scales with data, and it produces spectacular demo videos. Under a safety case, though, it has a structural problem: you cannot attribute a failure.

When a modular stack brakes for a shadow it mistook for an obstacle, you can localise the error — the depth model, the detector, or the fusion step. You regress that one component against its dataset, fix it, and re-certify that component. When an end-to-end network does the same thing, there is no subsystem to blame and no bounded thing to re-test. Every retrain is a full-stack revalidation.

That does not make end-to-end learning wrong. It makes it a different validation posture. A useful middle ground that appears in practice is a modular stack with learned components at each stage and classical interfaces between them — the “learned islands, deterministic bridges” pattern. The car reaps the data-efficiency of ML where perception is hard, and keeps auditable seams where the safety argument lives.

The consequence for engineering leads is a budget question, not a philosophy question. A modular ML map lets you decide, per subsystem, whether a learned model, a classical algorithm, or a hybrid meets the safety budget — instead of over-investing in one monolithic model whose failures you can neither localise nor cheaply re-certify.

What datasets do self-driving ML models depend on — and where do they fall short?

Learned perception is only as trustworthy as the data behind it. Three named benchmarks anchor most AV perception work: KITTI, the long-standing detection and depth benchmark; nuScenes, which added full 360-degree multi-sensor scenes; and the Waymo Open Dataset, which brought large-scale, high-quality LiDAR and camera data with dense labels. These are benchmark-class references — reproducible, publicly reported, and the common yardstick for regressing a component.

Here is the caveat every AV team learns the hard way. These datasets are heavily weighted toward the operational design domains they were collected in — clear-weather, structured-road, daytime driving dominates the distribution. Rare events, heavy occlusion, and adverse weather are underrepresented by construction. A detector that scores well on nuScenes can still degrade sharply on a foggy on-ramp it never saw enough of in training. Benchmark performance is a floor, not a certificate.

This is exactly why independent regression matters. When your components are modular and each maps to a named benchmark, you can quantify the drop on an out-of-distribution slice and decide whether it clears the budget — and you can do it for one model without re-running the entire stack.

How does ML perception handle rare events, occlusion, and weather?

It handles them imperfectly, and the good stacks are designed around that fact rather than pretending it away. The failure mode is the long tail: the situations that are individually rare but collectively guaranteed to occur over millions of miles. A camera-only detector loses a partially occluded cyclist; a mono-depth model misreads distance in low-contrast fog; a tracker drops an identity through a tunnel.

Two structural moves help. The first is redundancy through sensing — LiDAR and radar degrade differently from cameras, so combining detection, tracking, and multi-sensor inputs through sensor fusion recovers cases where any single modality fails. The second is honest uncertainty — a model that reports low confidence on an occluded object lets the planner behave conservatively instead of confidently wrong. For a deeper look at where these breakdowns actually happen in fielded systems, we walk through them in Challenges for Autonomous Vehicles: where CV perception breaks in production.

The engineering point is that no single learned model solves the long tail. The stack solves it — through redundancy, uncertainty propagation, and classical fallbacks that hold when the learned components lose confidence.

How do you re-certify one ML component without re-testing the whole stack?

This is the operational payoff of the modular framing, and it is where the ROI is concrete. If each subsystem has a defined input contract, a defined output contract, and a named benchmark, then a model update is a bounded change. You regress the new detector against KITTI, nuScenes, and your own out-of-distribution slices; you confirm its output distribution still satisfies the contract the tracker and fusion stages expect; and you re-certify that component in isolation.

Compare that to the monolithic case, where any change to the learned mapping invalidates the entire safety argument. The cost difference is not marginal — it is the difference between shipping a model update in a bounded validation cycle and freezing the stack because you cannot afford full revalidation. This is the same discipline we apply when validating perception model changes with A/B test statistics: scope the change, bound the test, and keep the evidence attributable.

Modularity is not an aesthetic preference. It is what makes the validation budget finite. It is also, incidentally, where our computer vision engineering practice spends most of its scoping effort — deciding, subsystem by subsystem, what to learn and what to compute.

How does sensor fusion change the ML architecture versus a camera-only model?

A camera-only model has one input distribution and one failure surface. Add LiDAR and radar and the architecture question changes shape. You now choose where fusion happens: early fusion combines raw or low-level features before detection; late fusion combines per-sensor detections after each modality has run its own network; and mid-level or “deep” fusion learns a joint representation somewhere in between.

Each choice moves the ML boundary. Early fusion asks a network to learn cross-modal features and is data-hungry. Late fusion keeps per-sensor models modular and classical-friendly at the combination step, which is easier to validate. The bird’s-eye-view representations now common in 3D detection are one way teams reconcile camera and LiDAR into a shared frame that a downstream detector can reason over uniformly.

Fusion also changes the training-data problem: you now need time-synchronised, calibrated multi-sensor data, which is precisely why datasets like nuScenes and Waymo — collected with full sensor suites — became the reference standard over camera-only predecessors.

FAQ

How does self driving machine learning actually work?

In practice it means a pipeline of learned models, each solving one bounded perception problem — detection, segmentation, depth, tracking, prediction — chained so the output of one stage feeds the next. It is not a single network mapping pixels to steering. The pipeline framing is what lets a team attribute any failure to a specific subsystem rather than a monolithic black box.

Which parts of an AV stack are learned with machine learning versus computed with classical algorithms, and why?

Semantic perception — object detection, segmentation, monocular depth, motion prediction — is learned because those problems have no reliable analytic solution. Geometry-driven tasks — calibration, ego-motion, stereo/LiDAR depth, state estimation in tracking — stay classical because deterministic code is cheaper to validate. The split tracks whether a defensible analytic solution exists and whether a learned failure is tolerable.

Is end-to-end machine learning better than a modular ML pipeline for self-driving perception?

End-to-end learning is attractive on demos but hard to validate: a failure cannot be attributed to a subsystem, so every retrain becomes a full-stack revalidation. A modular ML pipeline lets you localise, regress, and re-certify one component at a time. Many production stacks use a middle ground — learned components with classical, auditable interfaces between them.

What training data and benchmarks (KITTI, Waymo, nuScenes) do self-driving ML models depend on, and where do they fall short?

KITTI, nuScenes, and the Waymo Open Dataset are the common benchmark-class references for detection, depth, and multi-sensor perception. They are weighted toward the operational design domains they were collected in, so rare events, heavy occlusion, and adverse weather are underrepresented. Benchmark performance is a floor, not a certificate — a model can score well and still degrade on out-of-distribution conditions.

How does ML-based perception handle rare events, occlusion, and adverse weather that break naive models?

No single learned model solves the long tail; the stack does, through redundancy and honest uncertainty. Multi-sensor fusion recovers cases where a single modality fails because LiDAR and radar degrade differently from cameras. Models that report low confidence on occluded objects let the planner behave conservatively instead of confidently wrong.

How do you validate and re-certify a self-driving ML component without re-testing the entire stack?

If each subsystem has defined input/output contracts and maps to a named benchmark, a model update is a bounded change: regress it against its datasets and out-of-distribution slices, confirm its output still satisfies the downstream contract, and re-certify that component in isolation. A monolithic model has no such seam — any change invalidates the whole safety argument. Modularity is what keeps the validation budget finite.

How does sensor fusion change the machine learning architecture compared with a camera-only model?

Adding LiDAR and radar forces a choice about where fusion happens: early fusion learns cross-modal features (data-hungry), late fusion keeps per-sensor models modular and easier to validate, and mid-level fusion learns a joint representation in between. Fusion also requires time-synchronised, calibrated multi-sensor training data, which is why full-sensor-suite datasets like nuScenes and Waymo became the reference standard.

The map is the deliverable, not any single model. When we scope an AV perception engagement, the first question is never which network to train — it is which of the ten subsystems should be learned, which should be computed, and which hybrid clears the safety budget at the lowest validation cost. Get that decision right per subsystem and the stack becomes something you can certify; get it wrong and you inherit a monolith you can neither localise nor afford to re-test.