Ask what “machine learning” means inside a self-driving car and the common answer is a single large model that reads camera pixels and outputs steering angle. That picture is wrong in a way that matters for safety, and it is wrong for a specific reason: perception in an autonomy stack is not one model. It is a staged pipeline — detection, segmentation, depth, tracking, sensor fusion, prediction — and each stage is its own machine-learning component with its own accuracy profile and its own slice of a fixed, non-negotiable latency budget. That last part is where the naive reading falls apart. The autonomy stack has to produce a perception result in time for the next control cycle. If the planner runs at 10Hz, the entire perception pipeline has roughly 100 milliseconds to turn raw sensor frames into a coherent scene the planner can act on. A model that is 2% more accurate on an offline benchmark but 30ms slower can be net-negative for safety, because it pushes perception past the deadline and the loop starts acting on stale information. The relevant unit of goodness is not accuracy in isolation. It is accuracy delivered inside the latency contract. How does self-driving car machine learning work in practice? Concretely, “machine learning in a self-driving car” is a set of specialised models chained together, each consuming the output of the stage before it. Camera frames (and, in most serious stacks, LiDAR and radar returns) enter the pipeline. A detector finds and classifies objects. A segmentation network labels drivable surface, lane markings, and free space at the pixel level. A depth or 3D-detection stage places those objects in metric space. A tracker links detections across frames into persistent object identities with velocity estimates. Fusion reconciles the camera, LiDAR, and radar views into one scene. A prediction model forecasts where dynamic agents will be over the next few seconds. Only then does the planner decide what the vehicle does. Each of those stages is learned, and each one is measured differently. Detection is scored with mean average precision (mAP). Segmentation uses intersection-over-union (IoU). Tracking uses metrics like MOTA. These are all real and useful — but on their own they describe a model’s behaviour in an offline evaluation harness, not its behaviour under a live deadline. We treat those accuracy metrics as meaningful only when reported alongside the latency each model consumes. A detector’s mAP is half a spec sheet; the other half is how many milliseconds it takes to produce that mAP on the target hardware. This is the same discipline that governs how perception detection actually works in autonomous-car machine learning: the detector is the front door of the pipeline, and its latency sets the pace for everything downstream. The distinct ML components and what each contributes It helps to be precise about what each stage does, because the failure modes are stage-specific and so are the latency costs. Stage What it produces Typical metric Where latency concentrates Detection 2D/3D boxes + class labels mAP Backbone forward pass, NMS Segmentation Per-pixel semantic labels IoU / mIoU High-resolution decoder Depth / 3D Metric distance, 3D position Depth error, 3D mAP Point-cloud ops, projection Tracking Persistent IDs + velocity MOTA / IDF1 Association, state update Sensor fusion Unified multi-sensor scene Consistency, alignment Synchronisation, transforms Prediction Future trajectories of agents ADE / FDE Multi-agent forward passes The detector is usually a convolutional or transformer-based network — a 2D convolutional neural network backbone is the workhorse of most AV camera stacks, with 3D detectors handling LiDAR and radar. Segmentation runs a dense decoder that is often the single most expensive stage at high input resolution. The tracker is comparatively cheap in raw compute but sensitive to the quality of what detection hands it — bad boxes produce ID switches that ripple into prediction. Fusion is where camera, LiDAR, and radar get reconciled, and it is a deceptively large latency line item because it involves synchronisation and coordinate transforms, not just a forward pass. None of these stages is optional in a production stack, and none of them can be tuned in isolation without watching what it does to the rest of the budget. How is the latency budget divided across stages? This is the heart of the matter, and it is where most benchmark-driven teams go wrong. If the planner needs a fresh scene every 100ms, that 100ms has to be split across every stage — plus the sensor capture, plus the decode, plus the fusion overhead, plus a margin for the tail. Sustained latency under realistic load is the operationally relevant measure here, not the median frame time a model advertises. A pipeline whose average is 80ms but whose 99th-percentile is 140ms is a pipeline that misses its deadline several times a second, and in perception the tail is where the crashes live. Here is an illustrative budget for a 10Hz planner — the numbers are for shape, not a benchmark, and real allocations depend entirely on hardware and sensor suite: Illustrative 100ms perception budget (10Hz planner) Sensor capture + decode: ~10ms Detection (camera + LiDAR): ~30ms Segmentation: ~20ms Depth / 3D placement: ~10ms Tracking + association: ~5ms Sensor fusion: ~10ms Prediction: ~10ms Tail margin / jitter reserve: ~5ms The value of budgeting per component is not the specific numbers — it is that a regression becomes localisable. When end-to-end latency creeps up, a team that has budgeted per stage can point at the stage that moved instead of chasing a whole-pipeline slowdown across six models at once. This is exactly why a model that improves detection mAP by 2% but adds 30ms is a bad trade in most stacks: that 30ms has to come from somewhere, and there is nowhere in the table above to hide it without violating the contract. The same problem shows up when perception breaks the latency contract in production autonomous vehicles — the failure is rarely a wrong prediction, it is a late one. This is an observed pattern from real-time perception work rather than a published benchmark, but the structural point holds regardless of the exact allocation: latency is a shared, finite resource, and every stage is a tenant. Where does sensor fusion fit, and how do extra sensors change the picture? Fusion is the stage that most often surprises teams reasoning about ML in the abstract. Adding a LiDAR to a camera-only stack does not just add a sensor — it adds a detection path, a synchronisation problem, a calibration dependency, and a fusion stage that has to reconcile two views of the world that were captured at slightly different times. Each of those consumes budget. More sensors generally buy robustness (radar sees through fog, LiDAR gives clean metric depth), but they also multiply the latency accounting and the failure surface. The engineering question is never “should we add a sensor” in the abstract; it is “does the accuracy or robustness this sensor buys justify the milliseconds and the fusion complexity it costs, given the deadline we have to hit.” That trade-off is the substance of sensor fusion engineering — combining detection, tracking, and multi-sensor inputs into one coherent scene under a deadline. Fusion done well is nearly invisible; fusion done naively becomes the stage that quietly eats the tail margin. End-to-end learned driving versus staged modular perception There is a genuine architectural debate here, and it deserves a fair statement. End-to-end learned driving — one large network mapping sensor input closer to control output — is a real research direction with real advantages: fewer hand-designed interfaces, the ability to learn representations the modular decomposition might miss, and less accumulated error from stage-to-stage hand-offs. The naive “one model steers the car” picture is not a fantasy; it is a simplified reading of this line of work. The reason most production stacks remain staged and modular is not conservatism. It is that a modular pipeline is debuggable, verifiable, and budgetable. When something goes wrong in a staged stack, you can ask which stage produced the bad output and localise the fault. When an end-to-end model behaves unexpectedly, the fault is distributed across a network with no interpretable stage boundaries — and for a safety system that has to defend its behaviour, that opacity is expensive. The same latency-contract discipline applies either way: an end-to-end model still has to produce output before the next control cycle, and it is often harder to reason about where its time goes. For teams weighing when a large learned component earns its place, how the learned stack actually works in self-driving cars is the deeper treatment of that boundary. Our own posture is pragmatic: the modular pipeline is the default because it is the one you can hold accountable to a deadline stage by stage. Learned end-to-end components earn their place where they demonstrably improve a stage without blowing its budget or its verifiability — not because they are architecturally fashionable. FAQ What’s worth understanding about self-driving car machine learning first? It means a staged pipeline of specialised learned models — detection, segmentation, depth, tracking, sensor fusion, and prediction — chained together, each consuming the previous stage’s output. It is not one model turning pixels into steering; it is a sequence of components that together produce a scene the planner can act on, all within a fixed per-cycle deadline. What are the distinct ML components in a self-driving perception stack, and what does each contribute? Detection finds and classifies objects; segmentation labels drivable surface and free space per pixel; depth/3D places objects in metric space; tracking links detections into persistent identities with velocity; fusion reconciles camera, LiDAR, and radar into one scene; prediction forecasts where agents will go. Each is measured with its own metric — mAP, IoU, MOTA — and each consumes its own slice of the latency budget. How is the perception latency budget divided across ML stages, and why can a more accurate model still be the wrong choice? The control-loop period (e.g. ~100ms for a 10Hz planner) is split across capture, every ML stage, fusion, and a tail-margin reserve. A model that is 2% more accurate but 30ms slower can be net-negative because that 30ms has to come from somewhere, and pushing perception past the deadline makes the loop act on stale data. Accuracy only counts when it fits inside the contract. Where does sensor fusion fit, and how does adding sensors change the ML and latency picture? Fusion reconciles camera, LiDAR, and radar into a single scene, adding synchronisation, calibration, and coordinate-transform costs on top of each sensor’s detection path. More sensors generally buy robustness but multiply both the latency accounting and the failure surface, so the decision is always whether the robustness justifies the milliseconds and complexity given the deadline. How do you measure whether a self-driving perception model is meeting its deadline versus blowing it? Report each stage’s accuracy metric alongside the latency it consumes on the target hardware, and track sustained and tail latency under realistic load rather than median frame time. A pipeline whose 99th-percentile latency exceeds the control-loop period is missing its deadline repeatedly even if its average looks fine — in perception, the tail is where the danger concentrates. Where do end-to-end learned driving models fit versus staged, modular ML perception stacks? End-to-end models map sensor input closer to control output with fewer hand-designed interfaces and less stage-to-stage error, and they are a real research direction. Most production stacks stay modular because a staged pipeline is debuggable, verifiable, and budgetable stage by stage — which matters for a safety system that must account for its behaviour. Both still have to meet the latency contract. What does TechnoLynx assess first when scoping an automotive perception ML subsystem? We start with the latency contract: the control-loop period, the current per-stage budget, and where the tail latency actually lands under realistic load. Only then do we look at per-stage accuracy, because an accuracy improvement that violates the deadline is a regression, not a gain. Where this leaves a scoping conversation The practical takeaway is that “self-driving car machine learning” is not a model you tune for a leaderboard. It is a pipeline you budget for a deadline. When we scope an automotive perception subsystem — the same latency-budgeted approach we bring to computer vision engagements more broadly — the first artifact is not a model comparison. It is the per-stage latency budget against the control-cycle deadline, because until you know how the milliseconds are spent, every accuracy number is only half a claim. The open question for any specific programme is where the elasticity lives: which stage can give back milliseconds without losing the accuracy the planner actually depends on. That is a per-stack answer, and it is the one worth measuring before anyone argues about mAP.
