Ask most people how a self-driving car works and you get a version of the same answer: a big neural network watches the road and learns to drive. It is a clean story. It is also not how the perception stack is built. Autonomous car machine learning is not one model that “learns to drive” from data — it is a chain of learned components, each with its own job, its own failure modes, and its own set of geometry assumptions baked in during training. When people conflate the chain with a single end-to-end brain, they miss the place where most real perception errors are born: the detection stage, and specifically the shape of the box it draws around an object. That box matters more than it looks. An axis-aligned bounding box drawn around a vehicle turning across an intersection, or around a cyclist crossing at an oblique angle, does not tightly enclose the object. It encloses the object plus a wedge of road surface, plus whatever neighbouring vehicles happen to fall inside the rectangle. Those extra pixels inflate false positives and confuse the non-maximum suppression step exactly where safety margins are tightest. The naive framing hides this. The expert framing puts it front and centre: on-road objects carry orientation, and how you represent that orientation is a deployment-condition decision, not a modelling default. How does autonomous car machine learning work in practice? Strip away the marketing and the perception side of an autonomous vehicle is a pipeline of distinct, learned stages. Cameras and other sensors feed a detection stage that finds objects in each frame. Detections feed a tracking stage that stitches per-frame observations into persistent object identities across time. Tracks and ego-motion feed localisation and prediction, which estimate where everything is now and where it will be a second or two from now. Each stage is trained, and each stage inherits the assumptions of the data it was trained on. This decomposition is why the “one model learns to drive” mental model is misleading. It is not that end-to-end learning is impossible — research systems do exist. It is that production autonomous perception is overwhelmingly built as a chain because each stage is independently testable, independently improvable, and independently blameable when something goes wrong. If tracking drops an object at an intersection, you want to know whether detection missed it, merged it with a neighbour, or handed the tracker a box so loose that association failed. That diagnostic clarity is a property of the pipeline, not of a monolith. We cover the full learned chain in more depth in Machine Learning in Self-Driving Cars: How the Learned Stack Actually Works, and the broader engineering context lives on our computer vision practice page. The claim worth extracting here: autonomous car perception is a chain of learned components — detection, tracking, localisation — and each carries geometry assumptions that break when road conditions diverge from the training distribution. That divergence, not raw model capacity, is where most deployment surprises come from. What role does object detection play in the perception stack? Detection is the stage that turns raw pixels into structured objects the rest of the stack can reason about. A detector outputs, for each object it finds, a class label, a confidence score, and a geometric region — the bounding box. Everything downstream treats those boxes as ground truth. The tracker associates them across frames; heading estimation reads their shape and position; trajectory prediction extrapolates from the track history. If the box is wrong, the error propagates. There is no downstream stage that repairs a systematically biased detection. Most off-the-shelf detectors — the YOLO family, RT-DETR, and the R-CNN lineage — output axis-aligned bounding boxes (AABB): rectangles whose sides are parallel to the image edges. That choice is inherited from the datasets these models are trained and benchmarked on. COCO, PASCAL VOC, and similar detection corpora annotate objects with axis-aligned rectangles because it is cheap to label and adequate for the “is there a dog in this photo” task the datasets were built for. The problem is that this labelling convention becomes a modelling assumption, and it silently rides along into the driving domain where it no longer fits. How detectors move oriented detections into persistent identities is the job of the tracker — see How a Tracking Model Works: From Oriented Detections to Persistent Object IDs. Why do axis-aligned bounding boxes fail on turning vehicles and cyclists? Consider a car that is aligned with the camera’s viewing direction on a straight highway. An axis-aligned rectangle wraps it tightly — the box-to-object area ratio is close to one, and almost every pixel inside the box belongs to the vehicle. AABB works fine here. Now rotate that same car forty-five degrees as it turns across an intersection. The tightest axis-aligned rectangle that still contains the whole vehicle is now much larger than the vehicle itself. The corners of the box fill with road surface, lane markings, and — in dense traffic — parts of adjacent cars. Three concrete failures follow from that inflated box: False positives and merged detections. When two angled vehicles sit close together in traffic, their axis-aligned boxes overlap heavily even when the vehicles themselves do not touch. Non-maximum suppression, which uses box overlap to decide whether two detections are the same object, either merges them into one or suppresses a real detection. Both errors corrupt the object count the tracker relies on. Degraded localisation. The whole point of detection is to say where an object is. A box that encloses a large wedge of road reports a centroid and extent that are both off. At equal recall, that is lower intersection-over-union — measurably worse localisation. Broken heading cues. An axis-aligned box, by construction, throws away orientation. It cannot tell the downstream stack whether a vehicle is pointed along the lane or turning across it. For prediction, that is the single most important cue you can lose. This is not an edge case. Urban intersections, roundabouts, and dense parking are exactly the scenes where box orientation varies most and where safety margins are smallest. The failure clusters where you can least afford it. The choice of box geometry is settled inside the model’s detection head, which is where an AABB-vs-oriented decision is actually implemented. When does oriented detection earn its cost? The alternative is oriented (rotated) detection, sometimes written OBB — a box with an extra angle parameter, so the rectangle can align with the object rather than the image axes. An oriented box around a turning vehicle hugs the vehicle: the area ratio drops back toward one, the captured road surface shrinks, and NMS overlap between neighbouring cars falls. The measurable payoff is tighter localisation (higher IoU at equal recall) and fewer merged or missed detections in dense traffic — which feeds directly into more accurate heading estimation and trajectory prediction. But oriented detection is not free, and it is not always worth it. The angle has to be annotated in the training data, which is more expensive per object than an axis-aligned rectangle. The model has to learn to regress that angle, which adds training complexity and a class of failure — angle ambiguity near symmetry — that AABB simply does not have. The honest engineering question is not “is oriented detection better” but “does the driving scene diverge enough from the AABB assumption to justify the annotation and training cost.” The following rubric is how we frame that call. It is an observed-pattern decision heuristic drawn from perception engagements, not a benchmarked threshold table — treat it as a starting point for your own scene analysis. Decision rubric: AABB vs oriented detection for on-road perception Driving scene Object orientation spread Packing density Recommended box geometry Highway, free-flowing, ego-aligned traffic Low — most vehicles align with lane and camera Low AABB is adequate Arterial roads, moderate turning Moderate Moderate AABB acceptable; measure IoU loss on angled objects first Urban intersections, roundabouts High — vehicles turn across the view constantly High Oriented detection likely earns its cost Dense parking, curbside, cyclists at oblique angles High Very high — boxes overlap heavily Oriented detection strongly favoured The pattern is consistent: oriented detection earns its cost when object orientation varies widely and objects pack densely — the exact conditions of urban intersections and dense parking — and not before. Upright, well-spaced highway traffic tolerates axis-aligned boxes. This is why the geometry choice belongs to the deployment condition, not to a default in the model card. If you are choosing a detector at the same time, our comparison of RT-DETR vs YOLO for production pipelines covers the architecture side of that decision. How does box orientation feed heading and trajectory prediction? An oriented box does not just localise better — it carries information an axis-aligned box discards. The angle of the box is a direct, per-frame estimate of the object’s heading in the image plane. When the tracker links oriented detections across frames, the heading estimates form a short history that the prediction stage can integrate: a vehicle whose box angle is rotating clockwise frame over frame is turning, and the predictor can anticipate the arc rather than extrapolating a straight line. With axis-aligned boxes, that heading has to be reconstructed from the trajectory of the box centroid — which is noisy, lags reality, and is exactly wrong at the moment a vehicle begins to turn (when the centroid has barely moved but the heading has changed). So the cost of AABB is not confined to detection. It compounds through the chain: worse localisation feeds noisier tracks, noisier tracks feed later and less confident predictions, and the errors are largest at intersections where prediction matters most. This is the kind of cross-stage failure we discuss in Challenges for Autonomous Vehicles: Where CV Perception Breaks in Production. Can off-the-shelf detectors handle dense urban traffic? Sometimes, and it is worth checking before assuming you need a custom model. An off-the-shelf AABB detector may be perfectly adequate for a highway-dominated operational design domain, and forcing oriented detection onto a scene that does not need it just spends annotation budget for no localisation gain. The right move is to measure the divergence first: how often do your real driving scenes contain angled, densely packed objects, and how much IoU do you lose on those objects with an axis-aligned model? That measurement — production geometry (object rotation, packing density, viewing angle) versus off-the-shelf training assumptions — is precisely what our A2 geometry-divergence assessment is built to answer. Applied to the driving scene, it turns the AABB-vs-oriented question from an argument into a scoped, evidence-backed decision. FAQ How does autonomous car machine learning work? In practice it is not a single model that learns to drive — it is a chain of learned components. A detection stage finds objects in each frame, a tracking stage stitches detections into persistent identities across time, and localisation and prediction estimate where objects are and where they are heading. Each stage is trained separately and inherits the assumptions of its training data, which is why the pipeline is diagnosable stage by stage. What role does object detection play in the autonomous driving perception stack? Detection turns raw pixels into structured objects — class, confidence, and a bounding box — that the rest of the stack treats as ground truth. Tracking, heading estimation, and trajectory prediction all consume those boxes directly, and no downstream stage repairs a systematically biased detection. That is why the shape of the box the detector draws is a stack-wide decision, not a local one. Why do axis-aligned bounding boxes fail on turning vehicles and cyclists at intersections? An axis-aligned box drawn around a vehicle turning at an angle, or a cyclist crossing obliquely, encloses the object plus a wedge of road surface and neighbouring objects. That inflates the box-to-object area ratio, driving false positives, causing non-maximum suppression to merge or suppress real detections in dense traffic, and lowering localisation IoU. It also discards heading, the cue prediction needs most. When does oriented (rotated) detection earn its cost in on-road perception? Oriented detection earns its cost when object orientation varies widely and objects pack densely — urban intersections, roundabouts, and dense parking. In those scenes an oriented box hugs the object, cutting captured road surface, false positives, and NMS merge errors. Upright, well-spaced highway traffic tolerates axis-aligned boxes, so the geometry choice should follow the deployment scene rather than a model default. How does box orientation feed downstream heading estimation and trajectory prediction? An oriented box’s angle is a direct per-frame heading estimate; linked across frames it gives the predictor a heading history that reveals turns early. Axis-aligned boxes force heading to be reconstructed from centroid motion, which lags and is wrong exactly when a vehicle begins to turn. So AABB errors compound through the chain and are largest at intersections, where prediction matters most. What extra annotation and training cost does oriented detection add for automotive datasets? Oriented detection requires annotating an angle for every object, which is more expensive per label than an axis-aligned rectangle, and the model must learn to regress that angle — adding training complexity and angle-ambiguity failure modes AABB does not have. That cost should be quantified against the localisation and prediction gains before committing. Can off-the-shelf detectors handle dense urban traffic, or does it require a custom model? It depends on the operational design domain. Off-the-shelf axis-aligned detectors can be adequate for highway-dominated deployments, so the right first step is measuring how often real scenes contain angled, densely packed objects and how much IoU is lost on them. Only that divergence measurement justifies the move to oriented detection or a custom model. The instructive question for any perception team is not “how good is our detector” but “how far does our real driving scene diverge from the geometry the detector was trained to expect” — because that gap, named and measured through a geometry-divergence assessment, is what decides whether an axis-aligned box is a shortcut or a liability.
