Ask most teams how machine learning drives a self-driving car and you get a clean picture: camera frames go in, detections come out, the car reacts. That picture is not wrong so much as incomplete — and the missing part is where the failures live. The learned model is one link in a longer chain. The chain assumes the camera’s pose is known and correct, converts 2D pixels into 3D positions the planner can trust, and hands that geometry downstream. When something in the driving stack goes wrong, the instinct is to blame the model. Often the model did exactly what it was trained to do — the geometry underneath it moved. How does machine learning in self-driving cars work, in practice? A production perception stack is a pipeline, not a single network. Camera frames pass through a learned detector — a YOLO-family or DETR-family architecture, typically — that outputs bounding boxes, segmentation masks, or keypoints in image space. Those outputs are then projected into a shared 3D frame using the camera’s calibrated parameters, fused with other sensors, and passed to tracking and planning. The machine learning part is real and hard: training a detector to survive rain, glare, low light, and the long tail of objects a public road throws at it is genuine engineering. But the model’s job stops at the image plane. It tells you this pixel region is a pedestrian with some confidence. It does not tell you where that pedestrian is in the world — that step is geometry, and geometry depends on calibration the model never sees. This is the divergence that trips up evaluation. A better model gives you better detections in image space. It does not give you a better sense of where those detections are unless the projection from 2D to 3D is correct. We see teams pour effort into raising detection mAP while the operationally relevant error — 3D position error under a shifted camera pose — sits untouched. The rigorous account of that data path is worth reading alongside this one: how the perception pipeline actually moves data from sensor to planner. Where does the learned model end and geometry begin? The boundary is precise, and naming it correctly is half the battle. The learned model owns everything up to and including image-space outputs: what is in the frame, where in the frame, how confident. Its errors are model errors — missed detections, false positives, poor localization in pixels. These respond to more data, better augmentation, or a stronger architecture. Geometry owns everything after: turning a pixel box into a metric position. This is where the camera’s intrinsic parameters (focal length, principal point, lens distortion) and extrinsic parameters (the camera’s pose relative to the vehicle frame) do their work. A pixel at coordinate (u, v) only becomes a point in the world when you apply those parameters. Get the pose wrong and every projection is wrong — regardless of how good the detection was. The model has no channel to signal a geometry problem. It reports high-confidence detections whether or not the camera has shifted two degrees on its mount. That silence is the systems-integration gap: a perfectly trained perception model still mis-projects 3D-to-2D geometry the moment extrinsic calibration drifts, because the drift is invisible to the model. It is not a model-accuracy failure. It is a pipeline failure that wears a model’s face. For the mechanics of the two calibration classes, we cover what camera extrinsic calibration means and why it belongs in your safety evidence and what intrinsic parameters are and why their traceability matters separately, because each drifts for different reasons and shows up differently in the data. Why does a well-trained model still fail when calibration drifts? Consider a camera mounted on a windshield. Over months of vibration, thermal cycling, and the occasional door slam, its physical pose shifts by a small angle. The intrinsics are unchanged — the lens is the same. But the extrinsic pose, the rotation and translation the pipeline uses to place detections in the world, is now stale. The detector keeps working. It draws a clean box around a pedestrian at the correct pixel location. The projection step then places that pedestrian at the wrong distance and lateral offset, because it is using the old pose. A pedestrian standing at the roadside gets projected slightly into the lane, or slightly out of it. Downstream, tracking and planning inherit the error with no idea it originated in geometry. Two structural facts make this dangerous: The error scales with distance. A small angular pose error produces a small lateral error at 5 metres and a large one at 50 metres — exactly the range where you need position estimates you can trust. The failure is silent. Detection confidence stays high. Standard model metrics stay green. Nothing in the model’s output distinguishes a correct projection from a corrupted one. This is why “the model regressed” is so often the wrong diagnosis. The model is stable; the pipeline’s geometric assumption broke underneath it. If you want the failure-mode taxonomy in more depth, the perception failure modes that survive benchmarks walks through the ones that never show up in a held-out test set. What do intrinsic and extrinsic parameters actually control? It helps to separate the two, because they fail differently and get monitored differently. Parameter class What it encodes How it fails Typical detection method Intrinsic (focal length, principal point, distortion) The camera’s internal optics — how the lens maps rays to pixels Rare after manufacturing; changes with lens damage or hardware swap Factory calibration + hardware provenance checks Extrinsic (rotation, translation vs. vehicle frame) The camera’s pose on the vehicle Common; drifts with vibration, thermal cycling, minor collisions Reprojection residual monitoring over time (Failure-frequency framing above is an observed pattern across automotive perception engagements, not a published benchmark.) The practical takeaway: intrinsics are mostly a provenance problem — you verify the lens is the one you calibrated. Extrinsics are a monitoring problem — the pose will drift in normal operation, and the question is whether you notice before it degrades driving. A fleet that treats both as one-time factory steps is exposed on the extrinsic side for the entire vehicle lifetime. How do you tell model failure from calibration drift? This is the diagnostic that separates a mature perception program from a reactive one. The signal that cleanly separates the two is the reprojection residual — the geometric distance between where a known 3D landmark projects into the image using the current calibration, and where the detector actually observes it. Use this rubric when a perception regression lands: Did detection metrics move? If image-space mAP or recall dropped on a held-out set, suspect the model — data drift, a bad training run, a domain shift the network never saw. If image-space metrics held, the model is probably fine. Did reprojection residuals move? If residuals against fixed landmarks trended upward while detection metrics held steady, the geometry drifted. That is calibration, not the model. Is the error distance-dependent? Position errors that grow with range point to an angular pose error — an extrinsic drift signature — rather than a model localization problem, which tends to be range-agnostic in image space. Is it correlated with a physical event? A step change after a service visit, a curb strike, or a hardware swap points at pose or intrinsics, not the learned weights. Attributing correctly is the whole ROI. A fleet that can say “this regression is calibration, not the model” stops retraining networks that were never broken, and shortens time-to-detect on the failures that actually degrade driving. Separating model error from pose error is precisely the audit boundary our [production AI monitoring harness](Production AI Monitoring Harness) is built to enforce, and it sits inside our broader computer vision engineering practice. What does it take to run ML perception reliably across a fleet? Reliability over time is not a property of the model — it is a property of the system that keeps the model’s geometric assumptions honest. Three things have to be true at fleet scale. You need continuous reprojection monitoring, not one-time factory calibration, because extrinsic pose drifts across a vehicle’s operating life. You need attribution that distinguishes model regressions from geometry regressions, so engineering effort lands where the failure actually is. And you need provenance on the calibration itself — which parameters were in force when a given frame was processed — so that a perception failure can be reconstructed and explained rather than guessed at. Teams that miss this treat every perception incident as a model problem and burn cycles retraining. Teams that get it right build the monitoring boundary into the pipeline from the start. For the safety-evidence angle on the same pipeline, how ML perception meets automotive safety demands covers what an ASIL-graded program expects the perception stack to produce. FAQ What does working with machine learning in self-driving cars involve in practice? A learned detector — usually a YOLO- or DETR-family network — converts camera frames into image-space outputs like bounding boxes, masks, or keypoints. Those outputs are then projected into a 3D world frame using the camera’s calibration and fused with other sensors before reaching planning. The model’s job ends at the image plane; the world position of a detection depends on geometry the model never sees. Where does the learned model end and geometry/calibration begin in a self-driving perception stack? The model owns image-space outputs: what is in the frame, where in the frame, and with what confidence. Geometry owns everything after that — turning a pixel into a metric position using intrinsic and extrinsic camera parameters. The boundary is exact, and the model has no channel to report a geometry problem. Why can a well-trained perception model still fail when camera calibration drifts? When the camera’s extrinsic pose shifts, the detector still draws a correct box in pixels, but the projection step places that detection at the wrong distance and offset because it uses a stale pose. Detection confidence and standard model metrics stay green, so the failure is silent, and the position error grows with range — exactly where trustworthy estimates matter most. What role do intrinsic and extrinsic camera parameters play in a machine learning perception pipeline? Intrinsics (focal length, principal point, distortion) encode the lens optics and rarely change after manufacturing, so they are mainly a provenance concern. Extrinsics (rotation and translation relative to the vehicle) encode the camera’s pose and drift during normal operation, so they are a continuous monitoring concern. Both are required to project image-space detections into a usable 3D position. How do you tell whether a perception failure came from the model or from undetected calibration drift? Check whether image-space detection metrics moved (model) or reprojection residuals against fixed landmarks moved while detection metrics held (geometry). Distance-dependent position errors and step changes correlated with physical events like a curb strike point to pose drift rather than the learned weights. The reprojection residual is the signal that cleanly separates the two. What does it take to run machine learning perception reliably across a fleet over time? You need continuous reprojection monitoring rather than one-time factory calibration, attribution that separates model regressions from geometry regressions, and provenance on which calibration was in force for each processed frame. Reliability is a property of the system that keeps the model’s geometric assumptions honest, not of the model alone. The measurement that replaces the guess The interesting question is not “is our model good enough?” — it usually is. The interesting question is “when driving degrades, can we prove whether learning or geometry broke?” A perception stack that cannot answer that will keep retraining models that were never the problem, while the silent pose drift that actually caused the regression stays invisible. A robustness audit that separates model-quality failures from calibration-geometry failures is what turns that unanswerable question into a residual trend you can watch.
