Ask most teams to list the challenges for autonomous vehicles and you get a themed slide: safety, regulation, edge cases, public trust. All real. None of them tell an engineering lead which subsystem is about to miss its safety budget on a foggy off-ramp at 4pm. The useful version of this question is narrower and more uncomfortable — for each perception subsystem the vehicle runs, what condition degrades it, by how much, and what does that cost the safety margin. That reframing is the whole point of this article. A challenge you cannot pin to a subsystem, a trigger condition, and a measurable degradation is not a challenge you can budget against. It is a worry. And worries do not survive a design review. Why “challenges for autonomous vehicles” is the wrong unit The phrase invites a taxonomy of themes, and themes are unfalsifiable. “Edge cases are hard” is true of every machine-learning system ever shipped; it tells you nothing about where this stack fails. We see this pattern in vendor decks constantly — a confident list of hard problems, no mapping to the layer that actually breaks under each one. The perception stack, by contrast, is made of concrete subsystems: 2D detection, 3D object detection feeding the tracker, lane and drivable-surface segmentation, traffic-light and sign recognition, multi-object tracking, and the fusion layer that reconciles camera, radar, and lidar. Each has a distinct failure surface. Occlusion does not degrade lane detection the way it degrades pedestrian tracking. Low sun does not break lidar the way it blinds a forward camera. Treating “the AV” as a monolith hides exactly the information a reviewer needs. The correct unit of analysis is the (subsystem, trigger, degradation) triple. Name the subsystem, name the condition that triggers its failure, and state the degradation as something you could measure — a recall drop, a false-negative rate, a dropout duration. That is what converts “AV is hard” into a matrix a reviewer can score. This article stays inside the perception layer; how those learned components are assembled is covered in our walkthrough of how the AV perception stack actually works. Which CV subsystems are most fragile under occlusion? Occlusion is the failure most people underestimate because the demo videos never show it. A pedestrian steps behind a parked van, reappears two frames later, and the question is whether the tracker held the identity across the gap or spawned a new track — or, worse, dropped the object entirely. The subsystem that breaks first is multi-object tracking, not detection. A detector can miss an occluded object for a frame or two without catastrophe; the tracker is supposed to bridge that gap with a motion model. When occlusion runs longer than the tracker’s coast window, the track dies, the object’s predicted trajectory disappears from the planner’s world model, and the vehicle behaves as if the pedestrian was never there. In configurations we have reviewed, partial occlusion of 60–80% of a pedestrian’s silhouette is where detection recall starts falling off a cliff (observed pattern across engagements; not a published benchmark) — well before full occlusion, which teams tend to plan for. The trigger conditions worth budgeting are dense pedestrian scenes (crossings, transit hubs), parked-vehicle corridors, and cyclists weaving between cars. The degradation to measure is tracking recall and ID-switch rate under scripted occlusion, not clean-scene mAP. For the mechanics of how detections become persistent identities — and where that association fails — see our piece on how an object tracker works. How do weather and lighting degrade specific subsystems? Adverse conditions are not one challenge; they are several, and each hits a different layer. Lumping “bad weather” together is how teams end up with a safety case that passes on paper and fails in November. Low sun blinds forward cameras through glare and lens flare. The subsystem that breaks is traffic-light and sign recognition, because the relevant signal is a small, high-contrast region that glare washes out. Radar and lidar are unaffected, so fusion can partly cover — but only for geometry, not for the semantic content of a light’s color. Snow breaks lane and drivable-surface segmentation. Painted markings vanish under accumulation, and the segmentation model, trained largely on visible markings, has no signal to fall back on. This is a data-distribution failure, not a sensor failure. Fog degrades both camera detection range and lidar returns (backscatter), which is dangerous because it correlates failures across sensors the fusion layer assumed were independent. Heavy rain adds motion blur and dropout on camera, and creates spurious lidar returns from spray. The honest answer to “by how much” is: it depends on your training distribution, and you cannot quote a portable number. What you can do is measure per-condition degradation on held-out data captured in each condition. A vendor who reports a single aggregate accuracy number across all weather is hiding the November problem. Weather–subsystem degradation matrix Condition Subsystem hit first Failure mechanism What to measure Low sun / glare Traffic-light & sign recognition Small high-contrast region washed out Signal recall vs. sun-angle bins Snow accumulation Lane / drivable-surface segmentation Painted markings occluded; out-of-distribution Segmentation IoU on snow-captured data Fog Camera range + lidar (correlated) Contrast loss + backscatter break sensor independence Detection range vs. visibility; fusion disagreement rate Heavy rain Camera detection + lidar Motion blur, dropout, spray returns False-negative rate; spurious-return count Night Pedestrian detection Low SNR on unlit road users Per-class false-negative rate at night Each row is a (subsystem, trigger, degradation) triple you can score. That is the deliverable, not the prose around it. What happens when camera, radar, and lidar disagree? The comforting story about sensor fusion is that three sensors are more reliable than one. The uncomfortable truth is that fusion turns three inputs into one decision, and when the inputs disagree, the fusion policy is the safety-critical logic — not the sensors. Disagreement takes a few shapes. Camera sees a pedestrian that lidar (sparse at range) does not confirm. Radar reports a stationary return that camera classifies as an overhead sign, not an obstacle in-lane — the classic radar false-positive that early systems mishandled with fatal results. Lidar and camera disagree on the distance to a low-contrast object. In each case the fusion stack must resolve the conflict, and the resolution rule embeds a safety philosophy: does an unconfirmed camera pedestrian get treated as real (safe, but prone to phantom braking) or discarded (smooth, but prone to missing a real one)? There is no universally correct answer, which is exactly why this is a challenge rather than a bug. The right policy is conditional on which sensor is trustworthy in the current context — camera semantics are strong in daylight, lidar geometry is strong when the point cloud is dense, radar is strong for closing-velocity on metal. A fusion layer that applies a fixed weighting regardless of condition is the failure mode to look for. How these inputs are combined into detection, tracking, and a coherent world model is the subject of our deeper piece on sensor fusion engineering. This is also where the on-vehicle compute budget bites. Resolving disagreement well often means running heavier models or more association hypotheses, and that has a latency cost the planner cannot absorb indefinitely — the embedded-inference constraint that gates several of these challenges. The trade-off between perception quality and the latency contract is a whole problem of its own, treated in where perception breaks the latency contract. How do rare and long-tail events expose model limits? Trained perception models are interpolators. They are excellent at things resembling their training distribution and structurally weak at things outside it — and the road is an endless generator of the outside. A mattress on the highway, a person in a horse costume, a jackknifed truck presenting an orientation the training set never showed, an emergency vehicle with an unusual light bar. The failure mechanism is specific: the model does not know it does not know. A detector trained on the 80-odd everyday categories — the kind covered by standard benchmarks like the COCO dataset classes — assigns a novel obstacle to the nearest known class or to background, often with misleadingly high confidence. The danger is not the miss; it is the confident miss, because a well-calibrated system could at least flag uncertainty and hand control back. The practical response is not “collect more data” — the long tail is by definition unbounded. It is threefold: treat the drivable-surface / free-space model as a class-agnostic backstop so that something unexpected in the lane triggers caution even when classification fails; separate aleatoric from epistemic uncertainty so the stack can distinguish “noisy input” from “never seen this”; and run continuous mining of disengagements and near-misses to feed the tail back into training. The uncertainty distinction is worth understanding precisely, because conflating the two produces a system that brakes for rain the same way it brakes for the genuinely unknown — see aleatoric vs epistemic uncertainty. How do you turn these challenges into a testable matrix? The reviewer’s job is to refuse the themed slide and demand the triples. For each perception subsystem the vendor claims, ask: which conditions degrade it, what is the measured degradation on data captured in that condition, and what is the safety-budget allocation against that degradation. A claim that cannot fill those three columns is not scoreable. Vendor-claim scoring rubric For each subsystem × condition cell, score the vendor’s evidence: Named condition, no data — “we handle fog” with no per-condition metric. Score: 0. This is a worry, not a claim. Aggregate metric only — one accuracy number across all conditions. Score: 1. Hides the worst case. Per-condition metric on held-out data — recall/IoU reported per weather-and-lighting bin. Score: 2. Per-condition metric + degradation curve + safety-budget allocation — the degradation is quantified and mapped to a margin the planner can absorb. Score: 3. Only cells scoring 2 or above belong in a safety case. Where changes to a perception model are validated statistically before they ship, the A/B test statistics for AV computer vision piece covers how to keep those comparisons honest. The full production picture — how these perception subsystems are scoped, budgeted, and connected to planning — sits within our broader computer vision practice. FAQ How does challenges for autonomous vehicles actually work? Treated as themes — safety, regulation, edge cases — the question is unanswerable. In practice it becomes tractable only when each challenge is mapped to a specific perception subsystem, the condition that triggers its failure, and a measurable degradation. That (subsystem, trigger, degradation) triple is what an engineering lead can budget a safety margin against. Which CV subsystems are most fragile under occlusion, and how does that degrade pedestrian and object tracking? Multi-object tracking breaks before detection does. Detection can miss an occluded object for a frame or two; the tracker is supposed to coast through the gap with a motion model, and when occlusion outlasts its coast window the track dies and the object leaves the planner’s world model. Partial occlusion — roughly 60–80% of a silhouette — is where recall degrades sharply, earlier than teams who only plan for full occlusion expect. How do adverse weather and lighting degrade specific perception subsystems, and by how much? Each condition hits a different layer: low sun breaks traffic-light recognition, snow breaks lane segmentation, fog degrades camera and lidar together (breaking their assumed independence), and night raises pedestrian false-negatives. The “by how much” is not portable — it depends on the training distribution — so the answer is to measure per-condition degradation on data captured in each condition, never a single aggregate number. What happens when camera, radar, and lidar disagree, and how should a fusion stack resolve the conflict safely? Fusion collapses three inputs into one decision, so when they disagree the fusion policy becomes the safety-critical logic. The resolution rule should be conditional on which sensor is trustworthy in context — camera semantics in daylight, lidar geometry with a dense point cloud, radar for closing velocity — rather than a fixed weighting. A fusion layer that weights sensors identically regardless of condition is the failure mode to look for. How do rare and long-tail events expose the limits of trained perception models? Trained models interpolate well and extrapolate badly, so a novel obstacle gets assigned to the nearest known class or to background, often with high confidence. The danger is the confident miss, not the miss itself. The mitigations are a class-agnostic free-space backstop, separating aleatoric from epistemic uncertainty so the stack can flag “never seen this,” and continuous mining of disengagements back into training. How should a reviewer turn these challenges into a testable matrix to score vendor perception claims? Build a subsystem × condition matrix and score each cell by evidence quality: no data (0), aggregate metric only (1), per-condition metric on held-out data (2), or per-condition metric plus degradation curve plus safety-budget allocation (3). Only cells scoring 2 or above belong in a safety case, and any claim that cannot fill the (condition, measured degradation, budget) columns is a worry rather than a scoreable claim. The matrix is not paperwork; it is the risk register that sits beside a subsystem scope, flagging where each capability axis is most likely to miss its safety budget before the vehicle ever sees the condition that proves it.
