A perception stack that starts missing detections in low sun or on wet tarmac gets treated as a model problem almost by reflex. Pull the edge-case frames, retrain the detector, re-benchmark mAP, ship. That sequence is not wrong so much as premature: mAP is simply the number the training framework hands you first, so it becomes the number the investigation orbits. A reliability audit of the same workload starts one layer out — at the deployed surface where the detections are actually produced — and asks whether the regression even lives in the weights.
This is the perception instantiation of the audit surface described in our production AI reliability audit methodology. Same five surfaces, different instruments. What follows walks one camera-based inference deployment through them and shows where each finding landed.
What does a reliability audit look like on a perception workload?
The workload: a multi-camera object detector running on a fleet-deployed inference box, batched behind a model server, feeding a downstream planner. The complaint that triggered the audit: intermittent missed detections, concentrated in low-angle sun and wet-road conditions, reported from the field rather than caught by any monitor.
Before touching the model, the audit inventories four things — what the evals cover, what the monitors watch, how a new model version reaches a vehicle, and who gets paged when quality degrades without the service failing. In a perception workload, a large share of apparent accuracy regressions trace not to the detector but to camera calibration drift, firmware or ISP changes, resolution and preprocessing mismatches between training and inference, or a model-server upgrade that changed batching behaviour and dropped frames under load. None of those are fixed by a retrain. Worse, a retrain performed against them quietly bakes the operational fault into the weights, and the next stack change un-bakes it in a way nobody can trace.
That is the divergence point worth stating plainly: in perception, the regression you can see in mAP is frequently a symptom of an input-path fault, not of a model that has forgotten how to detect things.
Building the eval-coverage matrix from operating conditions
The first audit artefact is an eval-coverage map, and for a vision workload it takes the shape of an operating-condition matrix rather than a class-balance table. The axes that mattered here were lighting (direct low sun, overcast, night with oncoming headlights, tunnel entry/exit), surface and weather (dry, wet with specular reflection, spray), occlusion severity, and sensor variant — because the fleet ran two camera modules with different ISPs, and the evaluation set had been assembled almost entirely from the older one.
That last point is the kind of finding an mAP number cannot express. The aggregate score was healthy because the eval set was dominated by conditions the vehicle rarely struggled in, and the sensor variant carrying most of the field complaints held roughly a tenth of the eval frames. Coverage was not measured, so nobody knew.
| Operating-condition axis | What to bucket | Typical audit finding |
|---|---|---|
| Lighting | Low sun, overcast, night, tunnel transition | Transition frames underrepresented; scored inside “day” |
| Weather / surface | Dry, wet-specular, spray, fog | Wet-specular present but not scored as its own slice |
| Occlusion | None, partial, heavy, truncated at frame edge | Aggregate mAP hides heavy-occlusion collapse |
| Sensor variant | Camera module, ISP firmware, resolution, crop | One variant dominates eval set; fleet split is different |
| Distance / scale | Near, mid, far, small-object band | Far/small band regresses first under any input-path change |
Score each cell separately, and record the fleet-mileage weight beside it. A slice with 2% of eval frames and 30% of real driving exposure is a coverage defect, not a statistical footnote. This is what the eval-coverage section of the [production AI monitoring harness](Production AI Monitoring Harness) contains for a vision workload.
Which drift signals matter when the sensor is part of the model
Prediction drift — the output distribution moving — is the signal most teams instrument first, and it is the last one to move. For a camera workload, three earlier signals sit upstream of it:
- Sensor and calibration signals. Intrinsic and extrinsic calibration residuals, exposure and gain statistics, sharpness or focus proxies, and dropped-frame counts per camera. Calibration drift after a windscreen replacement or a mounting knock changes the geometry the detector was trained against, and it does so gradually.
- Input-distribution signals. Per-channel image statistics, resolution and aspect-ratio of what actually reaches the preprocessing step, and the distribution of operating-condition tags inferred at the edge. This is where an ISP firmware rollout announces itself, days before anyone connects it to detections.
- Serving-path signals. Batch composition, queue depth, per-frame latency against the planner’s deadline, and frames dropped under load. A model server upgrade that changed default batching produced a measurable frame-drop rate here at exactly the times of day when traffic density peaked.
Prediction drift still belongs in the inventory — detection-count-per-frame distributions and confidence-histogram shape are useful — but it is a confirmation signal, not a detection one. Sensor and input drift are physical; prediction drift is their downstream shadow. Teams working through the general distinction will find it developed further in data drift versus model drift; the perception-specific point is that the input side has hardware causes with maintenance fixes attached.
Model problem or input-path problem: the triage order
The audit’s most commercially useful output was a triage order that puts the cheap tests first. A perception retrain-and-revalidate cycle consumes labelling spend, GPU hours, and a full revalidation pass across every operating-condition slice; the audit either justifies that spend or removes it from the table before it is committed. In our engagements, ordering the checks this way is what converts a multi-week data campaign into a configuration change measured in days — an observed pattern across deployments we have audited, not a benchmarked conversion rate.
Triage checklist — run in order, stop when a check fails:
- Reproduce offline on the exact production frames. Pull the failing frames as the model server received them, not as the camera produced them. If the detector performs correctly on them, the fault is upstream of the model.
- Diff the preprocessing path. Resolution, crop, letterbox padding, colour space, normalisation constants, and interpolation kernel — training versus inference. A bilinear-versus-nearest resize difference is enough to cost small-object recall.
- Check calibration and sensor state. Calibration residuals, exposure behaviour in the failing condition, ISP or firmware version against the version present when the eval set was captured.
- Check the serving path under load. Frame-drop rate, batch composition, and whether the timing of failures correlates with throughput rather than with conditions.
- Check version pinning end to end. Model weights, preprocessing config, post-processing thresholds, and NMS parameters each carry their own version and their own cache.
- Only then, score the eval slices. If steps 1–5 are clean and the failing condition slice genuinely underperforms, the retrain is justified — and now it is scoped to a named slice rather than to “edge cases”.
In the deployment audited here, step 2 and step 4 both fired. A preprocessing normalisation mismatch introduced during a serving-framework migration, plus frame drops under peak batching. Neither would have been repaired by more labelled wet-road frames.
Rollout, rollback, and who holds the perception pager
A perception model release does not reach a fleet the way a web service reaches a load balancer. The audit checked whether the rollout path supported a per-fleet-segment canary — a defined subset of vehicles, ideally split by sensor variant and geography — and whether rollback could pin the previous model and its matching preprocessing config as a unit. It could not: the model version was pinned, the preprocessing config travelled with the container image, and rolling back one without the other produced a combination that had never been evaluated. That is a rollback path in name only, and the general design principle behind fixing it is set out in the audit methodology’s rollback section.
Ownership was the other gap. The perception service never returned errors. It returned fewer detections, and the planner absorbed the difference until a human noticed. Nobody was on call for “the answers got quietly worse” — the on-call rota covered availability, and quality degradation had no owner, no threshold, and no page. Naming that owner, with a defined signal and a defined authority to trigger rollback, was the cheapest remediation in the report.
What the audit measured, and what it cost
The findings are trackable on the same axes as any reliability audit, expressed in perception terms: eval-coverage delta across named operating conditions, time-to-detect on calibration and input-distribution monitors, time-to-rollback for a model-plus-config release, and per-release incident rate. The avoided misdirected retrain is the commercial anchor — not because retraining is wrong, but because retraining against an input-path fault is expensive and actively harmful.
Two findings here were configuration fixes shipped in days. One — a genuine far/small-object weakness under spray, confirmed only after the input path was clean — justified a scoped retrain against a defined slice. That is the honest split, and it is roughly what we expect going in: most of the report is remediation, some of it is a mandate to actually do the training work. Where a team wants that surface examined against their own deployment rather than a written example, the engagement side of this work is where it sits.
The open question this audit did not answer: how far the operating-condition matrix should extend before it stops being a coverage map and becomes an unmaintainable combinatorial grid. Four axes with four buckets each is already 256 cells, and nobody labels 256 slices. Where the useful cut-off falls is, in our experience, still decided case by case.
Frequently Asked Questions
What does a reliability audit applied to a perception inference workload mean in practice?
Reliability Audit Applied Perception is simpler than it looks. For Reliability Audit Applied to a Perception specifically, it means examining the deployed perception feature’s operational surface — eval coverage across real operating conditions, drift and sensor monitoring, the rollout and rollback path, and on-call ownership — before examining the model. In practice, the audit produces an operating-condition coverage map, a monitor inventory covering sensor and input signals, and a named owner for quality degradation., enumerate the axes the vehicle actually encounters, bucket each one, and score every slice separately rather than reporting an aggregate. Then record the fleet-exposure weight next to each cell: a slice with a small share of eval frames but a large share of real driving is a coverage defect that aggregate mAP will hide.
Which drift signals matter for a camera-based perception workload, and how do sensor or calibration drift differ from prediction drift? Calibration residuals, exposure and gain statistics, focus proxies, dropped-frame counts, and image-statistic distributions are the upstream signals; prediction drift in detection counts and confidence histograms is downstream of them. Sensor and calibration drift have physical causes and maintenance fixes, and they move earlier than prediction drift, which is why instrumenting only the output side detects the problem late.
How do you tell whether a perception regression is a model problem or a preprocessing, calibration, or serving-path problem? Reproduce the failure offline on the exact frames the model server received. If the detector performs correctly on them, the fault is upstream — diff the preprocessing path, check calibration and firmware state, then check frame drops under load before scoring eval slices.
What does a rollout and rollback path look like for a perception model release across a fleet or device population? A per-segment canary split by sensor variant and geography, with the model weights, preprocessing config, and post-processing thresholds pinned and rolled back together as one versioned unit. Rolling back a model without its matching config produces a combination that was never evaluated.
Who owns the pager when a perception service degrades gradually instead of failing outright? Someone must be named explicitly, because availability-based on-call rotas never fire for a service that returns fewer detections rather than errors. The owner needs a defined quality signal, a threshold, and the authority to trigger rollback without escalation.
Which audit findings in a perception workload are configuration fixes, and which genuinely justify a retrain and revalidation cycle? Preprocessing mismatches, calibration drift, firmware or ISP changes, and serving-path frame drops are configuration or maintenance fixes measured in days. A retrain is justified only once the input path is verified clean and a named operating-condition slice still underperforms — at which point the retrain is scoped to that slice rather than to “edge cases”.
Reliability Audit Applied Perception: deployment checklist
Most teams discover their biggest gaps during stakeholder alignment, not during technical validation. Revisit it when your workload shifts.
