An inspection cell starts passing defects to the next station. The reflex is to blame the model: pull the scrap bin, label the missed parts, retrain, redeploy. On a live line that reflex is often the most expensive available option — because the thing that changed was frequently not the model.
This is a worked example. One deployment, one regression, one audit, walked in the order we would actually walk it. The deployment is a multi-camera inspection cell checking a stamped and coated metal part for surface defects — scratches, coating voids, edge burrs — across four stations on a two-shift line. Scope is the part and its surface. Nothing here touches operators or people-monitoring; the cameras look at metal.
The regression as the line reported it
The signal did not arrive as a metric. It arrived as a downstream complaint: the assembly station two positions later started rejecting parts the inspection cell had passed. Quality logged an escape-rate rise over roughly a week. The ML team’s dashboard showed model accuracy flat against its last eval run, which is exactly what you would expect, because that eval run was executed against a frozen sample from before the change.
Three candidate causes were live at that moment, and all three produce the same regression signature at the escape-rate level:
- A camera on station 2 had been recalibrated during preventive maintenance; exposure came back slightly hotter.
- A supplier change had altered the coating’s surface finish — glossier, more specular reflection at the same lighting angle.
- A new model version had been rolled out to two of the four cells and not the other two.
A regression that reproduces on some cells and not others is a rollout question, not a model question — and it is answerable in an hour if the deployment records which cell runs which model version. In this deployment that record existed only as a Docker tag in an operator’s terminal history, which is why the hour became three days.
What the audit looks at, and in what order
The audit inspects four surfaces of the deployed inspection feature. It does not begin by looking at the model weights.
| Surface | Question on this line | What the audit found |
|---|---|---|
| Eval coverage | Do the eval sets contain each defect class at each station’s lighting and fixture geometry? | Scratch and void classes well covered; edge burr covered from station 1 geometry only. Specular-finish parts absent entirely. |
| Drift monitors | Is anything watching camera exposure, part mix, conveyor speed, supplier lot? | Latency and cell uptime monitored. No monitor on mean image brightness, no monitor on part-number mix per shift. |
| Release path | How does a model version reach a cell, and is the mapping recorded? | Per-cell manual deploy, no version pinning, no per-cell traffic record. |
| Ownership | Who acts when escape rate rises mid-shift? | Undefined. Quality noticed, controls engineering was blamed, ML team investigated four days later. |
The ordering matters. Eval coverage tells you what the release gate ever tested; drift monitors tell you whether anyone would have seen the change; the release path tells you whether the change is even attributable; ownership tells you how long the gap between event and action stays open. Retraining touches none of those.
Which conditions must an inspection eval set actually cover?
The useful unit of eval coverage on an inspection line is not “defect class” — it is defect class × station geometry × surface condition. That cross product is what the line produces, and it is where coverage gaps hide. On this deployment the practical checklist looked like:
- Every defect class, at every station that is supposed to catch it, in its own fixture geometry and lighting angle.
- Each surface finish variant currently in the approved supplier set, including the glossy one nobody thought to sample.
- Both shifts’ ambient light conditions, if the cell has any path for daylight — enclosures leak more than drawings suggest.
- Marginal parts, not only clear pass and clear fail. The false-reject rate lives entirely in the margin.
- Fixture states after maintenance, not only as-commissioned. A re-seated jig shifts part pose by a few millimetres and that is enough.
Coverage stated as a percentage of a labelled test set tells you almost nothing here. Coverage stated as a grid — classes down, station-and-condition across, cells either filled or visibly empty — is what a release reviewer can act on. That grid is the audit’s first deliverable, and on this line four of twenty-eight cells were empty.
Distinguishing a calibration regression from a genuine retrain case
This is the discrimination the line most needs and least often has instrumented. The audit’s approach is to test the cheap hypotheses first, because they are also the reversible ones.
Replay a fixed set of archived images — parts the current model passed and the downstream station rejected — through both the previous and current model version. If both versions fail the same parts, the model is not the variable; the input changed. Then compare live image statistics against the archive: mean brightness, histogram spread, specular-highlight area. On station 2, mean brightness had moved measurably after the maintenance recalibration, and the glossy-lot parts showed a highlight region sitting exactly where the coating-void classifier looks. Two input changes, compounding.
We see this pattern regularly in vision deployments — the model is the only component with a metric attached, so it absorbs the blame for changes in optics, fixturing and material. In our experience, an archive of raw frames with camera settings and part numbers stamped alongside them is the single cheapest artefact a line can keep, because it converts a week of argument into an afternoon of replay.
The genuine retrain case looks different: replay shows the previous version handling the parts correctly and the current one failing them, or the input statistics are stable and the defect population itself has shifted into a class the training set never contained. Then retrain — with the coverage grid telling you what to label.
Rollout and the shift-scale kill switch
A rollout on an inspection line has a constraint that a web feature does not: you cannot canary a fraction of requests, because each cell is a physical station with its own optics. The unit of gradual rollout is the cell, and it works only if the cell-to-version mapping is recorded and queryable.
The audit’s release-readiness position for this deployment was straightforward. Pin the model version, the preprocessing configuration and the camera profile together as one release object, deploy to one cell for a full shift with escape rate and false-reject rate compared against a held-back cell running the previous version, then widen. Rollback means repointing a cell at the previous pinned release object and confirming it on the next part, not rebuilding an image. The target we hold to is that any shift supervisor can revert a cell within the shift the problem appears in — time-to-rollback measured in shifts, not in weeks of ticket queue.
Ownership follows the same clock. The monitor that flags a brightness shift needs a named recipient at the line, not a Slack channel; the authority to revert a cell needs to sit with someone physically present on both shifts. Where the ML team owns the model and controls engineering owns the optics, someone still has to own the escape-rate number that spans both. That naming is part of the audit, not a follow-up.
What the audit produced, and what it did not
The deliverable set was the eval-coverage grid, a drift-monitor inventory with the four missing monitors specified and costed by sampling rate, a release-readiness checklist for the per-cell rollout, and an ownership matrix with escalation defined per shift. Ranked first in the remediation roadmap: brightness and part-mix monitors, because they were cheap and would have detected both root causes within a shift. Ranked last: the retrain, narrowed to the specular finish and the station-3 edge-burr geometry, which is a fraction of the relabelling the original reflex implied. Explicitly out of scope: the coating supplier qualification process, the PLC logic, and any judgement about the cell’s mechanical design.
The audit is engineering evidence about one deployed feature at one point in time. It does not certify the line, and it does not replace the quality system. What it changes is the sequence in which the next regression gets diagnosed — and on a two-shift line, sequence is cost.
If you are structuring this kind of engagement, the artefact set above is what our Production AI Monitoring Harness produces, and the vision-side engineering it sits on is covered under computer vision systems. The audit’s scope boundaries and the five operational surfaces it tests generally are set out in our production AI reliability audit scope checklist, which is the reference this worked example instantiates on a factory floor.
One open question we have not resolved cleanly across inspection deployments: how often should the eval-coverage grid be re-derived from live production data rather than curated by the team who built it? A grid maintained by hand drifts as quietly as the model it gates.
Frequently Asked Questions
What does a reliability audit applied to an industrial CV inspection deployment mean in practice?
Reliability Audit of an Industrial CV makes this clear: it means examining the deployed inspection feature’s operational surfaces — eval coverage per defect class and station, drift monitors on optics and part mix, the per-cell release and rollback path, and named ownership — before touching model weights. On this line the audit produced a coverage grid, a monitor inventory, a release checklist and an ownership matrix. Retraining was the last item on the roadmap, not the first., the useful coverage unit is defect class × station geometry × surface condition, not defect class alone. That means every class at every station meant to catch it, each approved supplier surface finish, both shifts’ ambient conditions, marginal parts as well as clear pass/fail, and fixture states after maintenance rather than only as-commissioned.
What drift signals are realistic to monitor on a live inspection cell — camera exposure, part mix, supplier surface finish — and at what sampling cost? Mean image brightness and histogram spread per camera are cheap enough to compute on every frame the cell already captures. Part-number mix and supplier lot are metadata joins against the MES, so their cost is integration rather than compute. Specular-highlight area is more expensive and is usually sampled — a fixed fraction of frames per shift is sufficient to detect a finish change.
How should a new inspection model version be rolled out across cells and stations, and what does the kill-switch back to the previous version look like on a running line? Pin model version, preprocessing configuration and camera profile as one release object, deploy to a single cell for a full shift, and compare escape and false-reject rates against a held-back cell before widening. Rollback is repointing that cell at the previous pinned object and confirming on the next part — executable by a shift supervisor within the shift, without an image rebuild.
How do you tell an inspection regression caused by camera recalibration or fixture change apart from one that genuinely needs a retrain? Replay archived frames of the disputed parts through both the previous and current model version. If both versions fail the same parts, the input changed, and the next step is comparing live image statistics against the archive. A genuine retrain case shows the previous version succeeding where the current one fails, or stable inputs with a defect population that has moved into an untrained class.
Who carries the pager when the inspection feature regresses mid-shift — the ML team, controls engineering, or the line supervisor — and what does the escalation path look like? Someone physically present on both shifts must hold the authority to revert a cell; that is usually the line supervisor. The ML team owns the model, controls engineering owns the optics and fixturing, and the audit’s job is to name a single owner for the escape-rate number that spans both, with escalation defined per shift rather than per team.
What does the audit’s remediation roadmap typically rank first in an inspection deployment, and what is explicitly out of scope? First is normally the cheapest monitoring that would have detected the actual root cause within a shift — on this line, brightness and part-mix monitors. Retraining is scoped narrowly to the genuine coverage gaps the grid exposes. Out of scope: supplier qualification processes, PLC and controls logic, and mechanical cell design.
Your Reliability Audit Industrial CV roadmap
Run your existing edge cases through the framework above before commissioning new validation infrastructure.
