A defect-detection model that scored 98% in a pilot cell and then loses points the moment it goes line-side is not a broken model. It is an unhardened one. Pilot accuracy is measured under conditions the production line will not honour — fixed lighting, a clean conveyor, hand-fed parts, an engineer watching the screen. The line honours none of that. The gap between those two environments is where most industrial computer-vision deployments quietly regress, and it is almost always invisible until a shift supervisor notices the scrap bin filling with parts the inspection station passed. The naive move at this point is to ship the pilot model to the line and watch it through the same dashboards used in the lab. That assumes pilot accuracy transfers unchanged. It does not. The expert move is to treat the move to production as its own engineering pass — a hardening pass — that anticipates the operational realities the lab never surfaced and instruments specifically for the regression modes the line will introduce. A deployment that survives that pass keeps inspection accuracy in line with the pilot. A deployment that skips it ships a regression the line cannot tolerate. Why Do CV Defect-Detection Pilots Fail When They Move to the Production Line? The pilot validates one thing: that the defect is detectable by a model under controlled conditions. That is a necessary result and it is worth having — it is the question the feasibility audit before pilot is built to answer. But “detectable in a cell” and “detectable on the line, every shift, for the next eighteen months” are different claims, and the pilot only ever made the first one. Across the industrial CV deployments we have worked on, the dominant failure is not the model architecture. It is the distance between the pilot’s data distribution and the line’s. A pilot collects a few thousand images under one lighting rig, one camera angle, one batch of product. The line presents the same model with morning sun through a skylight, a packaging redesign nobody told the vision team about, a conveyor running 15% faster than the pilot ever did, and a maintenance crew who repositioned the camera by two centimetres to clear a jam. Each of those shifts the input distribution away from what the model was validated against, and the model has no way to tell you it is now extrapolating rather than recognising. This is an observed pattern across our engagements, not a benchmarked failure rate — but the shape is consistent: the regression is rarely a cliff. It is a slow erosion that the lab dashboards, which were never designed to watch for distribution drift, do not register until the downstream quality numbers move. What Changes Between Pilot Lighting and Line Lighting? Lighting is the single most under-budgeted variable in the move to production, and it deserves its own treatment because it drives more drift than any other environmental factor we see. In the pilot cell, lighting is controlled, frontal, and stable. On the line it is none of those things. Ambient light leaks in through skylights and doors and changes across the day and the season. Overhead fixtures age and shift colour temperature. A reflective component that was matte in the pilot batch turns glossy after a supplier change, and specular highlights start tripping the defect classifier on parts that are actually fine. None of these are exotic — they are the normal physics of a working floor, and a model trained on a narrow lighting distribution treats every one of them as a novel input. The hardening response is not “buy a better camera.” It is to characterise the line’s actual lighting envelope before deployment — measure it across shifts, across the day, across product variants — and either control it with dedicated machine-vision lighting and optics or expand the training distribution to cover it. The deployments that survive are the ones that decided, deliberately, which of those two paths to take. The ones that regress are the ones that assumed pilot lighting was representative. A Deployment Hardening Checklist This is the diagnostic we run before signing off an industrial CV model for line duty. Each row is a question that the lab almost never forces you to answer, and a model that cannot answer it is not production-ready regardless of its pilot score. Hardening dimension The question the line forces Pass condition Lighting envelope Has the line’s lighting been measured across shifts and seasons, not assumed from the pilot? Training distribution covers, or fixturing controls, the measured envelope Part variance Does the model see packaging redesigns, supplier changes, and product variants in its validation set? Known variants represented; an unknown-variant path defined Conveyor dynamics Was the model validated at the line’s real speed and jitter, including motion blur? Validated at production speed, not pilot speed Camera stability What happens when maintenance bumps or repositions the camera? Drift-from-baseline alarm on framing/position Drift monitoring Is the model watched for input-distribution drift, not just output accuracy? Distribution-drift signal instrumented and alarmed Rollback path When the model misbehaves, can the line revert without stopping? Defined fallback (prior model or human gate) with a tested switch Ownership Who is paged when the model degrades at 02:00? Named on-call owner with a runbook If a row has no answer, that row is the regression you will ship. The point of the table is to surface those gaps while they are still cheap to close — before the line is depending on the station. How Do We Monitor Model Drift on a Production Line? The lab watches accuracy. Accuracy is a lagging signal: by the time it drops, defective parts have already passed and good parts have already been scrapped. The hardened deployment watches the inputs as well as the outputs, because input-distribution drift precedes accuracy loss. In practice this means instrumenting two layers. The first is a distribution monitor on the incoming images — feature-embedding drift, brightness and contrast statistics, defect-rate-per-shift trends — that fires when the line starts presenting the model with inputs unlike its training set. The second is the classic quality-control layer, where the model’s outputs feed control charts and the inspection process itself is watched the way any process is watched. Pairing the model with statistical process control on the production line turns the inspection station from a black box into a monitored process — one where an out-of-control signal tells operations something changed before the scrap numbers confirm it. The measurable targets that matter here are concrete: the production-line accuracy versus pilot accuracy delta, the time-to-detect when drift begins, and the time-to-rollback once it is detected. These are operational measurements you take from the deployed system, not industry estimates — and they are the numbers a hardened deployment can actually report, because it was built to produce them. What Rollback Strategy Keeps the Line Moving When the Model Misbehaves? Every defect-detection model will, at some point, degrade — after a supplier change, a lighting shift, a camera bump, or a packaging redesign. We do not promise zero line stoppages from a model regression; anyone who does is selling the pilot fantasy. What a hardened deployment does promise is that when the model misbehaves, operations has a defined path that keeps the line moving instead of an emergency. That path has two common shapes. The first is reverting to a previous model version that is known-good — which only works if the deployment kept versioned models and has a tested switch, not a redeploy-from-scratch scramble. The second is falling back to a human inspection gate or a permissive-pass-with-flagging mode while the model is fixed offline, trading some inspection coverage for line continuity. Which one is right depends on the cost of an escaped defect versus the cost of a stopped line, and that trade-off is a decision the deployment team makes deliberately, not one the line discovers at 02:00. The avoided cost here is real and measurable: a tested rollback path is the difference between a five-minute model switch and a multi-hour line stoppage while someone figures out how to turn the inspection station off safely. The reasoning behind treating rollback as a first-class artefact — alongside evals, drift, and ownership — is laid out in what a production AI reliability audit actually tests, which applies the same hardening lens beyond CV. Who Owns the Inspection Model in Production? This is the question that quietly determines whether everything above survives contact with reality. A pilot is owned by the team that built it. A production inspection model needs an owner who is on-call when it degrades, who holds the runbook, and who has the authority to trigger a rollback. Without that, the drift monitor fires into an empty inbox and the rollback path stays theoretical. We see ownership treated as an afterthought far more often than as a deliverable, and it is consistently the gap that turns a recoverable regression into a line incident. The hardening pass names the owner, writes the runbook, and tests the page — because a model nobody is paged for is a model that will fail silently. The line-side artefacts that encode this ownership, the eval gates, and the drift alarms are the subject of the industrial-CV inspection reliability artefacts that keep a line-side model running, which is where this hardening work gets signed against. FAQ Why do CV defect-detection pilots fail when they move to the production line? The pilot only proves the defect is detectable under controlled conditions — fixed lighting, clean conveyor, hand-fed parts. The line changes the input distribution through lighting drift, packaging redesigns, conveyor speed variance, and camera repositioning, and the model has no way to signal that it is now extrapolating rather than recognising. The regression is usually a slow erosion that lab dashboards never registered because they were not built to watch for distribution drift. What changes between pilot lighting and line lighting? Pilot lighting is controlled, frontal, and stable; line lighting is ambient, variable across the day and season, and subject to aging fixtures and reflective surface changes after supplier batches. A model trained on a narrow lighting distribution treats each of these normal floor conditions as a novel input. The fix is to measure the line’s actual lighting envelope before deployment and either control it with fixturing or expand the training distribution to cover it. How do we monitor model drift on a production line? Watch inputs as well as outputs, because input-distribution drift precedes accuracy loss. Instrument a distribution monitor on incoming images — embedding drift, brightness and contrast statistics, defect-rate-per-shift trends — and feed the model’s outputs into statistical process control charts. The targets that matter are the production-vs-pilot accuracy delta, time-to-detect on drift, and time-to-rollback. What rollback strategy keeps the line moving when the model misbehaves? Two common paths: revert to a known-good previous model version with a tested switch, or fall back to a human inspection gate or permissive-pass-with-flagging mode while the model is fixed offline. Which one is right depends on the cost of an escaped defect versus a stopped line. We do not promise zero stoppages from a regression — only that a hardened deployment turns the response into a defined five-minute switch instead of an emergency. Who owns the inspection model in production? A production inspection model needs a named on-call owner who holds the runbook and has the authority to trigger a rollback. Without that, the drift monitor fires into an empty inbox and the rollback path stays theoretical. The hardening pass names the owner, writes the runbook, and tests the page, because a model nobody is paged for is a model that will fail silently. The Hardening Pass Is the Real Deployment The pilot answered whether the defect is detectable. The hardening pass answers whether the line can depend on that detection through lighting drift, packaging changes, conveyor variance, and the inevitable day the model degrades at 02:00. Those are different questions, and the deployments that survive are the ones that treated them as such — measuring the lighting envelope instead of assuming it, instrumenting for distribution drift instead of waiting for accuracy to fall, and building a tested rollback before the line needed one. If your team has a working pilot and is now staring at the line, the right next conversation is not “which model” — it is which of the rows in the hardening checklist currently have no answer. That is the gap between a pilot that demonstrated and a deployment that survives. Our computer-vision engineering practice and the way we scope these engagements start from exactly that checklist, against the production AI monitoring harness we use to keep line-side inspection models honest.
