A pilot accuracy figure is not a property of the model. It is a property of the model plus the acquisition conditions it was measured under — fixed lighting, one packaging revision, a curated defect mix, a conveyor at demonstration speed. When a defect-detection pilot clears its acceptance threshold in a validation cell and then underperforms on the line, the model did not degrade. The inputs changed, and nobody wrote down which ones could.
That distinction decides how the next two weeks go. Teams that can attribute a line regression to a specific shifted input fix the cause and recover. Teams that cannot end up retraining against a moving target, shipping a new checkpoint every few days, and eventually treating the inspection signal as advisory noise that operators learn to click past.
This piece is the failure catalogue, not the remedy plan. The point is to have the list before deployment, so that the first bad shift is a bounded check rather than an investigation.
Which inputs actually shift between a validation cell and a live line?
Almost every lab-to-line regression we have looked at traces back to one of a small number of shifts. They are boring individually. Collectively they are why “it worked in testing” is the most common opening sentence in an inspection-hardening conversation.
| Shifted input | What the pilot assumed | What the line does | First visible symptom |
|---|---|---|---|
| Illumination | Fixed fixture, one exposure setting | Lamp output decays, replacement fixtures differ in colour temperature, skylights and bay doors change ambient contribution across a shift | Confidence scores compress toward the decision threshold; false rejects cluster by time of day |
| Product revision | One packaging or board revision | Artwork, substrate, or component placement changes without a model release | A defect class becomes systematically missed on one SKU only |
| Camera geometry | Calibrated once, untouched | Position and focus shift after a wash-down, a maintenance swap, or a knock from a fork truck | Localisation drifts before classification does; crops start clipping the region of interest |
| Conveyor speed | Demonstration speed, stable | Line runs at rate, with surges and slowdowns | Motion blur and dropped frames; failures correlate with throughput, not with defect type |
| Defect mix | Curated set, classes roughly balanced | Long-tail classes rare in normal running and dominant during a bad batch | Accuracy holds in aggregate and collapses exactly when it matters |
| Annotation policy | One labeller, one interpretation of borderline | Operators and QC apply a different borderline call | Disagreement on marginal units; “the model is wrong” reports that are really policy disputes |
Each row is a measurable input. That is the useful property: a shifted input can be instrumented and checked, whereas “the model got worse” cannot.
Why pilot accuracy overstates line accuracy
The pilot number and the line number are not measuring the same quantity, so comparing them directly invites the wrong conclusion.
The honest comparison is false-negative escape rate and false-positive reject rate measured on production throughput, over a full production cycle — not accuracy on the pilot set. A pilot set is usually enriched with defects so the model has something to learn from; production is overwhelmingly good units. The same model, unchanged, will show a very different false-positive burden per shift once the denominator becomes real volume, because a small false-positive rate against tens of thousands of units becomes an operator workload that stalls a line.
Class prevalence does the same thing in the other direction. A defect class that was 8% of the pilot set and is 0.3% of normal production looks well-handled right up until a supplier changes and that class becomes 15% of a batch. Aggregate accuracy will not warn you. Per-class recall on the classes you cannot afford to miss will.
We treat this as the standard reporting split on inspection work: pilot performance is reported as conditional, with the conditions listed, and the deployment target is stated separately in escape and reject terms. The structural reasons this gap exists — and how a hardening pass closes it — are developed in our broader treatment of computer vision defect detection in manufacturing, which covers the deployment sequence this catalogue feeds into. For the engineering side of the acquisition chain, our computer vision work is the relevant entry point.
Distribution shift or an under-represented defect mix?
These two get conflated, and the fix for one does nothing for the other.
A distribution shift changes the appearance of units that were already covered. Good parts start looking different: brighter, blurrier, differently framed. The signature is that scores move for both classes — good units drift toward the reject side, defects drift toward the accept side, and the whole score distribution translates or widens. Input-side statistics (exposure, white balance, focus measure, histogram moments) will have moved too.
An under-represented defect mix leaves good units looking exactly as they did. Input statistics are stable. What changes is that a specific defect class, rare or absent in training, arrives in volume and is missed consistently rather than marginally. Confidence on those misses is often high, which is the tell: the model is not uncertain, it is confidently applying a category it was never taught to reject.
The diagnostic sequence is short. Check input-side statistics against the deployment baseline first. If they are stable, it is a coverage problem, and retraining with the missing class is the right response. If they have moved, retraining will bake a temporary condition into the model, and the fix belongs upstream — in the fixture, the calibration, or the frame budget.
Early symptoms, before scrap or escape rates move
Scrap and escape rates are lagging indicators. By the time they move, product has shipped or been binned. The symptoms that lead them:
- Score-distribution compression. The gap between the good-unit and defect-unit score modes narrows. Nothing has crossed the threshold yet; the margin is eroding.
- False rejects clustering in time. Rejects grouped by shift, by hour, or by the interval after a wash-down point at a physical cause, not a modelling one.
- Rising manual-override rate. Operators overriding the model is a labelled dataset arriving in real time. A climb in override rate precedes measured accuracy loss.
- Per-SKU divergence. Aggregate performance holds while one product variant quietly degrades — the classic signature of an unannounced revision change.
- Latency creep at rate. Inference queueing when the line surges means frames are being dropped, and dropped frames are silent false negatives.
Only the last of these is a model-serving issue. The rest are input-side, which is why an inspection deployment needs monitoring on the image and the process, not only on the prediction.
When retraining is the wrong response
Retraining is the default reflex because it is the action the team knows how to take. It is the wrong first move whenever the cause is a condition that will keep moving.
If lighting has drifted, retraining teaches the model the current drift state and guarantees a second regression when the lamp is replaced. Fix the illumination or normalise the acquisition, then reassess. If the camera has moved after a wash-down, retraining absorbs a mechanical fault into the weights and hides it from maintenance. If the annotation policy differs between the pilot labeller and line QC, retraining resolves a definitional disagreement by arbitrary majority vote instead of settling it with the people who own the spec.
Retraining is the right response in one clean case: the input conditions are stable and verified against baseline, and the model is missing a defect class it demonstrably never saw. That is a coverage gap, and more data is the actual remedy.
A useful discipline before deployment is to surface these modes deliberately rather than waiting for the line to find them — a structured reliability audit against the catalogue above, with each row either instrumented or explicitly accepted as a residual risk. Our engineering services engagements on inspection work usually start there, because the list is cheaper to write than to discover.
How to test a pilot against anticipated line conditions
Before committing to deployment, run the pilot model against deliberately perturbed acquisition rather than only against a held-out split:
- Re-acquire a sample of known-good and known-defect units under the illumination extremes the line will actually produce — start of shift, end of lamp service life, doors open.
- Capture at production rate, not demonstration rate, and measure how many frames the pipeline drops when the line surges.
- Include every product revision currently in circulation, plus the one engineering is about to release.
- Re-run after a simulated wash-down and a camera remount, without recalibrating first, to see what geometry tolerance the model actually has.
- Reweight the evaluation set to production prevalence and report escape and reject rates at that prevalence, not at pilot prevalence.
- Have line QC re-label the borderline units independently and quantify the disagreement before blaming the model for it.
Anything that fails these becomes either a fixture change, a monitored signal, or a documented residual risk carried into the deployment. Nothing on the list requires a new model.
The uncomfortable part is that this catalogue does not shrink with model quality. A better architecture does not stabilise a lamp or announce a packaging revision. So the open question for any team about to move a pilot to a line is narrower than it looks: which of these shifts will you detect within one shift, and which will you find out about from a customer?
Frequently Asked Questions
What does it mean in practice when a CV defect-detection pilot fails on the production line?
Most CV defect-detection pilots collapse in production because live inputs diverge from the curated samples used during training, not because the model itself degrades. In practice this shows up as escaped defects, a false-reject burden that stalls the line, or both at once — and it is diagnosed as a modelling problem far more often than it is one., illumination over a shift and over lamp service life, product or packaging revisions released without a model update, camera position and focus after a wash-down or maintenance swap, conveyor speed and the frame budget it leaves, defect-class prevalence, and the annotation policy applied to borderline units. Each is measurable, which is what makes the failure attributable.
Why does pilot accuracy overstate line accuracy? A pilot set is typically enriched with defects and captured under one set of conditions, so it measures a different quantity than production. The honest comparison is false-negative escape rate and false-positive reject rate at production prevalence across a full production cycle, reported alongside per-class recall for the classes you cannot afford to miss.
How do we tell a genuine distribution shift apart from a defect mix the pilot set under-represented? Check input-side statistics — exposure, white balance, focus measure, histogram moments — against the deployment baseline. If they have moved and scores drift for both good and defective units, it is a distribution shift. If they are stable and a specific class is missed confidently and consistently, it is a coverage gap.
What early symptoms indicate a lab-to-line regression before scrap or escape rates move? Score-distribution compression, false rejects clustering by shift or by the interval after a wash-down, a rising operator override rate, per-SKU divergence while aggregate accuracy holds, and inference latency creeping when the line runs at rate.
When is retraining the wrong response to a line regression? Whenever the cause is a condition that will keep moving — drifting illumination, a shifted camera, or a disputed annotation policy. Retraining then encodes a temporary state into the weights and hides a mechanical or definitional fault. It is the right response only when inputs are verified stable and the model is missing a class it never saw.
How do we test a pilot against anticipated line conditions before committing to deployment? Re-acquire samples under the illumination and geometry extremes the line will produce, capture at production rate to expose dropped frames, include every product revision in circulation, and reweight the evaluation to production prevalence. Then have line QC independently re-label the borderline units and quantify the disagreement.
Three questions before your next pilot
Will line conditions match your test environment, can you deploy updates without halting production, and who owns false-negative risk? Revisit it when your workload shifts.
