The choice between model retraining and recalibration should not be made in advance when a vision model moves from a pilot line to new production lines. Set an acceptance threshold for accuracy drop against the pilot baseline before the first new line is tested. Then measure the actual drop on each line and let the result pick the tier. If the drop stays inside the threshold, requalify the model unchanged. If the loss traces to a fixable systematic offset, recalibrate. Retrain only when the drop is still there after recalibration. This order follows from what each action touches: recalibration changes the map applied to a model’s outputs without changing how it ranks inputs, while retraining changes the model’s parameters. Whichever tier a line lands in, change control and risk assessment decide how much requalification the change needs.
How do we decide whether a new line needs retraining, recalibration or just requalification?
Decide it per line, from a measurement. Run the existing model on a labelled sample from the new line, compare accuracy for each defect class with the pilot-line baseline, and escalate only as far as the measured drop requires.
TechnoLynx supports this decision through its Computer Vision R&D engagement. TechnoLynx defines the accuracy metric and the acceptance threshold for each defect class against the pilot baseline. TechnoLynx then designs a measurement protocol covering the camera angles and conveyor speeds of each new line, and builds whatever recalibration or retraining pipeline the measurement calls for. The first step is to define the accuracy metric and acceptance threshold for each defect class.
The three tiers ask for different amounts of work, so it pays to know which one you are in before anyone opens a training script.
- Requalify as-is. The measured drop sits inside the threshold. The model and its settings stay the same, and the line gets a documented check against predetermined acceptance criteria. WHO good manufacturing practice guidance for pharmaceutical validation expects results to be “reviewed, analysed and compared against the predetermined acceptance criteria” and statistically analysed where appropriate. It also expects qualification to be repeated when changes are made, depending on the outcome of risk assessment. We treat a new line as such a change, with risk assessment setting how much requalification it needs.
- Recalibrate. The drop exceeds the threshold, but it comes from a stable, systematic difference: a fixed mounting angle, a consistent shift in score distributions. The fix sits around the model. You can correct the image geometry before inference, re-tune the decision threshold for each class, or refit the confidence calibration. Then you re-test against the same threshold.
- Retrain. The drop is still above the threshold after recalibration, and a shift that compromises the model’s ranking itself requires retraining.
The EU’s Annex 15 guidance asks that planned changes which may affect product quality are documented and their impact on validated status assessed. In practice, the requalify tier is a short protocol, not a formality to skip.
Model retraining vs recalibration: what each tier can and cannot fix
The idea behind a tiered response is simple. The research paper behind drift-triggered reliability control puts it this way: “not all drift warrants retraining”. Moderate shifts mainly distort probability calibration, which recalibration can handle, while severe shifts can compromise ranking itself, which needs retraining. That framework acts only when drift coincides with a measurable fall in calibration error or discrimination against set thresholds, so harmless shifts in the data do not trigger updates. On a credit-risk dataset over nine annual windows, its chosen thresholds produced one retraining and two recalibrations, compared with nine retraining events under rolling retraining. The cost was 12 units against 45 under the paper’s weighted cost model. The data there is credit-risk records rather than images, so read the result as evidence that gating works, not as a prediction for your line.
Temperature scaling, a single-parameter recalibration, does not change the maximum of the softmax output. Class predictions stay the same, so the method “does not affect the model’s accuracy” (Guo et al., ICML 2017). Recalibration aligns confidence scores more closely with actual accuracy. It cannot restore recall the model lost because it now scores defective parts below good ones.
Recalibration also weakens as the shift grows. A study of current-generation image models found that post-hoc calibration works well in-distribution and under mild corruptions, but its effect falls sharply as shift severity rises. Under severe shift it can leave calibration worse than no calibration at all. The same study found that the best calibration method changed between natural images and biomedical images, which argues for evaluating on your own domain.
Tier decision table for a new line
| Measured on the new line | Likely cause | Tier | What changes |
|---|---|---|---|
| Accuracy per class within threshold, confidence lower bound above the limit | Benign shift | Requalify | Nothing in the model; documented check against acceptance criteria |
| Confidence scores off, ranking intact | Calibration distortion | Recalibrate | Calibration map refit on held-out line data |
| Drop persists after recalibration | Ranking damaged by shift | Retrain or fine-tune | Model weights, then calibration on top |
| Blur visible in images at line speed | Exposure too long or lighting undersized | Capture fix | Shorter exposure, stronger strobe, global shutter |
What accuracy drop against the pilot baseline should trigger retraining?
The threshold has to come from your pilot’s own launch quality, be set for each defect class, and carry a confidence bound. Google’s ML Test Score rubric (Breck et al.) says quality thresholds “must be set” as acceptable quality, for example based on the bounds of quality at the launch of the initial system, and that a responder should be alerted if quality moves outside them. The same rubric separates two failure shapes. Slow degradation over many versions suits loose thresholds measured on a validation set. A sudden drop in a new version suits a tighter comparison against the previous model. Moving to a new line looks like the sudden case, so compare against the pilot model’s results tightly.
Pharmaceutical visual inspection gives a useful template for the shape of the threshold, though not for its value. One article on qualifying human visual inspectors treats an accuracy limit of not less than 90% as reasonable, where the limit means 95% confidence that future accuracy will not fall below 90%. It allows a stricter not-less-than-95% limit for requalifying experienced inspectors. That article says inspectors whose lower bound fell below 90%, “especially for distinguishing defective units”, should undergo more extensive training. The lessons that carry over to a model are to set the limit on a confidence bound, not a point estimate, and to look at the defective classes on their own.
One vendor validation checklist gives an example of concrete retraining triggers. It names an F1 score falling more than 5% from the acceptance baseline on a golden dataset, a sustained rise in false-negative rate of more than 0.5 percentage points per week, new defect morphologies, and documented process changes. It argues that triggers should be documented conditions, not a calendar interval. The same source warns that F1 should never be the only gating metric, because a model at 95% F1 can still miss too many rare critical defects. Treat those figures as one vendor’s defaults.
Two practical constraints shape the threshold:
- Test size limits how fine the threshold can be. Beleites et al. found that 75 to 100 test samples are usually needed to validate a good but imperfect classifier with reasonable precision. Showing that one classifier beats another often needs hundreds of independent test samples. A threshold tighter than your new-line test set can resolve will produce verdicts set by noise.
- One global threshold hides class differences. If defect classes react differently to camera angle or speed, an overall accuracy figure can pass while one class fails. Our piece on stratified evaluation across conditions, sites and hardware covers how to break the metric down so this shows up.
Rabanser et al. separate “malignant” shifts that damage performance from benign ones by checking accuracy on the most shifted samples against a threshold that “can be user-specified to account for varying sensitivities to accuracy drops”. Their experiments also found that two to three orders of magnitude fewer labelled examples than the total sample size were enough to approximate target accuracy well. A new line does not need a full labelling campaign before you can make the call.
A related job is telling whether a drop comes from a camera or lighting change or from a genuinely new defect pattern. That diagnosis is its own problem: it decides whether you recalibrate or start collecting new classes, and it needs its own protocol.
How much accuracy do camera angle and conveyor speed cost?
We recommend checking the threshold by measuring each line rather than predicting the loss. In a 2026 study of OCR on a walking person’s camera, not an inspection line, OCR accuracy fell from 0.514 at a 0° viewing angle to 0.331 at 15° and 0.170 at 30°. Mean trial accuracy fell from 0.495 at slow walking speed to 0.173 at very fast speed. The authors note their controlled indoor results are likely an upper bound.
A camera vendor’s guide from The Imaging Source recommends global shutter sensors, external triggering and short exposures for fast lines. It says to divide conveyor speed by part pitch to get the minimum frame rate, so a 1.5 m/s conveyor with 50 mm pitch needs 30 inspections per second, and if blur remains, to shorten exposure and increase strobe intensity. Fix the capture first. A blurred image carries less information, and no calibration map puts it back. If a faster line still produces blur after the optics and lighting are fixed, recalibration will not recover the lost accuracy.
Edge or central GPUs for per-line updates
Where the model runs affects how practical each tier is. Whichever you pick, log every inference result with its image, score and decision: that log is what makes drift detection and retraining possible later. For the hardware itself, LynxBenchAI publishes measured throughput, latency, memory and energy results across accelerators. They are a starting point for shortlisting edge or server hardware, before your own model is validated on the device you choose.
Can we retrain the classifier on our latest labelled images without starting from scratch?
Yes. One multifaceted study of fine-tuning found that on a very scarce dataset, fine-tuning a pre-trained model still reached significant performance while training from scratch did not converge. Whether fine-tuning is worth doing now, and safe to deploy, depends on three things: how many new labels each class has, how you fold them in, and whether you check calibration again for every defect class.
TechnoLynx scopes this kind of retraining in a set order. TechnoLynx first fixes the acceptable regression on existing defect classes. It then assesses the volume and class balance of the new labels and recommends a fine-tune, a full retrain or a threshold-only update. The updated model is deployed with a rollback path in case regression testing catches a problem.
How much new labelled data before retraining is worth it?
Enough per class to move validation metrics further than the test set’s noise. Below that, keep collecting or update only the affected class head and its threshold. Published figures differ between sources. A camera-trap study found that improvement slowed after 150 images per class and was not substantial beyond 500, and that tuning only the last layer was competitive with deep tuning. A vendor checklist for industrial vision gives 500 to 2,000 labelled images per defect class as a working range, with at least 100 to 200 for rare classes. It stresses that diversity across lighting, batches and orientations matters more than raw count. Fine-tuning also gains less over training from scratch as the number of examples grows, and it works better when the source data is similar to the target. A model moving between near-identical lines is the favourable case.
If new labels come from different inspectors or shifts, measure their agreement before training. The same checklist sets a target of Cohen’s kappa or IoU of at least 0.85 between annotators.
Will retraining break calibration that currently works for other defect classes?
It can, and that is why regression testing has to cover the whole defect taxonomy, not just the classes in the new batch. Fine-tuning a well-trained detector on new data alone leads to catastrophic forgetting. In one CVPR 2022 experiment with an FCOS detector, plain fine-tuning reached 16.7% AP against an upper bound of 38.5%. Ultralytics’ documentation calls merging original-class examples into the fine-tuning set “the only reliable way to prevent forgetting”.
Thresholds move too. Retraining tends to change output scores, which breaks downstream users of the model. One remedy is to calibrate each release to a fixed score contract, so the calibrated thresholds keep their meaning even when raw-score thresholds shift. Research on calibrating continually updated models finds no guarantee that a model calibrated on new data stays calibrated on earlier distributions. It proposes calibrating on a replay buffer of earlier validation data. For detectors, a few epochs of fine-tuning with a calibration loss improved calibration error without sacrificing AP and without retraining from scratch. In practice, a retrain is not finished until every existing class has passed its threshold again on the recalibrated model.
Can a handful of labelled images adapt the model to a new line’s camera?
Sometimes, but the evidence here comes from digit-recognition benchmarks, not production lines. Few-shot supervised domain adaptation has worked with one labelled target sample per category on digit-recognition benchmarks, with accuracy rising as more samples per class were added. When target data is that scarce, freezing layers reduces overfitting. Ultralytics recommends training only the detection head for very small datasets, and one fine-tuning study found that fine-tuning only the high layers gave satisfactory results when training data was small. A caution applies: full fine-tuning can underperform linear probing on out-of-distribution data with large shifts.
A handful of images from a new camera is rarely enough to prove that a retrained model clears a per-class threshold with a confidence bound, given the test sizes discussed above.
Who can set up per-line qualification and retraining for our inspection model?
TechnoLynx can build the measurement and update loop that decides, line by line, whether your inspection model is requalified, recalibrated or retrained. The TechnoLynx engagement starts by defining the metric and acceptance threshold for each defect class against the pilot baseline. It then runs a measurement protocol across each line’s camera angles and conveyor speeds, and implements the recalibration or retraining pipeline the results trigger, with validation on every line and ongoing monitoring against the threshold. This sits within TechnoLynx’s Computer Vision R&D work and its broader AI engineering for manufacturing and automotive.
Some teams will not need us. A team that already owns its training pipeline, a golden dataset and documented change control has the pieces to run this decision itself. If the open question is only which accelerator to buy, published benchmark results are a cheaper first step than an engineering engagement.
The threshold is what you decide before measuring
A threshold agreed after seeing Line 4’s numbers is no longer an independent test of Line 4. Retraining triggers need documented criteria with quality engineering sign-off: the iFactory checklist warns that uncontrolled retraining produces model churn. The hardest threshold to set is the one for the rarest defect class, where the test set is smallest. That is where the confidence bound earns its place.
FAQ
Why do we need a validation set?
Post-hoc calibration methods such as temperature scaling need a hold-out validation set to fit their parameter. A separate test set then tells you whether the model meets its acceptance criteria without the result being biased by tuning.
How should defect images be labelled for retraining?
Have two or more annotators label the same images and measure their agreement before training. One industrial checklist targets kappa or IoU of 0.85 or more and calls for guideline revision below 0.75.
