A drift alert on a deployed defect-detection model is a reason to investigate. It is not an instruction to retrain. Drift detection in machine learning tells you that the inputs have changed, not whether the change matters: input distributions can shift while predictions stay stable. A practical guide to drift retraining puts it bluntly: treat drift alerts as investigation triggers, and check pipelines, schemas, labels, instrumentation and metric definitions before concluding that the model has degraded. A reliable retraining trigger needs more than a single aggregate drift score: segmented monitoring, staged rollout with holdout validation, and a canary evaluation before any retrain.
Concept drift vs data drift: which one justifies a retrain?
| Data drift (covariate shift) means the distribution of the inputs has changed compared with the training data, while the relationship between inputs and the target may be unchanged. In the notation Seldon’s Alibi Detect documentation uses, covariate drift is P(X) moving away from the reference while P(Y | X) stays the same, and concept drift is P(Y | X) itself changing, so the model may no longer approximate the true process. Alibi Detect also defines a third type, prior or label drift, where the output distribution P(Y) shifts. On an inspection line, a shift in defect rates is a shift in P(Y). |
Concept drift is the one that can justify a retrain, because the model’s learned mapping may no longer fit the process. Data drift justifies a retrain only when it pushes inputs into regions where the model’s performance actually drops. The same production guide notes that input distributions can change in features the model barely uses, in regions where predictions stay stable, or because of an expected business event, and that teams should check feature importance, prediction behaviour, cohorts and eventual outcomes before deciding on a retrain.
The most dangerous drift, concept drift that leaves the inputs unchanged, is invisible to unsupervised monitoring. A study of proxy monitoring on IEEE-CIS fraud-detection data found that covariate drift was detected in all five monitoring windows, while a simulated concept drift, made by flipping fraud labels (which also lowered the fraud rate), with an unchanged input distribution was detected in none, with every proxy reading identical to the baseline. The author describes this as a structural limit rather than a tuning problem: whether drift can be detected depends on whether it shows up in P(X) at all. The study recommends that governance assume concept drift with constant inputs will go undetected and compensate with periodic ground-truth checks. The data is from fraud, not inspection, but the logic carries over to any model whose inputs can look normal while the labels move.
That leads to the rule this whole page depends on. An unlabelled input-drift signal is a prompt to investigate. It cannot confirm degradation without labels. Sethi and Kantardzic’s MD3 work handles this by requesting labelled samples only when an unlabelled signal fires, and it counts a change as a false alarm, with no retraining, when those labels show no significant drop in performance.
How do I tell the pipeline “this is an expected new variant, don’t alarm” from real drift?
Give each new variant a small labelled holdout of its own and judge it against acceptance thresholds. Do not try to infer the answer from the drift score. A new SKU counts as expected shift when the model’s accuracy and false-negative rate on that variant’s holdout sit within the thresholds you agreed. If they do not, the variant becomes a candidate for real degradation, which means investigation first and possibly a retrain.
In the “Trimming the Risk” study of continuous training for deep-learning inspection systems, domain experts saw two changes on one line: ambient lighting caused data shifts because the camera exposure settings were inadequate, and a new defect was introduced that caused a dramatic drift in the input data. On another system, a slightly damaged ink dispatcher produced smudged prints, which was a new and difficult type of defect. The lighting change is a nuisance shift that the model may tolerate. The ink fault introduced a new type of defect. Both raise an input-drift score, and the drift score alone does not separate them.
Reference data is not always clean either. The Kaputt defect-detection dataset caps reference images at three per item, because most retail items sell infrequently and a perfect reference database is impractical, so some reference images show different packaging.
Stage new variants before they enter the training set
A new variant should go through a staged rollout. Evaluate the model on that variant’s holdout first, and fold its data into the main training distribution only after it passes. The order matters because retraining on drifted data has damaged inspection models. The inspection study warns that self-generated labels can cause silent performance degradation. It also found that continuous training on drifted data, even with ground-truth labels, does not guarantee a usable retrained model, because the pipeline’s steps and hyperparameters were tuned on in-distribution data. In one of its cases, vanilla continuous training on all recent production data caused catastrophic forgetting, with validation F1 falling by more than 13%. A filtered approach that fine-tuned only on inputs close to the training distribution improved F1 on both the original and the production data.
The study also names a limit of supervised inspection models: across successive training cycles, the model could not learn new defect patterns that were unlike the in-distribution ones. A truly new defect class needs human verification. Repeated training cycles with its approach did not teach the model such patterns.
The staged route has real costs. Each variant and lighting condition needs its own annotated holdout, which is a recurring labelling bill. Some boundary cases, such as a lighting change that coincides with a real sensor fault, will still need a person to look at them.
Per-variant and per-lighting slices catch what the aggregate hides
An aggregate drift or accuracy metric can look healthy while one variant fails, because gains elsewhere in the distribution offset the local loss. The production guide warns that a model with stable overall accuracy can degrade badly in a minority segment, and it recommends segmenting error rates by business-critical dimensions such as product type. In medical imaging, Oakden-Rayner and colleagues found that this kind of hidden stratification can produce relative performance differences of over 20% on important subsets. Google’s ML Test Score paper describes a team that had no way to detect whether its global service was serving poor predictions in a single country.
For defect detection, the slices that matter are variant × lighting condition, with line or station added where cameras differ. DriftLens shows how to do this on embeddings: it models reference distributions per batch and per predicted label, uses the per-batch distance to detect whether a window has drifted, and uses the per-label distances to show which labels are most affected. Segment-level results should inform retrain decisions. Aggregate metrics are for fleet health summaries. The production guide adds a useful distinction: a global decline with adequate labels and clean data is strong evidence of drift, while a decline concentrated in one cohort calls for checking that cohort’s data pipeline before any retraining. If you want to apply the same slicing before deployment, our piece on evaluating models across scenario buckets covers that side.
Slicing has a downside. More segments means more metrics, and a new SKU starts with very little data, so small-sample segments produce noisy alarms. The production guide asks for enough observations in the evaluation window for a metric to be statistically reliable before anyone declares a performance loss. Our position is that production-scale vision fails in ways small pilots never expose: severe class imbalance across thousands of SKUs, visually similar products, and a steady flow of unknown objects that off-the-shelf models misclassify without flagging.
What baseline should a defect model use when it has just gone live?
With no production history, build the baseline from a held-out, representative slice of the training and validation data. Switch to a rolling production baseline once enough stable production data has built up.
TechnoLynx builds this layer for inspection teams as the Production AI Monitoring Harness: agreeing what “normal” means for the model and line, choosing the baseline, drift statistics and thresholds, wiring alert routing to retrain triggers, and backtesting the thresholds and the shadow-canary gate before they run in production. A TechnoLynx engagement starts by framing what normal means operationally for the model.
Vendor and tool documentation points to the same starting point. AWS’s Machine Learning Lens recommends a reference dataset taken from the training data or a representative subset that performed well during validation. Azure Machine Learning recommends training data as the baseline for data drift and validation data as the baseline for prediction drift. Alibi Detect draws two disjoint sets from the reference distribution, one to train preprocessing and a held-out reference set to initialise the detector, so the detector does not become overly discriminative. DriftLens uses the training set as the baseline and the test set, or windows assumed to be drift-free, to estimate thresholds.
Once the line has run cleanly for a while, move the reference to production data. The production guide suggests establishing the performance baseline over a 15 to 30 day stable production window that reflects typical conditions, not peaks or anomalous weeks. Azure allows either training data or recent past production data as the reference. One caveat: a validation-set baseline was not captured on the line’s own cameras, so treat early alerts with extra scrutiny.
Which alert thresholds can you defend for a first deployment?
Start conservative, tie thresholds to a distance measure calibrated on the baseline, and treat any published number as a starting point. Practitioner sources quote PSI above 0.2 as typically meaningful, to be tuned to the domain and to feature importance. The production guide cites PSI above 0.25 and an accuracy drop above 2% as starting points, not universal rules, and says thresholds must sit above the metric’s normal noise floor. For image models, where per-pixel PSI means little, DriftLens shows a more principled route: sample many drift-free windows, compute their distances to the baseline, and set the threshold at the largest distance after trimming outliers. Its sensitivity parameter controls the trade-off directly: higher sensitivity lowers the threshold and raises both drift sensitivity and false alarms, and the authors default to discarding the top 1% of distances.
Two further habits stop thresholds from firing on noise. Alert on sustained divergence across several monitoring windows, not on a single spike. Alert on rate of change as well as level, because a metric falling 0.5% a week for four weeks is a different signal from a single 2% drop.
Worked example: a tiered escalation matrix
The production guide sets out a severity matrix for escalation. Re-derive its percentages from your own baseline noise before adopting them.
| Severity | Condition | Response | Owner | Timeframe |
|---|---|---|---|---|
| Green | All metrics within expected range | No action | Monitoring system | Ongoing |
| Yellow | Performance down 1-3% from baseline, or a proxy signal over threshold | Investigate; do not retrain yet | ML engineer | Within 48 hours |
| Orange | Performance down 3-5%, or a proxy over threshold for 5+ days | Escalate; start root-cause analysis | ML engineer and tech lead | Within 24 hours |
| Red | Performance down 5% or more, or outside the acceptable range for a regulated use | Escalate immediately; begin incident response |
Note that no tier says “retrain”. Retraining is a decision made after root-cause analysis.
Who on the line should receive drift alerts?
Route alerts to the people who can act on them: the ML engineer from yellow, with the tech lead brought in at orange. Avoid broad distribution lists. The guide’s ownership rule is that automated alerting does not replace human accountability, and it pairs daily automated checks with a weekly human review of segment-level trends. A canary playbook leaves the decision on a critical alert to the model owner, after reviewing sampled cases.
Shadow and canary gating before a drift alert can trigger a retrain
Shadow and canary are different tools. In a canary, a small share of live traffic goes to the new model and users see its predictions. In shadow mode, the new model gets a copy of live traffic and its predictions are logged for comparison but never used. MLflow’s deployment guidance recommends shadow over canary when ground truth is delayed, because a canary needs fast feedback to be useful. In inspection, ground truth depends on human verification; the sequence below runs shadow first and the canary after it.
The same guidance warns that noisy retrain triggers are the fastest way to burn trust in an automated pipeline. It recommends requiring sustained deviation across multiple windows and tying the trigger to ground-truth sampling wherever possible. The full sequence, put together from these sources:
- A drift signal holds across several consecutive monitoring windows, for example three.
- Triage rules out pipeline, schema, label and instrumentation faults, and finds which segment holds the change.
- A labelled check confirms that performance has dropped beyond the documented threshold, not just a proxy.
- The retrained candidate passes the same validation gates as any new model, with no shortcuts because the retrain was triggered automatically.
- The candidate runs in shadow against live inputs.
- A limited canary follows, for example 0.5-1% of traffic for 24-72 hours and then 5-25% before full rollout.
- A kill threshold stops canary traffic and routes back to the previous stable version.
The production guide works through this end to end for a transaction-risk model. A warning on newly onboarded merchants was traced to one cohort. Delayed outcomes confirmed more false negatives. A retrain on a versioned dataset was evaluated on the affected cohort and on temporal holdout data. The candidate then ran in shadow before getting limited canary traffic. The domain is different, but the order is the same for a new defect on one product line. Shadow and canary both add time to the promotion cycle and need enough samples for statistical power. A low-volume variant may need a longer window.
Who can build drift monitoring and retrain triggers for our inspection fleet?
TechnoLynx builds drift monitoring for deployed defect-detection models: per-variant and per-lighting segmentation, a baseline and thresholds the team can defend, alert routing to named owners, and the shadow and canary gate that a retrain must pass. When choosing a partner, ask whether they will leave you with documented thresholds and runbooks your own engineers can retune. The engagement is TechnoLynx’s Production AI Monitoring Harness, part of its wider AI engineering for manufacturing and automotive work.
The engagement covers monitoring and retrain gating.
Where labelled checks remain non-negotiable
Segmenting, baselining and gating all reduce wasted retrains. None of them removes the blind spot described earlier. Concept drift that leaves the inputs looking normal will pass every distribution check, and a supervised inspection model will not learn a defect that does not resemble anything it was trained on. The question each team has to settle is how often to pay for a labelled sample per segment, given the cost of a missed defect on that product. The answer will differ between a cosmetic scuff on packaging and a structural crack, and the labelling schedule should reflect that difference.
