The model did not get worse — the data changed A computer vision system that performed reliably for three months starts producing more false positives. The engineering team’s first response is to check the model. Is the binary corrupted? Did a deployment update go wrong? Was there a silent configuration change? Almost always, the model is bit-for-bit identical to the one that was performing well a quarter ago. What changed is the data — the images arriving at the model’s input are no longer drawn from the same distribution the model was trained and validated on. This pattern — stable model, shifting data, degrading performance — is the dominant failure mode for production computer vision systems. It is also the most under-monitored. Most CV deployment teams invest heavily in model evaluation at deployment time and almost nothing in data monitoring after. The model is treated as the intelligent component that might fail. The data is treated as a passive input assumed to be stable. That assumption is almost always wrong, and it is the source of a recurring pattern we see in post-deployment audits: teams reach for hyperparameters, architectures, and training recipes when the actual fault is sitting one step upstream, in the pixels entering the network. The published survey from Sambasivan et al. (2021), “Everyone wants to do the model work, not the data work”, documented that data cascades — compounding data quality issues across the lifecycle — affected 92% of surveyed AI practitioners. That is a published-survey figure, not a benchmark of any specific deployment, but the direction matches what we see in our own engagements: data work is the largest source of unmeasured risk in operational CV systems. Which data-quality problems most often cause CV systems to degrade after deployment? Four failure classes account for the bulk of post-deployment degradation we encounter, and they tend to compound rather than appear in isolation: Failure class What happens Where to look first Annotation inconsistency Labellers disagree; the model learns the disagreement as noise on boundary cases Inter-annotator agreement on the most recent labelling batches Domain shift Training conditions diverge from production (lens, lighting, product mix) Image statistics and feature activations vs. training reference Data drift Many small environmental changes accumulate over weeks or months Distributional distance (KL, PSI) at preprocessing output Concept shift The decision boundary itself changes — what counts as a defect changes Label policy reviews, downstream quality-team criteria These are observed-pattern classes — categories we see repeatedly across engagements, not a ranked study. The point of the table is to give a triage order: before touching the model, walk these four classes in sequence and quantify each one. The cost of doing this badly is high, because each class has a different remediation path, and treating drift as concept shift (or vice versa) wastes a retraining cycle. Why does annotation inconsistency set an invisible ceiling? The quality ceiling of any supervised computer vision model is set by the quality of its training labels. If two annotators examine the same image and disagree on whether it contains a defect — or on the defect boundary, or on the defect classification — the model learns that disagreement. The result is a model whose behaviour in ambiguous cases reflects the noise in the labelling process rather than a coherent decision criterion. Inter-annotator agreement is measurable. Cohen’s kappa for two labellers and Fleiss’ kappa for multiple labellers are standard. Both are rarely measured in practice. Across CV annotation engagements, we have reviewed pipelines where three annotators produced agreement rates below 70% on boundary cases — an observed-pattern from our work, not a benchmarked industry rate. The model was being trained on data where the “ground truth” was effectively a coin flip for nearly a third of difficult examples. Held-out accuracy looked acceptable because the easy cases dominated the metric, but performance on the boundary cases — the ones that actually matter to the downstream quality team — was close to random. The fix is not more annotations; it is better annotation protocols. Explicit criteria for boundary cases (at what size does a scratch become a defect, what level of discolouration counts as contamination, where exactly is the boundary of an anomalous region), calibration exercises where annotators align on edge cases before production labelling begins, and ongoing agreement monitoring that flags drift in annotator behaviour over time. These are data engineering tasks, not ML engineering tasks. They determine the model’s performance ceiling more than any architectural choice between, say, a YOLO variant and a transformer detector. Domain shift: training conditions are not production conditions Domain shift occurs when the production environment differs systematically from the training environment. The model learned features optimised for the training distribution — specific lighting conditions, camera angles, background characteristics, product appearances — and those features transfer imperfectly when the distribution moves along any of these axes. The sources of domain shift in production CV are predictable enough to enumerate: Camera and optics changes. A lens replacement, a camera firmware update, a cleaning schedule change, or physical repositioning changes image characteristics in ways that may be invisible to a human reviewer but measurable in image statistics. A ResNet or EfficientNet backbone trained against one lens distortion profile will produce different feature activations after a lens swap, even when the human-visible content looks identical. The PyTorch or TensorRT inference graph runs exactly as it did; the inputs no longer match. Lighting degradation. Industrial lighting degrades over time — bulb output decreases, colour temperature shifts, reflector efficiency drops. The change is gradual enough that operators do not notice it, but pixel-intensity histograms move measurably. A model calibrated under fresh lighting experiences a slow accuracy drift that may not cross any alert threshold until it has accumulated enough to affect throughput. Product evolution. In retail and manufacturing, the items being inspected change — new packaging designs, new variants, seasonal mixes. Each change introduces visual characteristics the model may not have seen during training. The off-the-shelf model failure patterns are particularly acute here: a model trained on last quarter’s product mix may fail systematically on this quarter’s new variant. Domain shift is distinct from drift in one important way: it usually has a single, locatable cause. A lens was changed. A line was reconfigured. A new SKU was introduced. If you can name the change and pin its date, you are looking at domain shift, not drift, and the remediation is targeted rather than statistical. What is data drift in computer vision, and how do you distinguish it from concept shift? Data drift is the gradual change in the production data distribution over time, without a single identifiable cause. It is the accumulation of small environmental changes — lighting aging, camera micro-shifts, seasonal variations, process parameter changes — that collectively move production data away from the training distribution. The challenge with drift is that no single change triggers an alert. Each individual shift is within tolerance. The cumulative effect crosses a threshold only after weeks or months of gradual movement, at which point performance has already declined without any monitoring signal pinpointing when the decline began. Concept shift is a different animal. With drift, the inputs change but the right answer for a given input stays the same. With concept shift, the mapping changes: an image that should have been flagged as a defect under last quarter’s quality policy is now considered acceptable, or the reverse. Concept shift typically originates outside the CV system — a customer specification changes, a regulatory tolerance tightens, a downstream quality team redefines a category. Retraining on the original labels will not help, because the labels themselves are out of date. The practical test: if you re-annotate a recent production sample under the current policy and the model’s predictions still disagree with the new labels, you are dealing with drift (or concept shift compounded with drift). If the model’s predictions agree with the old labels but disagree with the new labels on the same images, you are looking at concept shift, and the first fix is a labelling policy update, not a retraining run. Detecting drift requires statistical monitoring at the boundary where it is most measurable. Track pixel intensity distributions, feature activation distributions from an intermediate layer, and preprocessing output statistics against a reference baseline captured at training time. KL divergence, Population Stability Index, and Kolmogorov–Smirnov tests on per-channel summaries are the usual instruments. Our recommendation is to place drift detection at the pipeline’s preprocessing stage, where distributional change is cleanly observable before it propagates into the model and where the runtime cost is low enough to run continuously. The feedback loop that most teams skip The standard CV deployment lifecycle is: collect data, label data, train model, evaluate, deploy, monitor accuracy. The piece that is usually missing is the feedback loop — routing production failures back into the training pipeline as new training data. Production failures are the most valuable training data the system produces. False positives reviewed and corrected by human operators, false negatives discovered through downstream quality checks, edge cases flagged for review — these are exactly the cases where the model is weakest, in the exact conditions where the model operates. Incorporating them into the training pipeline, with the same annotation quality controls used for the original dataset, produces a model that improves specifically where it is failing rather than uniformly across all cases. The infrastructure for this loop is non-trivial: a capture mechanism for production failures, a labelling pipeline with quality controls (and ideally inter-annotator agreement tracking), a retraining schedule that incorporates the new data without regressing on cases the model already handles correctly, and a shadow-deployment stage that compares the candidate model against the incumbent on live data before promotion. MLflow or a similar experiment-tracking system is usually enough to manage the retraining metadata; the harder part is the organisational discipline to actually feed the loop. The alternative — retraining on the original dataset whenever performance degrades — produces a model perpetually optimised for the past rather than adapted to the present. It looks like progress because each retraining cycle restores some accuracy on the original held-out set, but it solves the wrong problem. Data quality remediation runbook When production CV accuracy degrades, work these four steps before modifying the model architecture or hyperparameters. Detect — identify data quality degradation signals. Monitor input distribution statistics (pixel intensity histograms, feature activation distributions from an intermediate layer) against training-time baselines using KL divergence or Population Stability Index. Track inter-annotator agreement (Cohen’s kappa) on incoming labelling batches and alert when agreement drops below your documented threshold. Log the rate of production failures — false positives corrected by operators, false negatives discovered downstream — and alert when the rate exceeds the post-deployment baseline. Confirm — verify the root cause is data, not model. Confirm the deployed model binary, configuration, and preprocessing pipeline are identical to the validated version. Rule out corrupted weights, configuration drift, or firmware changes. Inspect recent camera, lighting, and environmental conditions for known domain-shift sources: lens replacements, lighting degradation, product mix changes, camera repositioning. Compare feature activation distributions from current production data against the training reference at the preprocessing output to quantify distributional distance. Remediate — apply targeted data quality interventions. For annotation inconsistency: run a calibration exercise where annotators re-align on boundary cases using explicit criteria (defect size thresholds, discolouration levels, region boundaries) before resuming production labelling. For domain shift: capture a representative sample of current production images, annotate them with quality-controlled labels, and add them to the training set to cover the shifted distribution. For data drift: update the reference baseline statistics to reflect the current validated production distribution, and recalibrate monitoring thresholds so alerts reflect the new operating conditions. For concept shift: update the labelling policy first, re-label a recent production sample under the new policy, and only then proceed to retraining. Retrain — execute a controlled retraining cycle with validation gates. Combine the original training dataset with remediated production data, ensuring failure cases routed through the feedback loop are included with quality-controlled annotations. Validate the retrained model against both the original held-out test set and a held-out sample of recent production data. It must not regress on cases it previously handled correctly. Deploy behind a shadow evaluation period: run the candidate in parallel with the current model, compare outputs on live data, and promote only after the candidate meets accuracy thresholds on both historical and current distributions. Building data quality into the deployment, not after it Data quality is not a pre-deployment task that can be checked off and forgotten. It is an ongoing operational concern that requires monitoring infrastructure, annotation quality processes, and feedback loops that persist for the lifetime of the production system. The data readiness assessment before deployment establishes the baseline: is the training data representative of the production environment, is the annotation quality sufficient, is the class distribution reflective of production conditions. The monitoring infrastructure after deployment tracks drift from that baseline. The feedback loop continuously improves the baseline as the production environment evolves. Each of the three is cheap relative to the cost of an unexplained accuracy decline in a deployed system; together they form the smallest sufficient operational structure for keeping a CV system honest. If your computer vision system is experiencing accuracy degradation after deployment and the root-cause investigation has focused on the model rather than the data, a Production CV Readiness Assessment includes data quality diagnostics — annotation consistency analysis, distribution shift measurement, and feedback loop design — as core components. FAQ Which data-quality problems most often cause CV systems to degrade after deployment? Four classes dominate: annotation inconsistency (labeller disagreement learned as noise), domain shift (training conditions diverging from production along axes like optics, lighting, or product mix), data drift (gradual cumulative distribution change with no single cause), and concept shift (the label policy itself changing). They tend to compound, which is why triage walks all four before any retraining decision. What is data drift in computer vision, and how do I detect it before users do? Data drift is the slow movement of the production input distribution away from the training distribution, driven by many small environmental changes rather than a single event. Detect it by tracking pixel intensity distributions and intermediate feature activations against a training-time reference using KL divergence or Population Stability Index, evaluated continuously at the preprocessing output rather than only at scheduled audits. How do I distinguish data drift from concept shift, and why does it matter for retraining? If you re-annotate recent production images under the current labelling policy and the new labels still disagree with the model, you are seeing drift. If the model agrees with the old labels but disagrees with the new labels on the same images, you are seeing concept shift. The distinction matters because drift is fixed by retraining on new data with existing labels, whereas concept shift requires a labelling-policy update first — retraining without it just re-learns the obsolete decision boundary. What annotation-quality issues silently corrupt CV training pipelines? The main one is inter-annotator disagreement on boundary cases — ambiguous defect sizes, borderline discolouration, edge regions of segmentation masks. Without explicit criteria and calibration exercises, the model is trained on contradictory ground truth and reports inflated accuracy because easy cases dominate the metric while boundary cases approach random. Cohen’s kappa or Fleiss’ kappa, measured on incoming batches, is the standard early-warning signal. How do I monitor live image distributions to flag drift early? Place statistical monitoring at the preprocessing stage of the pipeline, where distributional change is observable before it reaches the model. Track per-channel pixel statistics and feature activations from an intermediate layer against a reference captured at training time, and use KL divergence, Population Stability Index, or a Kolmogorov–Smirnov test to score the distance. Set thresholds informed by your operating tolerance, not by default values. What ongoing data-quality framework keeps a deployed CV system healthy? Three components, all persistent: a pre-deployment data readiness assessment that fixes a representative baseline, post-deployment distributional monitoring against that baseline at the preprocessing stage, and a feedback loop that routes production failures back into a quality-controlled labelling and retraining pipeline. Shadow-evaluate any retrained model against both historical and current production data before promotion.
