The model did not get worse — the data changed A computer vision system that performed reliably for three months starts producing more false positives. The engineering team’s first response: check the model. Is it corrupted? Did an update go wrong? Was there a configuration change? Usually, the model is identical to the one that was performing well. What changed is the data — the images arriving at the model’s input are no longer drawn from the same distribution that the model was trained and validated on. This pattern — stable model, shifting data, degrading performance — is the dominant failure mode for production computer vision systems. It is also the most under-monitored, because most CV deployment teams invest heavily in model evaluation at deployment time and minimally in data monitoring after deployment. The model is treated as the intelligent component that might fail; the data is treated as a passive input that is assumed to be stable. That assumption is almost always wrong. A 2022 Google Research study found that data quality issues account for more model failures in production than algorithmic limitations. As reported in Sambasivan et al. (2021, ‘Everyone wants to do the model work, not the data work’), the published survey reports documented that data cascades — compounding data quality issues — affected 92% of surveyed AI practitioners. Why does annotation inconsistency set an invisible ceiling? The quality ceiling of any supervised computer vision model is set by the quality of its training labels. If two annotators examine the same image and disagree on whether it contains a defect — or on the defect boundary, or on the defect classification — the model learns that disagreement. The result is a model whose behaviour in ambiguous cases reflects the noise in the labelling process rather than a coherent decision criterion. Inter-annotator agreement is measurable (Cohen’s kappa, Fleiss’ kappa for multiple annotators) but rarely measured in practice. In our experience across CV annotation engagements, we have reviewed annotation pipelines where three annotators produced agreement rates below 70% on boundary cases (an observed pattern, not a benchmarked industry rate) — meaning the model was being trained on data where the “ground truth” was effectively a coin flip for nearly a third of difficult examples. The model’s reported accuracy on a held-out set reflected this noise: high accuracy on easy cases, near-random performance on boundary cases, and an overall metric that looked acceptable but masked a systematic weakness. The fix is not more annotations — it is better annotation protocols. Explicit criteria for boundary cases (at what size does a scratch become a defect? what level of discolouration counts as contamination? where exactly is the boundary of an anomalous region?), calibration exercises where annotators align on edge cases before production labelling begins, and ongoing agreement monitoring that flags drift in annotator behaviour over time. These are data engineering tasks, not ML engineering tasks — and they determine the model’s performance ceiling more than any architectural choice. Domain shift: training conditions ≠ production conditions Domain shift occurs when the production environment differs systematically from the training environment. The model learned features optimised for the training distribution — specific lighting conditions, camera angles, background characteristics, product appearances — and those features transfer imperfectly to a distribution that differs along any of these dimensions. The sources of domain shift in production computer vision are predictable: Camera and optics changes. A lens replacement, a camera firmware update, a cleaning schedule change, or physical repositioning of the camera system changes the image characteristics in ways that may be invisible to human inspection but measurable in the image statistics that the model relies on. A ResNet trained on images with one lens distortion profile will produce different feature activations when the lens is replaced, even if the human-visible content is identical. According to a 2023 survey by Cognilytica, over 80% of production AI system failures trace back to data pipeline issues rather than model architecture limitations. Lighting degradation. Industrial lighting degrades over time — bulb output decreases, colour temperature shifts, and reflector efficiency drops. The degradation is gradual enough that human operators may not notice it, but the statistical properties of the images change measurably. A model calibrated under fresh lighting will experience a slow accuracy drift as the lighting ages, and the drift may not cross an alert threshold until it has accumulated enough to affect production outcomes. Product evolution. In retail and manufacturing environments, the products being inspected change over time — new packaging designs, new product variants, seasonal product mixes. Each change introduces visual characteristics that the model may not have seen during training. The off-the-shelf model failure patterns are particularly acute here: a model trained on last quarter’s product mix may fail on this quarter’s new variant. Data drift: the slow degradation Data drift is the gradual change in the production data distribution over time, without a single identifiable cause. It is the accumulation of small environmental changes — lighting aging, camera positioning micro-shifts, seasonal variations, process parameter changes in manufacturing — that collectively shift the production data away from the training distribution. The challenge with data drift is that no single change triggers an alert. Each individual shift is within tolerance. The cumulative effect crosses a threshold only after weeks or months of gradual degradation — at which point the model’s production performance may have declined significantly without any single monitoring signal indicating when the decline began. Detecting data drift requires statistical monitoring of the production data distribution: tracking the statistical properties of the model’s input data (pixel intensity distributions, feature activation distributions, preprocessing output statistics) against reference baselines from the training data. Our recommendation is to implement drift detection at the pipeline’s preprocessing stage where distribution shifts are most measurable, using statistical tests (KL divergence, Population Stability Index, or simpler distributional comparisons) that flag when the production distribution has moved beyond a documented tolerance from the training reference. The feedback loop that most teams skip The standard CV deployment lifecycle is: collect data → label data → train model → evaluate → deploy → monitor accuracy. What is usually missing is the feedback loop: route production failures back to the training pipeline as new training data. Production failures — false positives reviewed and corrected by human operators, false negatives discovered through downstream quality checks, edge cases flagged for review — are the most valuable training data the system produces. They represent exactly the cases where the model is weakest, in the exact conditions where the model operates. Incorporating these cases into the training pipeline (with appropriate annotation quality controls) produces a model that improves specifically in the areas where it is failing. This feedback loop requires infrastructure: a mechanism to capture production failures, a pipeline to label them with quality-controlled annotations, and a retraining schedule that incorporates the new data without losing performance on cases the model already handles well. The infrastructure cost is non-trivial. The alternative — retraining on the original dataset whenever performance degrades — is a pattern that produces a model that is perpetually optimised for the past rather than adapted to the present. Data quality remediation runbook When production CV accuracy degrades, follow these four steps before modifying the model architecture or hyperparameters. Detect — identify data quality degradation signals. Monitor input distribution statistics (pixel intensity histograms, feature activation distributions) against training-time baselines using KL divergence or Population Stability Index. Track inter-annotator agreement metrics (Cohen’s kappa) on incoming labelling batches and flag when agreement drops below your documented threshold. Log the rate of production failures — false positives corrected by human operators and false negatives discovered through downstream quality checks — and alert when the rate exceeds the post-deployment baseline. Confirm — verify the root cause is data, not model. Confirm the deployed model binary, configuration, and preprocessing pipeline are identical to the validated version — rule out corrupted weights, configuration drift, or firmware changes. Inspect recent camera, lighting, and environmental conditions for known domain shift sources: lens replacements, lighting degradation, product mix changes, or camera repositioning. Compare feature activation distributions from the current production data against the training reference at the pipeline’s preprocessing stage to quantify distributional distance. Remediate — apply targeted data quality interventions. For annotation inconsistency: run a calibration exercise where annotators re-align on boundary cases using explicit criteria (defect size thresholds, discolouration levels, region boundaries) before resuming production labelling. For domain shift: capture a representative sample of current production images, annotate them with quality-controlled labels, and add them to the training set to cover the shifted distribution. For data drift: update the reference baseline statistics to reflect the current validated production distribution, and recalibrate monitoring thresholds so alerts reflect the new operating conditions. Retrain — execute a controlled retraining cycle with validation gates. Combine the original training dataset with the remediated production data, ensuring production failure cases (false positives and false negatives routed through the feedback loop) are included with quality-controlled annotations. Validate the retrained model against both the original held-out test set and a held-out sample of recent production data — the model must not regress on cases it previously handled correctly. Deploy the retrained model behind a shadow evaluation period: run it in parallel with the current model, compare outputs on live production data, and promote only after the retrained model meets accuracy thresholds on both historical and current distributions. Building data quality into the deployment, not after it Data quality is not a pre-deployment task that can be checked off and forgotten. It is an ongoing operational concern that requires monitoring infrastructure, annotation quality processes, and feedback loops that persist for the lifetime of the production system. The data readiness assessment before deployment establishes the baseline: is the training data representative of the production environment, is the annotation quality sufficient, is the class distribution reflective of production conditions? The monitoring infrastructure after deployment tracks drift from that baseline. The feedback loop continuously improves the baseline as the production environment evolves. If your computer vision system is experiencing accuracy degradation after deployment and the root cause investigation has focused on the model rather than the data, a Production CV Readiness Assessment includes data quality diagnostics — annotation consistency analysis, distribution shift measurement, and feedback loop design — as core components.
