Computer vision turns pixels into an interpretation by learning, from labelled examples, which patterns of pixel values go with which answer, such as “good part”, “scratch” or “missing component”. In computer vision for manufacturing, that interpretation can fail when production images differ from the training images. In a 2024 study of deep learning inspection systems, a print-inspection model scoring 93.2% F1 in validation dropped to 45.7% F1 on production data, and a glass-bottle model fell from 93.3% to 76.6%. The production data had shifted, through on-site lighting and new defect types the training set did not contain.
That gap is the subject of this article.
From pixels to a pass/fail signal
Machine vision is the industrial systems discipline: imaging-based automatic inspection, process control and robot guidance, built by integrating cameras, lenses, lighting and software into a working line. Computer vision is the underlying method of automated image analysis, which machine vision combines with other technologies to inspect products and guide robots. The output can be a simple good-part/bad-part signal or a richer record of the identity, position and orientation of each object in the image. For the full comparison, see our piece on the difference between computer vision and machine vision.
The tasks themselves are familiar. Recognition says what is in the image, detection says where it is, segmentation marks which pixels belong to it, motion analysis follows it across frames, and scene reconstruction recovers 3D structure. For detection and recognition, the strongest algorithms are currently built on convolutional neural networks. Deep learning for computer vision also removed an old constraint: conventional machine vision needed the optics and lighting to produce a “simple” physical difference (dark defects on light parts), whereas deep learning can work where that difference cannot be engineered. The step-by-step mechanics are covered in how computer vision works step by step. What matters for a factory is the price of that flexibility. A deep learning model learns its defect patterns from labelled images, and those images often fail to cover every natural variation the line produces.
The validation-to-production gap in computer vision for manufacturing
The same drop from clean to shifted conditions appears in production data, in real-world benchmarks and under synthetic capture shifts.
| Study | What was measured | Result |
|---|---|---|
| Continuous-training study of two industrial inspection systems (2024) | Validation F1 vs production F1 | Print inspection 93.2% to 45.7%; glass bottles 93.3% to 76.6% |
| Logical anomaly detection under capture shifts (2026) | Image-level AUROC on clean images vs a random mix of brightness reduction and zoom-out | Normal-reference detectors lost 9.7 to 34.1 points; CSAD fell from 97.7% to 68.1%, SALAD from 96.4% to 62.3% |
| Baitieva et al., benchmark analysis (2025) | Eleven models, nine datasets, seven metrics | Models with 99.9% image-level AUROC on MVTec AD showed significant degradation on real-world data |
| RobustAD benchmark, CVPR workshop (2025) | 4,442 real images, 3 use cases, 6 real distribution shifts | All models did well in the source domain and degraded significantly on every target domain |
Two details in those results deserve attention. In the 2026 capture-shift study, the two detectors with the highest clean scores showed the largest degradation. And Baitieva et al. found several newer models underperforming older ones once evaluated off MVTec AD, which they read as the field being biased toward that one benchmark. A leaderboard position is evidence about the benchmark, not about your line.
This is the core of our position. Object detection and tracking are often treated as solved problems because open-source models exist for them, but real conditions and scale change the picture, and models that work in demos fail in production on edge cases, lighting and throughput. Throughput is the quiet one. In the 2020 normalizing-flow defect detection study, running YOLOv5 on every frame from four cameras was too slow on a modest computer without a GPU. Baitieva et al. likewise set aside diffusion-based detectors as impractical where rapid training and real-time inference are required.
Lighting and glare change what the camera sees
Cognex states that poor lighting is the most common cause of poor machine vision performance, and that no camera or software makes up for it. NI goes further: perhaps no other aspect of vision system design has caused more delays and cost overruns, partly because lighting was historically specified last, if at all.
How much light a part sends to the camera depends on its shape, reflectivity, texture and colour, so a curved specular surface responds to a directional light very differently from a flat diffuse one. Light reflecting off a smooth part straight into the lens creates glare, and a different light position can throw an important spot into shadow. In a 2020 factory-floor toolkit for reading gauges on legacy machines, direct reflections on gauge faces caused the images to be misinterpreted, and the team fitted anti-glare screens to fix it.
NI names overhead factory lights and sunlight as the most common ambient contributors, along with stray task lighting from neighbouring stations. In the 2024 continuous-training study, domain experts traced one production data shift to on-site ambient lighting and inadequate camera exposure settings. The RobustAD authors focus extensively on lighting as an important real-world variant: time of day changes colour temperature, brightness and shadows together, while synthetic lighting augmentation mostly varies only brightness and contrast.
The practical consequence is about sequencing. NI’s advice is to test two or three lighting options on the bench and then on the actual floor, and, when designing a station from scratch, to settle lighting first and build the inspection around it. We would add that the floor test should span the shifts and seasons the line actually runs through, because a single afternoon of images certifies a single afternoon of light.
Camera movement is a separate failure, not more lighting
Changes in camera positioning, such as mounting angle or position relative to the line, change the pose and size of objects in the image. In the 2020 gauge-reading toolkit, any disturbance to the camera position required a complete manual reset of the system. RobustAD found that scale, position and rotation changes produced the worst degradation, because most models are not inherently robust to them, and one method still suffered under pose changes despite training with affine augmentation.
The 2026 capture-shift study offers a reason why a correct part gets flagged. The authors hypothesise that, because its detectors compare each test image with normal images from one lighting setup and one camera position, a darker image or smaller object can score as anomalous even when every part is correct. The authors are clear that their brightness and zoom perturbations were synthetic and narrower than real capture changes, which also add sensor noise, uneven illumination and object shift.
The same study gives the bridge to monitoring. Where a detector keeps its ranking within each condition and only its scores move, a single threshold stops working, and monitoring lighting and camera placement with condition-specific thresholds can address the problem outside the detector. Where the detector loses discrimination within a condition, condition-specific thresholds cannot restore it.
Why does a model trained on thousands of images of our part still miss defects?
Manufacturing datasets tend to show more or less the same component over and over, with defects rare in a mature process, so a model trained on them can end up as brittle as a rules-based system, or more so. With a near-identical background and defects of the same type in the same place, the model is more likely to learn shortcuts that do not generalise to new defects, or that are not causally tied to the defect at all. Defective products typically make up only 1 to 5% of output, according to a source cited in a 2025 glass-inspection study, which leaves the defect class thin.
The 2023 study on robustness in manufacturing defect detection shows this failure staying hidden from validation. A classifier trained on uniform “mending plate” images performed almost perfectly on its validation set, yet failed on a holdout set prepared the same way and acquired on another day. A classifier trained on similar defects across a diverse range of parts scored lower on validation (0.925 against 0.995) but held that score on the holdout set, because the variety forced it to learn the defect rather than the background. The validation split could not reveal the problem, since a random split assumes the same distribution in every subset and no unseen data.
The authors also found that moving from a classifier to an object detector, which must localise the defect, improved performance, and that object detection with diverse data performed best. That is a useful design rule for inspection: buy diversity in the training set before buying more images of the same part, and prefer a task formulation that makes the model point at the defect. Augmentation helps only so far: RobustAD found augmentations useful only when they resemble the real test variation in kind and extent. How these architecture and training choices play out in deployment is covered in deep learning for image processing in production.
How labels shape what the model learns to see
Deep learning inspection systems derive defect patterns from labelled images. Labelling is expensive, and Baitieva et al. note that human operators can miss up to 25% of certain defect types, contaminating training data. In their noisy-label experiment, so-called unsupervised one-class models lost significant performance as the share of mislabelled images rose, and truly unsupervised methods remained too unreliable for industrial deployment. “No labels needed” is rarely true in practice: a one-class model still needs a training set that really is clean.
The 2020 normalizing-flow defect detection study points out that defects are infrequent and slow to gather, which weighs heavily on small and medium-sized manufacturers. The 2024 continuous-training study warns that retraining on self-generated labels can cause silent performance degradation. On its glass-bottle line, retraining naively on drifted production data with the model’s own labels cut validation F1 by more than 13%.
Accuracy and AP hide the expensive error
An inspection system trades throughput (fewer false alarms sent for manual checking) against the chance of shipping a defective part, and the missed defect is usually the more serious error. The false-positive rate still decides whether automation pays, because it sets how much manual work is actually avoided. Image-level AUROC, the metric most recent papers use, does not account for the relative importance of errors, even though misclassifying a defective part costs significantly more than a false alarm in production. The 2023 robustness study adds that AP may not be the right metric for inspection tasks.
Baitieva et al. use PG2, the share of good parts correctly passed when only 2% of bad parts are let through, as a measure of how much operator workload a model can remove at an acceptable miss level. Measuring and interpreting hardware performance numbers is a separate discipline again; here, the point is that a model’s headline score and its value on the line can point in different directions.
How can you tell a deployed computer vision model has started misreading the images it receives?
Neural networks always produce an output for any properly formatted input, even a meaningless one, and cannot by themselves tell in-distribution inputs from drifted ones. In the 2024 continuous-training study, confidence-based filtering accepted most production images even as false acceptance rose fivefold on one line and tenfold on the other, because the model stayed confident on unfamiliar data. A model that is wrong and confident looks healthy on a dashboard.
Detection therefore has to come from outside the model, on three fronts. The first is input drift against training: tests such as Kolmogorov-Smirnov or the Population Stability Index compare new data with the training distribution. A 2022 monitoring framework applied Kolmogorov-Smirnov variants and the Bhattacharyya coefficient to compare training-time and production values. High-resolution images are hard to compare directly, so in practice they are often reduced to semantic embeddings, for example with PCA or autoencoders, before similarity is measured. The second is capture conditions: monitoring lighting and camera placement lets condition-specific thresholds address score shifts outside the detector (see the capture-shift section above). The third is production performance against ground truth, which needs labels from sampled parts, since monitoring performance degradation in supervised inspection depends on ground-truth labels.
Drift alerts need interpreting, not obeying. In the 2022 framework’s supply-chain data, every Kolmogorov-Smirnov test was highly statistically significant, yet none corresponded to an operationally meaningful change in performance. The same authors note why image inspection needs monitoring all the same: training sets can be a fraction of a percent of the images a model will see in its lifetime, so production images often differ widely from training. Meanwhile, in the 2024 study, glass-bottle images flagged as drifted were classified correctly about as often as incorrectly. Our reading is that a drift signal should prompt sampling and labelling, and only a confirmed performance drop should prompt retraining.
Diagnostic checklist for a running inspection model
- Lighting and exposure are monitored, including ambient sources such as daylight and overhead lights.
- Camera placement is monitored, since a disturbed camera can require a full reset.
- New input data is compared with the training distribution using a test such as Kolmogorov-Smirnov or PSI.
- Retraining follows a significant performance drop or detected data drift.
- A labelled sample of production parts is reviewed regularly, because performance monitoring needs ground truth.
- Retraining data excludes low-confidence and heavily drifted images rather than absorbing the model’s own labels.
- Reported metrics include PG2, good parts passed at a 2% miss rate, not only AUROC.
The retraining trigger can be time-based, performance-based, on demand or a mix, and periodic retraining is easier when it lines up with planned production pauses.
TechnoLynx treats this monitoring as reliability engineering, a discipline separate from model accuracy: drift detection for inspection models, built to catch silent quality loss before it reaches customers.
Validation that predicts the line
The studies above share one lesson. Validation is only as honest as the conditions it samples. Baitieva et al. argue that success on MVTec AD is no longer enough to predict deployment readiness and that validation protocols must reflect real deployment conditions. RobustAD proposes a relative-drop metric because practitioners care about how much performance falls on production stations relative to the qualifying environment. We would qualify an inspection model on a holdout set captured on different days, under the line’s real lighting range and after a deliberate camera re-mount, and report the drop alongside the headline score. If that drop is large, we would not treat the model as ready, however good its benchmark looks. What stays open is how wide the qualifying shifts must be for a given line: RobustAD does not cover camera blur or extreme pose changes.