Deep learning inspection models struggle once production inputs differ significantly from the data they were trained on. A 2024 study of two industrial deep learning inspection systems found that validation F1 of 93.2% fell to 45.7% on the critical production data the authors collected for one use case, and from 93.3% to 76.6% for the other. A 2025 remanufacturing benchmark found the same pattern: models scored an average F1 above 96% on a random split, then fell to 63–71% when the test set came from a component category the models had not trained on. The 2024 paper points to new defects, lighting and background changes and underrepresented inputs. The 2025 benchmark points to new components and unknown defect patterns.
Why a vision QC model that passed its pilot underperforms on the production line
A defect-detection pilot can pass and still fail on the production line when it is measured on data drawn from the same pool as the training set. Standard evaluation randomly splits one pool of data into training, validation and test subsets, which assumes all of them share one distribution and that no “new” data will turn up in the test split. On a real line, data drifts slowly and a long-tailed distribution produces defects nobody has seen before, so a good score on an equal split may say little about production. The 2025 remanufacturing authors make the same criticism: inspection models are typically tested on randomly selected, identically distributed data, which does not take distribution shift into account.
Production data can shift gradually, suddenly or seasonally, and the shift can bring new defects, new backgrounds, blur, noise or different illumination. In one of its cases, a slightly damaged ink dispenser on the printing machine produced smudged prints, a defect type the training data had never covered, and the model performed poorly on it. In the other case, ambient lighting combined with poor exposure settings shifted the data, and a new open-mark defect on glass bottles made predictions “almost random”.
Worked example: near-perfect validation, failure the next day
A 2023 study on robustness in manufacturing defect detection shows the pilot trap directly. A binary OK/defect classifier trained on images of mending plates performed almost perfectly on its validation set. It then failed on a holdout set prepared the same way and differing only in being captured on another day. The authors read this as overfitting: the classifier had learned spurious features of its training images instead of the defect itself.
The images show nearly identical parts again and again, defects are rare, and the background is often the part itself, so a model can learn shortcuts that have no causal link to the defect. The same authors warn that a model overfit to homogeneous, imbalanced data may be as brittle as a classical rule-based system, or more so. Training on similar defects across a varied set of flat metal parts produced a classifier AUC of 0.92 that held on held-out data. Switching from classification to object detection improved results further, because having to localise the defect pushed the model towards more robust features.
We would run the “another day” test before any pilot result is allowed into a business case: capture fresh images on a different shift or day, keep them away from training entirely, and score against those.
Overconfident scores make the failure silent
Deep networks will produce an output for any correctly formatted input, even a meaningless one, and the models in the 2024 study could not tell in-distribution inputs from drifted ones. When that study moved from validation to production data, the false acceptance rate of a confidence-based filter rose five times for one use case and ten times for the other, while rejection rates stayed low and stable. In other words, the models kept assigning high confidence to unusual, novel images.
The same authors say that even calibrated confidence scores should be treated with caution, because they may not capture epistemic uncertainty under drift and their reliability can degrade. Their own fix adds a distribution-based filter on top of confidence, which cut overconfident false acceptances by up to six times and let fewer than 9% of false predictions through. In practice, a “confidence threshold” on a vendor dashboard is not a drift alarm, and it should not be bought as one.
Handling unexpected objects or defects, instead of forcing every image into a known class, is covered in unknown-object and unexpected-defect handling.
How dust, heat, vibration and lighting break computer vision on the factory floor
Some of what looks like model drift is physical. Automation World (2026) lists dust, debris, vibration, temperature changes and lighting variation as causes of measurement errors, false readings and outright sensor failure in industrial vision. iFactory, a vision vendor, gives an illustration rather than a measurement: a model scoring 99.6% in a climate-controlled lab dropping to the mid-80s within a week beside a stamping press, with no change to the model. Its explanation is mechanical. A ten-degree swing in ambient temperature can expand or contract standard mounting materials enough to move the camera and introduce error without any alarm.
Retraining is the wrong first response to this kind of degradation. iFactory argues that retraining a model to compensate for a misaligned camera leaves the misalignment in place and teaches the model to chase a moving target, which gets worse the next time conditions shift. Calibration belongs to the whole optical setup, not only the camera: changing the lens, the light or the background means the system has to be recalibrated. A 2021 production-line study of glass medicine bottles found the same problem even with a dedicated light source, because the darkroom was not fully enclosed and outside light affected image quality as bottles entered and left it.
| Symptom on the line | Likely physical cause | First check |
|---|---|---|
| Accuracy declines gradually over weeks | Dust or oil mist building up on the lens | Inspect and clean optics before touching the model |
| Intermittent errors with no clear pattern | Vibration moving the camera slightly when a nearby machine cycles | Correlate errors with neighbouring machine cycles; consider mounting off the machine frame |
| Day shift and night shift perform differently | Ambient or overhead lighting drift | Log ambient light; check exposure settings and enclosure |
| Calibration in doubt after a change | Lens, light or background altered since setup | Run a golden go/no-go part at the start or end of each shift |
Most cameras have no built-in predictive maintenance. Automation World notes that, apart from factory-calibrated sensors and some IO-Link cameras that can report being bumped or running outside their temperature range, maintaining the camera is left to the user. Mind Core, a practitioner blog (2026), states the broader point: controlled lighting and repeatable part presentation set the ceiling on what any inspection model can achieve, so the fixture is the first purchase decision. Our article on how lighting drift on a production line breaks a CV inspection model works through the lighting case in detail.
Rule-based machine vision versus AI inspection for defect detection
The difference is where the decision logic comes from. Cognex describes rule-based systems as following step-by-step if-then instructions programmed by vision engineers (“if the bottle is filled to at least this level, let it pass”), while AI systems learn their decisions from a database of reference images. The two approaches also need different people and inputs: deep learning does not need an experienced machine vision programmer, but it does need someone who knows the product well enough to supply hundreds of images of defective and good parts.
| Rule-based machine vision | AI (deep learning) inspection | |
|---|---|---|
| Decision logic | Programmed if-then rules | Learned from labelled reference images |
| Main burden | Rules programmed by vision engineers and retuned as conditions drift | Supplying hundreds of images of defective and good parts |
| Strength | Still the prevalent technology for well-defined checks | Complex defects and images with significant variation |
| Typical weakness | Programming for every possible result is time-consuming or impossible | Brittle when trained on homogeneous, imbalanced data |
Vendors put numbers on the gap, and those numbers should be read as vendor claims. Keyence says rule-based systems commonly produce false rejection rates of 5–15% or higher on complex parts. Overview.ai warns buyers about rule-based threshold logic marketed as AI that needs constant retuning as conditions drift. Cognex itself notes that a combination of the two approaches is often the most efficient solution. A learned model is not automatically more robust: the 2023 robustness study’s warning about overfit models being as brittle as rules-based ones applies here.
AI and manufacturing QC: when a custom vision system is worth the extra cost
An integrated off-the-shelf product is the sensible default when the part is uniform, presentation and lighting are controlled, and defect classes are few and well represented. Mind Core puts it directly: if a part reaches the camera identically under controlled light, several models will do. Custom engineering earns its cost when one or more of the following holds.
Rare defects. Underrepresented inputs are the ones a model is most likely to misclassify. A defect that occurs once a quarter will not appear in any reasonable collection window, and Mind Core advises saying explicitly that those defects stay under human inspection. The 2023 robustness study points to designing the training set on purpose: diverse backgrounds, and clustering to find which data actually matters.
Multi-part variation. Detecting anomalies across several component types with one model is much harder than for a single type. Models can end up learning what a component looks like instead of what a defect looks like, because a given component’s images are often mostly OK or mostly defective.
Takt time and line integration. The accept or reject decision has to arrive within the line’s takt time every cycle, and Overview.ai notes that cloud round trips add latency and a network dependency. The 2021 bottle study inspected moving bottles at around 95 per minute, with rates fluctuating during stops and refuelling, which made image-trigger timing and detection-area position uncertain. Its authors also note that many existing designs required rebuilding the production process, cutting production rate and raising cost.
Who owns retraining. Finite labelled datasets cannot cover every natural variation, so models need continuous training on fresh labelled samples, and self-generated labels can cause silent degradation. GSK’s 2023 automated visual inspection case study describes this loop: label new production images, retrain offline, then revalidate under change control before release.
TechnoLynx’s position from retail and operational computer vision, which applies here by analogy, is that off-the-shelf models fail because scale breaks assumptions smaller deployments never expose (severe class imbalance, visually similar items, constrained edge hardware, unknown objects silently misclassified), and that these compound failures need an engineering approach, not a model swap. TechnoLynx builds and deploys custom computer vision systems, including anomaly detection. The reliability artefacts an industrial CV inspection pack needs cover what should exist alongside the model once it runs.
What to test before trusting a vendor’s accuracy figure
Ask for escape rate and false reject rate per defect class, measured on a defect-enriched sample, and treat a single accuracy figure as answering a question nobody asked. When defects are a fraction of a percent of output, a model that labels everything good scores above 99% accuracy, so a vendor quoting 99.4% may be describing a system that has never caught a defect. Escape rate (genuine defects passed as good) is what customer complaints track, and false reject rate (good parts condemned) is what scrap and rework track. The two pull against each other at every threshold setting. Missed defects are usually the more serious error, but the 2023 robustness study notes that the false positive rate usually decides whether automating inspection pays at all, because it sets how much manual work is avoided.
A procedure that holds up:
- Write the pass criteria first. Overview.ai advises defining identical criteria and a common scoring rubric for accuracy, latency, integration effort and time to deploy before engaging any vendor, and running no more than two or three proofs of concept at once.
- Build a defect-enriched test set from the reject bin and the parts your quality team has kept, since a week of normal production may contain only a handful of defects; remember the enrichment when translating results back to line rates.
- Keep the test set physically independent. GSK’s validation strategy used one physical defect set for training and a separate new set for testing, because splitting the same kit risks near-identical images on both sides and a biased evaluation.
- Include messy and borderline parts, because a system shown only clean, hand-picked examples will flatter every vendor equally.
- Run site acceptance on live parts from your line. Overview.ai, a vision vendor, proposes a false-accept rate near 0% and a false-reject rate at or under 1%, verified within about five days, and says accuracy quoted on a vendor dataset does not count.
- Test on images captured on a different day, since the mending-plates classifier above failed on exactly such data.
- Name an owner for the sensitivity threshold and log every change with a date and reason; a threshold moved several times without a record makes every earlier quality figure incomparable.
- Log every operator override with the operator, timestamp and image; a rising override rate means the system is being worked around.
Long continuous runs are worth asking for too. The 2021 bottle study validated its system with a 72-hour continuous test on the production line, chosen to cover the plant’s round-the-clock shift pattern, and reported a total error rate of 0.727‰.
For a formal contract basis, VDI/VDE/VDMA 2632 is the relevant guideline series: Part 1 defines terms, Part 2 covers requirement and system specifications, and Part 3 addresses acceptance testing of classifying vision systems. VDI published Part 3 in October 2017 to introduce characteristics describing classification performance, where no established parameters had existed for systems with attributive results. VDI itself notes that, given the variety of classifying systems, a general assessment scheme is scarcely possible.
FAQ
What is a vision system in manufacturing?
A machine vision system uses cameras and image processing to count, classify, or accept and reject items on a line. It makes those decisions either from programmed rules or from a model trained on reference images.
Why is AI important in manufacturing quality inspection?
A 2026 practice review describes AI-driven computer vision increasingly replacing manual and rule-based inspection, because conventional procedures are limited in scalability, prone to human error and costly to run. The 2021 bottle study adds that manual inspectors struggle with some defect types and lose attention over long shifts, while a machine does not.
