Computer science machine learning is traditionally divided into three broad types. Supervised learning learns from labelled examples, unsupervised learning finds structure in unlabelled data, and reinforcement learning learns from rewards. Deep learning is the branch that learns its own features from raw input, where classical methods rely on features designed by hand. Standard validation, such as the holdout method, trains the model on one dataset and tests it on a held-out slice of that same data. When the production camera, lighting or product mix is different, the accuracy you measured in the lab no longer describes the system you deployed. In our judgement, that gap matters far more than the choice of algorithm, and it is why off-the-shelf CV fails in production.

The rest of this article covers the types and the deep learning distinction briefly, then looks at what breaks between the lab and the production line, how to test for it and what to do when it happens.

Types of machine learning and the tasks each one fits

The split comes down to what feedback the learning system gets. In supervised learning, a “teacher” supplies each input with its desired output. In unsupervised learning, no labels are given. In reinforcement learning, a program acts in a dynamic environment and is rewarded for reaching a goal. Google’s introductory material adds generative AI as a fourth category: models that create new text, images or music by learning patterns from existing data.

Type	Feedback the model gets	Typical tasks	Where it shows up in visual inspection
Supervised	Labelled input and output pairs	Classification (discrete categories), regression (continuous values)	Usually the best performer of the three, but needs annotated data.
Weakly supervised (inspection research)	A small labelled set plus unlabelled data	Classification that exploits cluster structure in the data	Uses a small annotated set alongside unlabelled images.
Unsupervised	No labels; structure in the data itself	Clustering, dimensionality reduction	Flagging defective parts without labels, including “cold-start” models fitted on non-defective images only.
Reinforcement	Rewards and penalties from an environment	Sequential decisions, control	:
Generative AI	Patterns learned from existing data	Creating text, images, music	Generating data instances that resemble a given class.

Classification fits when the output is limited to a set of values, such as which folder an email belongs in. Regression fits when the output can be any number in a range, such as a forecast temperature. Some real systems combine the types. One published example describes autonomous vehicles that use supervised learning for object recognition, unsupervised learning for scene understanding and reinforcement learning for navigation and control. Reinforcement learning is covered in how agents learn through trial and error with reinforcement learning.

How deep learning differs from classical machine learning

The main difference is how features are extracted. A classical method such as a support vector machine works on features extracted in advance with techniques the researchers designed, and training learns how to map those feature vectors to labels. A deep network takes raw signal values, such as pixels, and derives its own features. Because the network has to discover those features itself, it needs many more training samples than traditional approaches do.

Deep learning uses models made of multiple processing layers that learn representations of data at multiple levels of abstraction. Backpropagation indicates how to change the parameters that compute each layer’s representation from the layer before it. Deep learning is a subset of machine learning, which is in turn a subset of artificial intelligence.

Neural networks surpassed many earlier methods, but deep learning does not win every task. One comparison on the Malicia malware dataset found that a classical random forest beat deep neural networks of 2, 4 and 7 layers on every feature set tested: 99.78% best accuracy against 99.21%. Deep learning is also generally more expensive to compute. The signals that point to a classical method on vision problems are covered in our piece on ML data quality and drift in CV, so we leave that question there.

Applications of machine learning on a production line: visual inspection

Visual inspection checks whether manufactured products meet quality standards and catches both functional and cosmetic defects. Human inspectors have traditionally done this work. Human inspection does not scale well and is subjective, and the human visual system excels at variety, while inspection means observing the same product type repeatedly. In automated visual inspection, deep learning approaches dominate the current state of the art.

The application shows how the ML types play out in practice. PatchCore handles the cold-start case, where only images of good parts exist. It combines ImageNet-pretrained embeddings with a memory bank of nominal patch features and reports state-of-the-art detection and localisation with competitive inference times. Supervised methods usually perform best, but they depend on annotated data. A paper sharing EU H2020 STAR project results describes data collection and annotation as expensive and still an open challenge. Labelling quality also falls over time: the same project’s experiments found human labelling accuracy dropped as annotators tired. Active learning, which picks the images most worth a human’s labelling effort, reduced the labelling needed, but how much depends on the image pool, the use case and the strategy.

Two findings from that project are worth keeping in view. Baseline inspection classifiers reached 93% to 98% accuracy. Under DeepFool, FGSM and PGD adversarial attacks, accuracy fell below 30%. And there is no universal inspection solution: most systems are custom-built for one product type.

Why lab accuracy stops describing a deployed vision model

Off-the-shelf vision models fail when production data differs from the distribution the lab score was measured on. Generalisation means performing accurately on new examples drawn from the same unknown distribution the training data came from. The holdout and K-fold methods split one dataset into training and test parts, so they do not test a change of distribution. The WILDS benchmark covers 10 datasets reflecting distribution shifts that arise in real-world applications. On every one, standard training gave substantially lower out-of-distribution accuracy than in-distribution accuracy, and the gap remained for models trained with existing shift-robust methods.

The recurring production failure is not a bad architecture choice. It is an off-the-shelf model deployed without enough testing on representative production data, whose limits only come out once users or operators are relying on it. We hold that problems treated as solved, such as object detection and tracking, are not: real conditions and scale change the picture, and models that work in demos fail on the edge cases, lighting and throughput of actual deployments.

Covariate shift, label shift, and both at once

Covariate shift means the input distribution changes while the relationship between inputs and labels stays fixed. Label shift means the label distribution changes while the inputs for a given label stay fixed. In practice features and labels often shift together, and an estimation method built for one kind of shift can give wrong answers when the shift is really the other kind. Zhang et al. (ICML 2023) simulated moving an ICU mortality model from Midwestern to Southern US hospitals and attributed 29.5% of the performance drop to the change in age distribution (0.0108 of a 0.0366 total).

Underspecification

D’Amour et al. (JMLR 2022) identify a second failure mode, distinct from mismatch between training and deployment domains. A pipeline is underspecified when it can return many predictors with equally strong held-out performance. Those predictors are treated as interchangeable, but they can behave very differently once deployed. The authors show it in computer vision and healthcare machine learning, including medical imaging, among others. The practical consequence: two training runs with identical validation curves are not evidence that the shipped model will behave like either of them.

The system around the model

Sculley et al. (NeurIPS 2015) observed that ML systems are relatively fast and cheap to build and deploy but difficult and expensive to maintain. They suggest a mature system might end up at most 5% ML code and at least 95% glue code. Input signals owned by other teams can change behaviour without warning, and even an “improvement” upstream can damage the model consuming it. Static tests are not sufficient evidence that a system is still working in a changing world. Live monitoring with automated response is critical for long-term reliability.

What kinds of edge cases break public detection and classification models?

Many edge cases that break public models are ordinary changes in the scene. Cameras get moved or replaced, lighting changes, products are redesigned in new colours, shapes or packaging, and a second facility has different backgrounds and camera positions. Objects may appear smaller, further away or more occluded than in the training set, and cases that were rare at launch become routine. The camera, lighting, product, site and object changes are data drift. Concept drift is different: the definition of the target itself changes, for example a new defect definition.

Public benchmarks show how steep the drop can be. Roboflow, citing a published study, reports that a DenseNet-121 classifier scored around 2% on ImageNet-A, a set of naturally occurring hard images chosen to expose model weaknesses. That is a drop of roughly 90%, and the same model’s ability to flag out-of-distribution images on ImageNet-O was close to chance. The cause is partly what models learn in the first place. Image classifiers often pick up pixel relationships humans do not notice instead of the spatial arrangement of an object’s parts. A classifier trained only on brown horses and black cats may decide that brown patches mean “horse”. Modifying those patterns on a legitimate image produces adversarial images the system misclassifies. Safety-critical failures sit at the far end of this range: in 2018 an Uber self-driving car failed to detect a pedestrian, who was killed.

Testing an inspection model on data from the deployment site

Test it on data from the deployment site, kept separate from the validation split, and diagnose any gap instead of just recording it. A sequence we would follow:

1. Keep two evaluation sets. Evaluate on a validation split from the training distribution, then separately on data from the target site. The difference then reflects distribution shift, not overfitting.
2. Capture the target set from the real line. Production brings moved cameras, new lighting, redesigned products and second sites with different backgrounds.
3. Stress-test the conditions you know will vary. Our view is that occlusion, scale, lighting extremes and rare defect types each need their own slice, so that a good average cannot hide a failing slice.
4. Compare candidate models, not just one. Underspecified pipelines can return equally good validation scores from predictors that behave very differently in deployment, so validation scores alone cannot pick the model.
5. Decompose any drop. DISDE (Cai et al.) splits a performance drop into three parts: more hard but familiar examples, a changed relationship between features and outcome, and examples rare or unseen in training. The last two call for different fixes.
6. Plan for unlabelled monitoring. After launch, new data often arrives without labels, so estimating performance on it is a step toward reliable ML. Production ML runs as a loop of labelling, experimentation, staged evaluation and monitoring for performance drops.

The artefacts that keep this loop running on a factory line, from acceptance sets to drift triggers, are set out in industrial CV inspection production reliability. TechnoLynx’s Production AI Monitoring Harness builds the eval, regression and drift harness for that loop.

The price of finding a model’s limits after launch

The model’s limits come out while users or operators already depend on it. Mitigations include collecting more data, and data collection and annotation are expensive. Cai et al. note that even advanced industrial monitoring systems detect degradation without attributing it to a cause.

An informed fix also works better than a blind one. In the ICU case, adding data targeted at the attributed cause cut the performance drop by 36.6% and beat naive retraining on the merged datasets. The cost of late discovery is therefore partly the cost of fixing without a diagnosis.

Matching the fix to the diagnosed shift: fine-tune, retrain or rebuild

Fine-tuning on newly collected examples addresses failures on inputs rare or unseen in training. When the relationship between inputs and the right answer has changed, new labelled data or a changed set of inputs is needed. Retrain when production evidence shows a measured decline, changed images, repeated failure patterns, new requirements or annotation errors, not because a set amount of time has passed.

What the diagnosis shows	What it means	Action it points to
Poor results on examples rare or unseen in training	Covariate shift toward “newer examples”	Collect those examples and retrain or fine-tune.
Same inputs, different correct answer	Change in the label given the input (concept drift)	Collect and label new data, possibly change the inputs.
The outcome depends on a variable the model does not see	A missing feature hides a relationship shift	Rebuild with the feature: in Cai et al.’s census case, adding education raised target accuracy from 61% to 70%.
Training data covered too narrow a population	Selection bias	Retrain on broader samples: target accuracy rose from 66% to 73% in the same study.
Images look different but performance holds	Drift without harm	No retraining yet.
Accuracy collapses under crafted inputs	Adversarial vulnerability	Adversarial training was the most stable defence in the STAR experiments.

One warning applies across all of these rows. A fix aimed at the wrong kind of shift can make things worse. Cai et al. show a domain-adaptation method (DANN) that reduced feature-shift degradation but increased degradation from the label relationship, so it performed poorly overall.

Telling coverage gaps from failures more data will not fix

Collecting more data fixes gaps in coverage. Examples rare or unseen in training are coverage gaps, and collecting those examples and retraining addresses them. Three other failures are not coverage gaps. Underspecification is one: a failure mode distinct from training-to-deployment mismatch. Reliance on pixel patterns humans do not notice is another, and it leaves classifiers open to adversarial images. The third is a general degradation on new distributions: Recht et al. and Taori et al. observed that models trained on the same data degrade on new distributions regardless of training method or model capacity.

An axis-aligned box drawn around a long, rotated object also takes in background, and no amount of extra labelling changes that. Whether to switch to oriented bounding boxes for rotated detection is a model-class decision, not a data decision. For any other failure, ask one question first: would a thousand more correctly labelled images of this exact case change the model’s behaviour? If the honest answer is no, the problem lies in the model, its inputs or how it was chosen.

FAQ

Is machine learning in computer science?

Yes. Machine learning is a field of computer science and a subset of artificial intelligence, and deep learning is a subset of machine learning.