Deep learning wins in computer vision when you have plenty of labelled data and the compute to train on it. Classical or hybrid methods win when data is scarce, when the compute or latency budget is tight, when the problem is mostly geometry or known objects, or when you need to explain why the system made a call. That condition-by-condition answer matters more than any general ranking of computer vision and deep learning against older techniques. The case for it is simple. Deep learning has pushed image processing further than anything before it, but that has not made traditional computer vision obsolete. And in TechnoLynx’s approach, the largest gains frequently come from restructuring what is computed, not just how.
Computer vision and deep learning: what changed in 2012, and what came before
Computer vision is the field. Deep learning is one family of methods inside it. Before convolutional networks took over, a vision pipeline was assembled from hand-designed parts: a detector that finds interest points, a descriptor that summarises the pixels around them, and a matcher or classifier on top. SIFT (scale-invariant feature transform), HOG (histogram of oriented gradients) and the Viola-Jones cascade for face detection are the standard examples of that era.
Lowe’s SIFT paper shows what the classical approach is good at. SIFT features are invariant to image scale and rotation and robust across a substantial range of affine distortion, 3D viewpoint change, noise and illumination change. A typical 500x500 image yields about 2,000 stable features, although the count depends on image content and parameter choices.
The turning point came with the convolutional network report usually called AlexNet. LeCun, Bengio and Hinton (Nature, 2015) describe it as a breakthrough that almost halved the error rate for object recognition and set off the computer vision community’s rapid adoption of deep learning. The difference in kind is that deep learning learns layered representations straight from data through backpropagation, rather than having an engineer specify them. Sun et al. (2017) put that success down to three things: high-capacity models, more computational power, and large-scale labelled data. Each of those three is a cost as well as a capability, and the rest of this comparison follows from that.
How much labelled data does a deep learning vision model need?
Flexible deep learning models need far more data than more constrained models to reach the same performance. Thompson et al. (The Computational Limits of Deep Learning, 2022 revision) generalise an insight they attribute to Andrew Ng: traditional techniques do better when data is small, and flexible deep learning models do better when there is more of it. In their illustration, the flexible model needs 1,500 observations to match what a correctly specified model reaches with 15. That figure comes from an illustration, not a vision benchmark. It still shows the shape of the trade-off: flexibility costs samples.
The returns on extra data also diminish. Sun et al. (2017) found that performance on vision tasks increases logarithmically with the volume of training data.
Two approaches reduce that appetite, and each has limits.
- Transfer from a pretrained network. Kornblith et al. found a strong correlation between ImageNet accuracy and transfer accuracy (r = 0.99 for fixed features, 0.96 for fine-tuning). On two small fine-grained classification datasets, though, ImageNet pretraining gave minimal benefit. Their conclusion is that ImageNet architectures generalise well while ImageNet features are less general than often assumed.
- Smaller models and augmentation. Brigato and Iocchi showed that with only a few samples per class and no augmentation, low-complexity CNNs perform comparably to or better than state-of-the-art architectures. Standard data augmentation alone improved recognition by large margins in their tests.
The practical reading is that scarce labelled data is not by itself a reason to abandon deep learning. It is a reason to shrink the model, augment hard, and check whether the pretrained features actually match your domain. If your parts or defects look nothing like ImageNet’s photographs, assume transfer will help less than the headline correlation suggests, and test it.
Compute and hardware: training scale, inference cost and CPU pipelines
Deep learning is expensive by design, not by accident. Thompson et al. argue that the same flexibility that lets it outperform expert models also makes it dramatically more computationally expensive. Their measured scaling for ImageNet is stark: every halving of the remaining error needs roughly 5,000 times as much computation, with a 95% confidence interval of about 1,500x to 17,500x. Computation alone explains 71% of the variance in ImageNet performance in their data.
The training bill grew to match. The same paper notes that AlexNet trained on 2 GPUs for 5 to 6 days in 2012, while ResNeXt-101 in 2017 took 8 GPUs for over 10 days. Much of the growth came from running models longer on more machines, far beyond what GPU gains supplied.
Inference is a separate budget, with its own measures of time, memory and power. Canziani et al. measured ImageNet networks for practical use and found that accuracy and inference time sit in a hyperbolic relationship, that the operations count is a reliable estimate of inference time, and that an energy constraint puts an upper bound on achievable accuracy and model complexity. If the target is an embedded board with a fixed power envelope, that bound is the binding constraint, not the leaderboard. Compression helps: pruning, quantisation and low-rank factorisation can keep a trained network’s accuracy while cutting the floating point operations it needs.
In Lowe’s 2004 tests, recognising all the objects in his test scenes took under 0.3 seconds on a 2GHz Pentium 4. Schönberger et al. (CVPR 2017) found hand-crafted descriptors much faster to extract than learned ones even when the hand-crafted ones ran on CPU, which made learned features impractical for pipelines processing millions of images. For a high-throughput manufacturing inspection line or a CPU-only edge device, that gap can settle the question before accuracy enters it. If deep learning does go into production, the architecture and deployment choices are a topic of their own, covered in our piece on deep learning for image processing in production.
Can you debug a deep vision model the way you debug a classical pipeline?
Not in the same way, and some tools that promise to close the gap fail basic sanity checks. A classical pipeline is a chain of stages you can inspect: you can draw the detected keypoints, look at the matches that survive a ratio test, and see which threshold rejected a candidate.
Deep networks reach high discrimination power at the cost of low interpretability of their black-box representations, as Zhang and Zhu’s survey of visual interpretability puts it. The same survey names semantic debugging of network representations as one of the bottlenecks that better interpretability could remove. Saliency maps are meant to show “where the model is looking”. Adebayo et al. ran sanity checks on those methods and found that some popular saliency methods are independent of both the model and the data-generating process. Methods that fail those checks are inadequate for finding outliers in the data, explaining what the model learned, or debugging the model.
Before a saliency method goes into a debugging or audit workflow, check that it passes those sanity checks: a map that does not depend on the model cannot explain it.
Failure behaviour: shortcut learning versus brittle hand-tuning
The two families fail differently, and knowing which failure you can tolerate is a large part of the decision.
Deep models can learn shortcuts. Geirhos et al. define them as decision rules that perform well on standard benchmarks but fail to transfer to harder testing conditions such as real-world scenarios. The danger is that a shortcut performs well on standard benchmarks, so benchmark scores do not reveal it. Learned components can also overfit to the conditions they were trained on. Schönberger et al. saw learned descriptors with high variance across datasets, which they read as overfitting to specific image transformations from training data that did not cover the full appearance space.
Benchmark gains also do not always survive into the downstream task. In the same evaluation, learned descriptors beat SIFT on raw matching but were typically only on par with SIFT once embedded in a full reconstruction pipeline, which shows that better matching does not necessarily lead to better reconstruction results. Score the system on the output you ship, not on an intermediate metric.
Classical methods have their own limits, which Lowe measured and reported. Lowe reported reliable recognition of textured planar surfaces over up to 50 degrees of rotation in depth, but only about 30 degrees for 3D objects, with illumination change more disruptive there. That is a narrow operating envelope, but it is a known one. Lowe’s limits were measured and published; shortcuts, by Geirhos et al.’s definition, pass standard benchmarks and fail under harder real-world conditions.
When do traditional computer vision methods still beat deep learning?
Classical methods still win in several measured cases involving geometry, matching and known objects.
- Camera pose and localisation. Sattler et al. (CVPR 2019) found that CNN-based absolute pose regression does not reach the accuracy of 3D structure-based methods. Current pose regression approaches did not consistently outperform even a handcrafted image retrieval baseline.
- Wide-baseline image matching. Jin et al. (IJCV 2020) showed that with proper settings, classical solutions may still outperform the perceived state of the art.
- Local features for reconstruction. Schönberger et al. concluded that advanced hand-crafted features still perform on par with or better than learned features in image-based reconstruction, with advanced SIFT variants outperforming learned features on almost all metrics they tested.
- Known objects with applicable physics. Thompson et al. point to recognising known objects such as vehicles as a case where techniques built on engineering and physics knowledge already outperform deep learning.
- Small data in general. Non-neural approaches outperformed neural ones when data was limited in the text classification study Thompson et al. cite, with the reverse when data was plentiful. That study is from text, not vision, so treat it as consistent with the vision evidence above rather than as a vision result in itself.
A caution on the matching results: they depend on proper settings. Jin et al.’s result holds “with proper settings”, so it does not describe a classical baseline left at untuned settings. For a business-level look at where these methods pay off, see our article on the benefits of classical computer vision.
Restructuring the problem often beats a bigger model
The most useful question is often not “deep learning or classical?” but “which parts of this problem actually need learning?”. Calculating the same thing the same way is often too strong a limitation, and the largest improvements frequently come from changing what is computed, not just how. Thompson et al. reach a similar conclusion from the compute side: beyond incremental efficiency gains, fundamental rearchitecting is needed to lower computational intensity.
In computer vision, a related line of work is hybrid methods. O’Mahony et al. (2019) review hybrid methods that combine classical and deep techniques to improve performance and to tackle problems not suited to deep learning. They note these combinations have been popular in panoramic and 3D vision, where deep models had not yet been fully optimised. Jin et al.’s benchmark also found that tuning the classical stages of structure-from-motion pipelines improves results for both algorithmic and learned methods.
Typical shapes of a hybrid, as we would consider them:
- a learned detector proposes regions, and classical geometry (homography, epipolar constraints, RANSAC) verifies and measures them;
- classical preprocessing (rectification, normalisation, background subtraction) narrows the input so a small network can do the part that really is appearance-driven;
- known physical structure, such as a fixed camera, a calibrated rig or a part drawing, is encoded directly instead of being relearned from thousands of labelled frames.
The third pattern echoes what Thompson et al. describe for symbolic and neuro-symbolic approaches, which build a high-level representation of the system so it can be understood and explored with less data.
Decision table: choosing between deep learning, classical and hybrid
| Condition | Lean towards | Why |
|---|---|---|
| Plentiful labelled data | Deep learning | Flexible models overtake expert models once data is plentiful |
| Few samples per class, no augmentation | Small CNN | Low-complexity CNNs match or beat state-of-the-art architectures with scarce data |
| Domain far from ImageNet, fine-grained classes | Test transfer before relying on it | ImageNet pretraining gave minimal benefit on small fine-grained datasets |
| CPU-only or high-throughput pipeline | Classical features, or a compressed network | Hand-crafted descriptors extract much faster, even on CPU |
| Fixed power envelope at the edge | Size the model to the energy budget | Energy constraints bound achievable accuracy and complexity |
| Camera pose and localisation | Structure-based geometry | Structure-based methods beat CNN pose regression |
| Decisions must be explained or audited | Classical or hybrid with inspectable stages | Deep nets trade interpretability for discrimination power |
| Deployment conditions differ from training data | Hybrid, with stress tests beyond the benchmark | Shortcuts pass benchmarks and fail in harder real-world conditions |
Before training anything, it is worth checking whether a prompt-adaptable model already covers the task; our article on image understanding with zero- and few-shot models looks at where that route runs out. Whether to build a custom model at all, rather than buying one, is a separate decision, covered in when to build a custom computer vision model.
A rule that holds up in practice
Start from the constraint that cannot move: the labelled data you can realistically get, the hardware and latency budget at deployment, and how much explanation the decision needs. If all three are generous and the appearance is too varied to describe, deep learning earns its cost. If any one of them is tight, design the classical or geometric part first and add learning only where the remaining errors are about appearance. Then evaluate on the downstream output under conditions harder than your benchmark, where shortcut learning and benchmark-to-pipeline gaps show up.
The remaining design question is where exactly the learned part should start and stop. Drawing that boundary for a production vision system is design work TechnoLynx’s Computer Vision R&D team takes on.
