Image segmentation is the part of computer vision that decides which pixels belong to which thing. A classifier tells you “there is a cat in this frame.” A detector draws a box around the cat. A segmenter labels every pixel — cat, road, pedestrian, tumour boundary, defective weld — and that pixel-level commitment is what downstream systems actually use to plan, measure, or trigger an action. The interesting engineering question is not whether to segment, but which family of methods to reach for, and where the classical layer still beats a fine-tuned deep model. This article walks through the main segmentation families, how they compose with other CV stages, and where a hybrid pipeline saves an order of magnitude of compute over a uniformly-deep stack. What does image segmentation actually produce? A segmenter outputs a label map the same shape as the input frame. Each pixel carries either a class index (semantic segmentation), a class + instance ID (instance segmentation), or both plus a “stuff vs things” distinction (panoptic segmentation). The downstream consumer is rarely a human — it is a path planner, a measurement routine, or a quality-control rule that needs boundaries it can integrate over. Three properties matter when picking a method: Boundary sharpness. Medical scans and inspection systems care about exact contours; a few pixels of slop change a diagnosis or a defect count. Throughput envelope. A driving stack needs to segment at sensor rate; a satellite tile can run offline overnight. Label-budget realism. Pixel-perfect annotation is expensive. Some methods tolerate weak labels; others collapse without dense masks. Most production decisions fall out of these three constraints, not out of which architecture is currently fashionable. What classical segmentation methods still do well Before convolutional networks, segmentation meant rules over pixel statistics — thresholding, edge detection, watershed, region growing, active contours. The deep-learning narrative often treats these as obsolete. They are not. They are the right tool when the scene is constrained, the budget is tight, or the deep model needs a cleaner input. Thresholding (Otsu’s method and its adaptive variants) still dominates document scanning, blob-counting on microscopy slides, and any case where the foreground/background separation is genuinely bimodal. It runs in microseconds on a CPU. Edge detectors (Canny, Sobel) and morphological operators (open, close, skeletonise) underpin most industrial inspection pipelines, where parts arrive in known orientation under controlled lighting and the question is “did the seal close cleanly” rather than “what is in this image.” Region-growing and watershed give acceptable cell-and-nucleus separation in histology, often as a pre-step before a small CNN refines the boundary. The classical step crops the region of interest so the network sees a tight crop instead of a 4K slide. The pattern across these is the same: when the scene is controlled and the question is narrow, a classical method costs cents per million frames and is auditable line-by-line. That auditability matters in regulated settings where a deep model’s behaviour is harder to defend. For the broader case for classical preprocessing — and where it composes cleanly with deep stages — see our explainer on feature extraction and image processing for computer vision. Deep segmentation architectures: what to use when When the scene is open-world — outdoor driving, satellite tiles with weather and shadows, generic object segmentation — deep models win. The relevant families: Fully Convolutional Networks (FCNs). The first architecture that produced dense per-pixel predictions end-to-end. Mostly of historical interest now, but the encoder-decoder shape it introduced is still the template. U-Net. Designed for biomedical images, now the default when training data is small and boundary sharpness matters. The skip connections between encoder and decoder let high-resolution detail survive the bottleneck. DeepLab (v3+). Atrous (dilated) convolutions widen the receptive field without losing resolution, which helps with large objects and structured scenes. Strong on Cityscapes-style driving data. Mask R-CNN. Detect first, then segment inside each detection. The right choice for instance segmentation where you need both “how many” and “what shape.” Segment Anything (SAM) and successors. Prompt-based, promptable masks. Useful as a labelling accelerator and as a zero-shot segmenter for novel object classes, less useful as a production runtime model because of its size. Vision-transformer segmenters (SegFormer, Mask2Former). Higher accuracy ceilings on benchmarks; heavier and harder to compress for the edge. Quick decision table Scenario First pick Why Controlled industrial inspection Thresholding + morphology Microsecond latency, auditable, no training data Medical scans with small labelled set U-Net Skip connections preserve boundaries; works at ~hundreds of labelled volumes Outdoor driving / urban scenes DeepLabV3+ or SegFormer Wide receptive field, robust to scale variation Count + shape of discrete objects Mask R-CNN Per-instance masks, plays well with tracking One-off labelling or rare classes SAM-family (prompted) Zero-shot masks, then distill to a smaller runtime model Satellite / aerial tiles DeepLab or U-Net at tile scale Tolerates large class imbalance; offline throughput-bound Each row is a starting point, not a verdict. The real choice is shaped by latency budget and label budget more than by which paper is currently cited most. How segmentation composes with detection, tracking, and classical pre-processing Segmentation rarely sits alone in a production pipeline. A typical stack for autonomous perception looks like: classical ROI extraction → detector → per-instance segmenter → tracker → planner. The classical front end is doing real work — it crops the frame to the road surface, suppresses sky and ego-vehicle pixels, and stabilises against camera shake. The detector then runs on a smaller, denser tensor; the segmenter only needs to refine masks inside detected boxes; the tracker links masks across frames. This staged design is where the ROI claim comes from: production CV systems that explicitly choose between classical and deep components per stage routinely run at 3–10× lower compute cost than uniformly-deep pipelines for the same task accuracy, in our experience across CV engagements. That is an observed pattern across deployments, not a benchmarked rate — the multiplier depends on scene complexity and on how much of the frame is genuinely “interesting.” But the direction is consistent enough that we treat per-stage classical/deep selection as a default design move, not an optimisation. The same composition logic applies to medical imaging (registration → ROI crop → U-Net refinement → measurement), to manufacturing inspection (calibration → defect detector → segmentation for defect area → pass/fail rule), and to satellite analysis (orthorectification → tile-level classifier → segmenter for the classes that matter). For more on how classical and deep components actually cooperate, see our piece on the business case for keeping classical computer vision in the stack. Where deep segmentation actually fails Deep segmenters fail in predictable ways, and naming the failure modes is what stops them from shipping into safety-critical systems unchecked. Distribution shift at the pixel level. A model trained on daytime urban scenes will produce confident-but-wrong masks at dusk, in rain, or under unusual lens artefacts. The output looks plausible, which is worse than an obvious failure. Thin structures. Powerlines, catheters, weld seams, and small instruments are routinely under-segmented because the loss function is dominated by large regions. Label-budget collapse. Models trained on 50 dense masks plus 5,000 weak labels often outperform the same architecture on 100 dense masks alone, but tooling for mixed-supervision training is still immature in most stacks. Calibration drift. Predicted probabilities are not calibrated out of the box. Downstream rules that threshold the mask probability (for example, “treat pixels above 0.5 as defect”) need either Platt scaling or temperature scaling, otherwise the operating point silently shifts as the model is retrained. None of these are reasons to avoid deep segmentation. They are reasons to keep a classical baseline that the deep model has to beat, and to keep a small held-out evaluation set that exercises the failure modes deliberately. Why bother integrating with the broader CV stack A segmentation mask is not the deliverable. The deliverable is the decision or measurement that the mask enables. Pattern recognition stages downstream of segmentation — counting, classifying instances, tracking through occlusion — depend on mask quality but are themselves separate engineering concerns. For how those layers compose more broadly, our overview on computer vision and image understanding walks through the full pixels-to-reasoning path. The practical implication: when a segmentation model “fails,” the symptom usually shows up two stages later — a tracker losing identity, a defect counter under-reporting, a planner taking an unexpected turn. Debugging starts at the symptom and walks back to the mask. Treating segmentation as an isolated benchmark exercise misses this; treating it as a contract that downstream stages depend on changes how you measure it. What we look at when picking a segmentation approach When TechnoLynx is asked to choose a segmentation strategy for a client system, the conversation we have is rarely about architectures first. We ask about the scene constraints (how controlled is the environment), the label budget (how many dense masks can realistically be produced), the latency envelope (per-frame, per-tile, per-batch), and the deployment target (cloud GPU, on-prem GPU, Jetson-class edge, CPU-only). The architecture choice falls out of those four answers more reliably than out of any leaderboard. The hybrid stance — classical where it earns its keep, deep where it has to — is not a compromise. It is what production CV systems actually look like once cost-per-frame becomes a constraint someone is measuring. FAQ Where does classical feature extraction (SIFT, ORB, HOG) still beat deep features in 2026? In image registration, ROI extraction before a CNN, low-power preprocessing on edge devices, and constrained industrial inspection where the scene geometry is controlled. The classical methods are deterministic, auditable, and run in microseconds on a CPU — properties a deep model rarely matches. How does feature extraction compose with deep CV (CNN features, ViT embeddings) in a hybrid pipeline? Classical stages typically sit at the front (calibration, ROI cropping, motion stabilisation) and sometimes at the back (geometric measurement on top of a deep mask). Deep stages handle the open-world recognition in between. The composition is staged, not interleaved. What does Nixon and Aguado’s feature-extraction framework get right that deep-only stacks miss? It treats feature extraction as a designable layer with explicit invariances (to rotation, scale, illumination) rather than as an emergent property of a trained network. That framing survives the deep-learning era because the invariances are still real engineering requirements — they just get satisfied in different ways. Which feature-extraction techniques translate into ML model inputs versus pure visualization? Edge maps, HOG descriptors, and Gabor responses are routinely concatenated with deep features as additional channels for small-data settings. Watershed and active-contour outputs are usually used as supervision signals or as post-processing, not as raw model inputs. When should an engineering team write a classical-CV feature stage instead of fine-tuning a model? When the scene is controlled, the question is narrow, training data is scarce, and auditability matters. Industrial inspection, document processing, and microscopy frequently hit all four conditions. Open-world recognition rarely hits any of them. How does feature extraction sit alongside image segmentation and pattern recognition in a production pipeline? Feature extraction prepares the input; segmentation commits pixels to classes; pattern recognition operates on the resulting regions to count, classify, or track. The three are sequential contracts, and the system’s reliability depends as much on how cleanly the contracts are defined as on the accuracy of any single stage. How TechnoLynx works on segmentation problems We design CV pipelines stage by stage rather than picking a single architecture. That usually means a classical pre-processing layer where the scene allows it, a deep segmenter sized to the latency envelope, and an evaluation harness that exercises the failure modes deliberately. We deploy across cloud GPU, on-prem, and edge targets, and we treat the per-frame cost as a first-class constraint alongside accuracy. If you have a segmentation workload where the current stack is either too expensive to run or too opaque to defend, get in touch — that conversation usually starts by mapping the pipeline rather than benchmarking the model. Image credits: Freepik
