A vendor shows you SAM segmenting a defect on a sample board in one click, no training data, no fine-tuning. The mask is clean. The temptation is to conclude that segmentation is a solved problem for your line. It is not — at least not the way that demo implies. The Segment Anything Model (SAM), released by Meta AI, is a genuinely useful piece of engineering. It produces high-quality masks from prompts — a point, a box, or a rough region — across an enormous range of image content it never saw during training. On staged pilot frames, it looks like it eliminates the annotation problem and the segmentation problem at once. The mistake is treating that pilot-condition behaviour as a production release criterion. Pilot-condition accuracy is not the same thing as a line-side release model, and a promptable general-purpose segmenter behaves very differently once lighting drifts, packaging gets redesigned, and conveyor variance enters the frame. This article is about where SAM actually belongs in an inspection pipeline: as a labelling accelerator and coarse prior it is excellent, and as a raw production segmenter without drift telemetry it quietly corrupts your rejection rate. How does SAM work? SAM is a promptable segmentation model. Its architecture has three parts: a heavy image encoder (a Vision Transformer backbone) that runs once per image, a lightweight prompt encoder that turns a point or box into an embedding, and a fast mask decoder that combines the two into a segmentation mask. Because the expensive encoder pass is separated from the cheap prompt-conditioned decoding, you can prompt the same encoded image many times cheaply — the design that makes interactive labelling feel instant. What “promptable” means in practice is that SAM does not know what a defect is. It knows how to find coherent regions given a hint about where to look. Point at a solder blob and it will segment the blob; point at the pad and it will segment the pad. The semantics live in the prompt, not in the model. That is a strength for labelling and a liability for autonomous inspection, because on a running line nobody is standing there placing prompts frame by frame. Two properties matter for the rest of this discussion. First, SAM’s masks are strong but general — boundaries are geometrically clean but not tuned to your specific defect taxonomy. Second, its zero-shot generalisation is what makes it feel magical on a demo and what makes it unstable under domain shift: a model trained to segment anything has no anchor to keep it segmenting the right thing when the input distribution moves. Promptable segmentation versus a task-trained inspection segmenter The clearest way to see where SAM fits is to contrast it with a segmenter trained specifically for your inspection task. Dimension SAM (promptable, general) Task-trained segmenter (e.g. fine-tuned U-Net / YOLO-seg) What it needs at inference A prompt per object Nothing — runs autonomously per frame Defect semantics None; supplied by the prompt Learned from your labelled defects Behaviour under drift Degrades unpredictably; no task anchor Degrades measurably against known classes Annotation cost to deploy Near-zero (that is its whole point) High (requires labelled dataset) Throughput on a line ViT encoder is heavy per frame Can be sized to line cycle time Best role Labelling accelerator, coarse prior Release inference component The row that decides most architectures is the first one. Autonomous line-side inspection has no human placing prompts. To run SAM without a prompt you fall back to its “segment everything” mode, which produces a field of unlabelled masks you then have to classify and associate with a defect ontology — reintroducing exactly the task-specific model you were trying to avoid. This is the same trade-off we discuss in choosing a detector that survives production: generality is cheap to start with and expensive to sustain. Should SAM be a production inference component, or only a labelling accelerator? For the vast majority of inspection lines we have seen, the honest answer is: as a labelling accelerator, deploy it enthusiastically; as a raw production segmenter, do not — distil it into a task-specific model first. The labelling case is strong and low-risk. Using SAM to bootstrap segmentation masks cuts annotation effort on inspection datasets by a large margin (observed across TechnoLynx engagements; not a published benchmark), because an annotator clicks once and corrects rather than tracing a boundary pixel by pixel. That shortens the path to a validated line-side model without putting SAM anywhere near the release inference path. The lighter FastSAM variant is worth evaluating here too — we cover its trade-offs in how Fast Segment Anything works and where it fits on the line. The production-segmenter case is where the divergence bites. Teams that deploy raw SAM as the production segmenter without drift telemetry tend to see mask instability inflate false-reject rates and revert to manual inspection within a quarter — the exact failure a production reliability pack is designed to prevent. The problem is not that SAM is inaccurate; it is that its accuracy is not stable across the conditions a line actually presents, and without instrumentation you cannot see the drift until the reject bin fills up. A quick decision rubric Use this before committing SAM to any inference path: Is a human in the loop placing prompts? If no, SAM is not an autonomous segmenter without a “segment everything” + classifier stage bolted on. Do you have prompt-stability telemetry? If no, you cannot detect when a refreshed line silently shifts the mask. Is there a fallback path when masks degrade? If no, a bad mask goes straight to the rejection logic. Can you meet line cycle time with the ViT encoder? If no, you need a distilled model regardless of accuracy. If any answer is “no”, SAM belongs in your annotation tooling, not your release model. How does SAM behave under lighting drift, packaging redesign, and conveyor variance? These three are the everyday reality of a manufacturing line, and each stresses SAM differently. Lighting drift changes the contrast that SAM’s encoder uses to find region boundaries. A mask that hugged a defect edge under morning diffuse light can bleed or shrink under a warmer afternoon spectrum. Because SAM has no defect-specific anchor, it has nothing to hold the boundary in place — it simply segments whatever coherent region the new contrast suggests. Packaging redesign is a distribution shift SAM has no reason to survive gracefully. A relabelled carton, a new tray colour, or a changed component footprint alters the visual field. A task-trained model will report degraded accuracy against its known classes — a signal you can measure. SAM will just produce different masks, confidently, with no error signal attached. Conveyor variance — jitter, speed changes, partial occlusion, motion blur — degrades the input quality the encoder depends on. The failure here is subtle: masks do not disappear, they shift. A mask that is a few pixels off does not throw an exception; it flows straight into the area or shape logic that decides pass/fail. The common thread is that all three failures are silent. SAM does not know it is wrong, because it was never told what “right” is for your task. That is precisely why the parent reliability work insists that pilot-condition accuracy is not a release criterion — the demo frame and the drifted frame both produce a mask, and only one of them is trustworthy. What drift telemetry and fallback path does a SAM-driven mask need? If SAM output touches the release path at all — even as a coarse prior a downstream model refines — it must pass the same line-side drift and rollback checks as any other inference component. Whether SAM is a labeller or an inference component, its output has to clear the industrial-CV validation lens before release. You can see the broader gate structure in our walkthrough of where reliability gates belong at each stage of an ML pipeline. Concretely, three things are non-negotiable: Prompt-stability telemetry. Log how the mask responds to controlled prompt perturbation and how mask geometry (area, boundary length, centroid) drifts frame over frame and shift over shift. A mask whose area distribution moves when the line refreshes is your earliest warning. A defined fallback path. When telemetry crosses a threshold, the system must route to a conservative default — human review, a held-out backup model, or a hard hold — rather than pushing a suspect mask into the reject logic. A rollback point. Versioned evidence of the mask behaviour that was signed off, so you can prove what changed and revert to it. This is where a scorecard and versioned artifacts earn their place. None of this is exotic. It is the same discipline any drifting perception model needs; SAM just makes the need more acute because its failure mode is silent by construction. When is it worth distilling SAM into a smaller task-specific model? Distillation — using SAM to generate a large corpus of masks, then training a compact task-specific segmenter on them — is the move that resolves most of the tension above. You keep SAM’s annotation leverage and lose its production liabilities: the distilled model runs within line cycle time, degrades measurably against known defect classes, and can be validated and monitored like any release model. It is worth distilling when the line runs continuously (throughput matters), when the defect taxonomy is stable enough to train against, and when you need a model you can actually sign off and monitor. It is not worth it when the task is genuinely exploratory, the defect set is still churning, or the volume is low enough that human-in-the-loop review with SAM assistance is economically fine. The decision to use SAM as a labelling accelerator at all is one that belongs in the feasibility lens, before any line-side reliability artefacts exist — it is a manufacturing-CV feasibility question first, as we frame in our production AI reliability view and on the production AI reliability practice. FAQ What matters most about SAM (Segment Anything Model) in practice? SAM is a promptable segmentation model with a heavy Vision Transformer image encoder, a lightweight prompt encoder, and a fast mask decoder. You give it a point, box, or region and it returns a clean mask. In practice this means SAM finds coherent regions given a hint — but the defect semantics live in the prompt, not the model, so it does not know what a defect is on its own. What is promptable segmentation and how does it differ from a task-trained inspection segmenter? Promptable segmentation needs a prompt per object and carries no task-specific defect knowledge; a task-trained segmenter runs autonomously per frame and has learned your defect classes. The practical difference is behaviour under change: a task-trained model degrades measurably against known classes, while SAM degrades unpredictably because it has no task anchor to hold it to the right target. Should SAM be used as a production inference component on an inspection line, or only as a labelling accelerator? As a labelling accelerator, deploy it enthusiastically — it cuts annotation effort substantially and shortens the path to a validated model. As a raw production segmenter without drift telemetry, avoid it: teams that do this tend to see mask instability inflate false-reject rates and revert to manual inspection within a quarter. If SAM must touch inference, distil it into a task-specific model first. How does SAM behave under lighting drift, packaging redesign, and conveyor variance on a real line? All three cause silent failure. Lighting drift changes the contrast SAM uses for boundaries, so masks bleed or shrink; packaging redesign shifts the visual distribution with no error signal; conveyor variance shifts masks by a few pixels that flow straight into pass/fail logic. Because SAM has no defect-specific anchor, none of these failures raise an exception — the mask is always confident, even when wrong. What drift telemetry and fallback path does a SAM-driven mask need before it can be released? It needs prompt-stability telemetry (logging how mask geometry drifts frame over frame and shift over shift), a defined fallback path that routes to human review or a backup model when telemetry crosses a threshold, and a versioned rollback point tied to signed-off behaviour. Any SAM output touching the release path must clear the same industrial-CV validation and rollback checks as any other inference component. How do SAM-generated masks fit into the industrial CV validation pack and scorecard? Whether SAM is used as a labeller or an inference component, its output must pass the industrial-CV validation lens — line-side drift checks and a rollback point — before release, and the evidence is versioned in a scorecard. The scorecard records what mask behaviour was signed off so you can prove what changed and revert if a line refresh shifts the masks. When is it worth distilling SAM output into a smaller task-specific model for line-side throughput? Distil when the line runs continuously (throughput matters), the defect taxonomy is stable enough to train against, and you need a model you can sign off and monitor. Distillation keeps SAM’s annotation leverage while shedding its production liabilities — the compact model runs within cycle time and degrades measurably. Skip it when the task is exploratory, the defect set is still churning, or volume is low enough that human-in-the-loop review with SAM assistance is economically fine. The question that decides whether SAM helps or hurts your line is not “how good are its masks on the pilot frames?” — they will be good. It is “what happens to those masks the first time the line refreshes, and will I see it before the reject bin does?” Answer that with prompt-stability telemetry and a fallback path, and SAM becomes a labelling accelerator worth its weight. Ship it raw against clean pilot frames and you have built a silent drift generator wired directly into your rejection rate.
