Prompt the Segment Anything Model on a captured board image and it hands you a clean mask around whatever region you clicked. It is tempting to read that mask as a defect call. That reading is where AOI lines quietly break. SAM is a class-agnostic segmentation prior. It is very good at proposing where a region’s boundary is, and it knows nothing about whether that region represents a scratch, a solder bridge, or a benign silkscreen change your supplier introduced last week. The mask tells you where the pixels are. It does not tell you whether the line should reject the part. That gap — between locating pixels and making a pass/fail decision — is the whole story of where SAM belongs in an automated optical inspection (AOI) pipeline, and where it must not sit. What does the Segment Anything Model actually output? SAM takes an image and a prompt — a point, a box, or a coarse mask — and returns one or more high-quality binary masks for the region the prompt indicates. It was trained on a very large, diverse dataset of natural-image masks, which is exactly why it generalises so well to boundaries it has never seen. That generality is the feature. It is also the constraint. The model has no notion of your defect taxonomy. It does not carry a calibrated probability that a region is anomalous; it carries a confidence that the mask fits the region boundary. Those are different quantities, and conflating them is the single most common mistake we see when teams first bring SAM near an inspection line. A mask with high boundary confidence around a benign packaging variation looks identical, in the model’s output, to a mask around a genuine defect. The model was never calibrated against your defect distribution, your camera’s lighting drift, or the specific way your line ages its reference conditions. So the honest one-line summary: SAM answers “where is this region?” with impressive quality, and stays silent on “is this region a reason to reject?” Everything downstream depends on respecting that silence. Why raw SAM masks cannot drive a pass/fail signal Say you wire SAM masks directly into reject actuation: prompt the image, threshold the mask area, flag anything above a size cut. On day one, on your pilot images, it might even look reasonable. The failure is structural, and it arrives later. When you use a raw mask as the rejection signal, you couple your line’s reject rate to a foundation model that was never tuned to your process. Three things then bite, usually in this order. Lighting and material drift shift what SAM segments, and because the model is class-agnostic, that shift silently changes your effective decision boundary with no alarm anywhere. Benign process changes — a new label revision, a matte-versus-gloss finish — produce clean, confident masks that a size threshold happily rejects. And because the segmenter carries no defect probability, you have nothing to monitor: there is no calibrated score to plot against escape rate over time. This is the same compound-failure pattern we describe in the CV-specific view of how uncalibrated components become unmonitored line failures: a model that produces plausible output under drift, with no signal that tells you the decision boundary has moved. On an AOI line that shows up as a false-reject rate that walks away from its baseline, or worse, as escapes you only discover downstream. A promptable foundation model is not exempt from the requirement that every reject decision be measurable and monitorable — it is one of the harder cases because its output looks so clean. We go deeper on the specific breakage patterns in our companion piece on what SAM does on an inspection line and exactly where it breaks. The short version: the break is never in the mask quality. It is in what you asked the mask to mean. Where SAM does fit: three roles, ranked by how safe they are SAM earns its place in an AOI program in roles that use its region-proposal strength and never ask it to own the decision. Here is the decision surface we work from. Role What SAM does Owns the reject decision? Safe on a live line? Annotation acceleration Proposes defect-mask boundaries a human confirms/edits No Yes — offline, human-in-loop Region proposal front end Isolates candidate regions for a calibrated classifier No Yes — if the classifier is version-pinned and monitored Direct pass/fail signal Thresholds raw masks into reject actuation Yes No — uncalibrated, unmonitored failure point Annotation acceleration is the highest-confidence use. Instead of hand-tracing every defect boundary, an annotator prompts SAM, accepts or nudges the proposed mask, and moves on. Teams doing this cut labelling time per image substantially rather than tracing pixel by pixel (observed across our industrial-CV engagements; not a published benchmark). The output is a releasable training set, faster — and a human still confirms every mask, so no uncalibrated model touches a decision. Region proposal is the next tier. Here SAM isolates candidate regions and hands each one to a purpose-trained, version-pinned defect classifier that does carry a calibrated defect probability. Used this way, isolating the candidate region before classification can reduce false-reject rate by giving the classifier a cleaner input than a full-frame crop. The critical detail: the reject decision lives in the classifier, which you can version, calibrate, and monitor — not in SAM, which you cannot calibrate against your defects. Direct pass/fail is the role to refuse. It is in the table only to be named as the anti-pattern. How do you combine SAM with a calibrated classifier? The pattern that keeps the reject decision measurable is a two-stage pipeline with a hard boundary between proposal and decision. Stage 1 — proposal (SAM): prompt or auto-prompt the frame, produce candidate region masks. Treat these as hypotheses about where to look, nothing more. Stage 2 — decision (your classifier): crop each candidate region, run a version-pinned defect classifier that outputs a calibrated probability per defect class, and apply your reject threshold on that probability. The classifier is where all your monitoring hooks live: score distributions, false-reject and escape tracking, drift telemetry against a fixed baseline. Because the classifier is version-pinned, a model change is a deliberate, logged, revalidated event — not something that drifts in when a foundation-model weight update lands. SAM can be swapped or upgraded in Stage 1 with far lower blast radius, because it never decides anything; a change there affects which regions get looked at, and the calibrated Stage 2 still gates the actual reject. This separation is what lets you answer the question a line owner will eventually ask: “why did we reject this part?” With raw SAM the answer is “the mask was big enough,” which is not a defensible reason. With a two-stage pipeline the answer is “the version-pinned classifier scored this region at 0.94 against the solder-bridge class, above the 0.80 threshold we calibrated and signed off.” One of those answers survives an audit. This is the reliability posture the production AI reliability practice is built to hold — sustained inspection-model uptime with accurate false-reject and escape tracking, which raw masks driving actuation cannot give you. What has to be in place before a SAM-derived component ships to a line? A promptable foundation model does not get a validation exemption. Anything SAM touches — an annotation set, a region-proposal stage — has to pass the same validation lens that turns “it worked in the pilot” into “this is releasable line-side.” Use this as a gate checklist. Annotation provenance recorded. If SAM proposed masks, log which masks were model-proposed versus human-drawn, and confirm every proposed mask was reviewed. A training set silently seeded with unreviewed SAM masks inherits the model’s class-agnostic blind spots. Decision ownership is the classifier, not SAM. Confirm no code path lets a raw mask reach reject actuation. The pass/fail boundary must live in a calibrated, version-pinned model. Classifier calibrated against your defect distribution. Not SAM’s training distribution — yours, including the benign process variations your line actually produces. Version pinning on the decision stage. The classifier weights are pinned and any change is a logged, revalidated event. Monitoring on the calibrated score. False-reject rate, escape rate, and score-distribution drift are tracked against a fixed baseline, not the mask. Lighting/material drift coverage. The validation set includes the drift conditions your line experiences, so a shift shows up as a monitored signal rather than a silent boundary move. Getting a SAM-derived component through this gate is the same discipline we apply to any line-side model, and it connects directly to how reliability gates belong at each stage of an ML pipeline — the proposal stage and the decision stage carry different gates precisely because they carry different risk. When is a purpose-trained segmentation model the better choice? SAM’s generality is exactly wrong for some inspection tasks. When your defect set is narrow, stable, and well-characterised — a fixed family of solder faults on a repeating board layout, say — a purpose-trained segmentation model fine-tuned on your labelled defects will usually beat SAM as a front end, because it segments your defects specifically rather than any region. It is smaller, faster on line-side hardware, and its behaviour is defined entirely by your data, which makes drift monitoring more tractable. SAM shines when your regions are diverse, novel, or expensive to label from scratch — which is why annotation acceleration is its strongest role. As the task narrows and stabilises, the case for a purpose-built model grows. Many programs use SAM to bootstrap the first labelled set, train a dedicated model on it, and then retire SAM to the annotation tool where its generality keeps paying off. If your defect set is stable enough that a detector is viable, our comparison of detectors that survive production line conditions is the better starting point than any foundation-model segmenter. FAQ What should you know about segment anything model sam in practice? SAM takes an image plus a prompt — a point, box, or coarse mask — and returns high-quality binary masks for the indicated region. It was trained on a large, diverse mask dataset, so it generalises to boundaries it has never seen. In practice it means you get an excellent region locator that carries no knowledge of your defect taxonomy. What does SAM actually output — promptable, class-agnostic masks — and why does that matter for defect inspection? SAM outputs promptable, class-agnostic masks with a confidence that the mask fits the boundary, not a probability that the region is a defect. That matters because a mask around a benign packaging change looks identical to a mask around a real defect. The model was never calibrated against your defect distribution or lighting drift, so its output cannot be read as a defect call. Where does SAM fit in an AOI pipeline: annotation acceleration, region proposal, or defect decision — and where should it not sit? Annotation acceleration is its safest role — a human confirms every proposed mask offline. Region proposal is next: SAM isolates candidate regions for a calibrated, version-pinned classifier that owns the decision. It should never sit as the direct pass/fail signal, because that couples reject actuation to an uncalibrated, unmonitored model. Why can’t raw SAM masks be used as a pass/fail rejection signal on a production line? Because raw masks carry no calibrated defect probability, lighting and material drift silently shift what SAM segments and therefore your effective decision boundary, with no alarm. Benign process changes produce clean masks a size threshold happily rejects. And with no calibrated score, you have nothing to monitor against escape rate over time. How do you combine SAM with a calibrated, version-pinned defect classifier so the reject decision stays measurable and monitorable? Run a two-stage pipeline: SAM proposes candidate regions (Stage 1, hypotheses only), then a version-pinned classifier scores each cropped region with a calibrated defect probability and applies the reject threshold (Stage 2). All monitoring — false-reject tracking, drift telemetry, score distributions — lives in the classifier. SAM can be upgraded with low blast radius because it never decides anything. What has to be in place before any SAM-derived component is releasable to a line-side AOI deployment? Annotation provenance recorded and every proposed mask reviewed; decision ownership held by the classifier, not SAM; the classifier calibrated against your own defect distribution; version pinning on the decision stage; monitoring on the calibrated score; and validation coverage for the lighting and material drift your line experiences. A promptable foundation model does not get a validation exemption. When is a purpose-trained segmentation model a better choice than SAM for a specific inspection task? When your defect set is narrow, stable, and well-characterised. A model fine-tuned on your labelled defects segments your defects specifically, runs faster on line-side hardware, and has behaviour defined by your data, which makes drift monitoring more tractable. SAM stays valuable for the diverse, novel, or hard-to-label regions where annotation acceleration pays off. The question to carry back to your own line is not “can SAM segment this defect?” — it almost always can. It is “which stage owns the reject decision, and is that stage calibrated against my defect distribution and monitored against my drift?” If the honest answer is a raw mask threshold, you have located a failure point before it located your escape rate for you.
