Point a team at the Segment Anything Model and ask it to “just detect the parts on the conveyor,” and you will hit the wall fast. SAM does not output labelled, oriented boxes. It outputs pixel masks. That single fact is where most SAM-for-inspection projects go wrong — and where the interesting engineering actually starts. The confusion is understandable. SAM produces tight, clean object outlines that look like detection results. But it is a promptable, class-agnostic segmentation model, not a detector. It will happily segment a rotated part on a conveyor without ever telling you what the part is or drawing the angle-aware box your downstream measurement stage needs. The gap is not a limitation to work around; it is a clue about where SAM belongs in the pipeline. What is the Segment Anything Model, and what does “promptable, class-agnostic segmentation” mean? SAM, released by Meta AI, is trained to answer one question: given a prompt, which pixels belong to the object being pointed at? The prompt can be a point, a box, or an automatic grid of points that tiles the whole image. From that prompt SAM emits one or more binary masks — pixel-precise regions — ranked by an internal confidence estimate. Two words in “promptable, class-agnostic segmentation” carry most of the meaning. Promptable means SAM does not decide on its own what to segment. You steer it. Click a point on a part and SAM returns the mask for that part. Draw a rough box and SAM tightens it into a pixel mask. Run automatic mode and SAM proposes masks for everything it can find. This is genuinely useful for annotation and interactive tooling, but it also means SAM has no notion of “the objects that matter” unless a prompt tells it. Class-agnostic means SAM never names what it segments. It knows this is a coherent object but not this is a bolt or this is a scratch. There is no label, no class probability, no category. That is a deliberate design choice — it makes SAM generalise to objects it never saw in training — but it is exactly why SAM cannot stand in for a classifier. This is the same property that makes SAM valuable in medical imaging as a segmentation front-end rather than a diagnostic model: the model finds the region; something else assigns meaning. So SAM gives you accurate geometry with no semantics. A detector, by contrast, gives you semantics (a class) plus coarser geometry (a box). The two are complementary, not interchangeable. How do you turn a SAM mask into an oriented bounding box? This is the mechanical heart of the article, and it is simpler than it sounds. A binary mask is a set of pixels. The tightest rotated rectangle that encloses that set is its minimum-area rotated rectangle — the rotated hull, computed directly from the mask contour. In OpenCV this is a two-call operation: cv2.findContours to get the outline, then cv2.minAreaRect to fit the rotated rectangle. The output is a centre point, a width and height, and an angle. That angle is the whole point. An axis-aligned bounding box (AABB) has no orientation — it only tells you the leftmost, rightmost, topmost and bottommost extents. A rotated part on a conveyor produces an AABB full of background, and two adjacent parts at an angle produce overlapping AABBs that a tracker cannot separate. The minimum-area rectangle derived from a SAM mask carries the object’s true angle for free, because the mask already traced the object’s true outline. If you want the full picture of why orientation changes the geometry problem, our explainer on when rotated bounding box detection actually matters in inspection covers the failure modes that AABBs create on packed, angled scenes. The mask-to-box conversion is where SAM earns its place in production geometry — but only if the conversion is validated. A few things bite in practice: Concave or holed masks. A minimum-area rectangle is a convex hull fit; a C-shaped or ring-shaped part will get a box far larger than the object. When the object’s shape violates the rectangle assumption, the derived box is misleading and you need a different geometry primitive. Multi-blob masks. Automatic mode can merge or split objects unpredictably. A single mask covering two touching parts yields one box spanning both — a silent measurement error. Angle wraparound. minAreaRect reports angles in a bounded range and the convention has changed across OpenCV versions; a box that is 89° or -1° is the same physical orientation, and code that does not normalise the angle will produce garbage rotation statistics. None of these are reasons to avoid the technique. They are the boundary conditions you validate before trusting the derived box. Where exactly AABB-versus-oriented geometry is decided in a pipeline is worth understanding on its own, and our note on where the detection head commits to box geometry makes the trade-off explicit. Can SAM replace an object detector, or does it still need a classification stage? No, and this is the misreading that sinks projects. SAM supplies orientation-bearing masks. It does not supply labels. A production inspection pipeline that must report “reject: part rotated beyond tolerance” needs three things — the object located, the object measured, and the object named — and SAM covers only the first two. The correct architecture puts SAM upstream of, or alongside, a classifier: Stage What it does Candidate technology Segmentation Produce a pixel mask per object from a prompt SAM (Segment Anything Model) Geometry Fit minimum-area rotated rectangle to the mask OpenCV minAreaRect Classification Assign a class label to the masked region A CNN classifier (e.g. a ResNet or EfficientNet head) Decision Apply tolerance rules to angle + class Application logic Read the table row by row and the division of labour is clear: SAM never claims a row it cannot fill. The classification stage is not optional overhead — it is the part that converts geometry into a decision. If you want the wider map of how detection, classification and mask stages relate, our overview of object recognition models and what actually feeds a tracker situates SAM in the broader detection-classification landscape. There is a second architecture worth naming, and it is the one that usually pays off. Rather than run SAM live in the pipeline, run it once, offline, to bootstrap an oriented-box dataset, then train a purpose-built rotated-box detector — the kind used in oriented bounding box detection for logistics CV — for production inference. This sidesteps SAM’s runtime cost entirely. When does SAM-bootstrapped annotation save cost versus manual angle labelling? Annotating oriented boxes by hand is slow and error-prone precisely because of the angle. A human drawing an axis-aligned box only marks four extents; drawing a rotated box means judging an orientation by eye, and inter-annotator variance on that angle is real. SAM changes the economics: a single point-click produces a mask, and the mask’s rotated hull encodes the orientation without anyone drawing an angle. The annotator’s job collapses from “trace an oriented rectangle” to “click the object and confirm.” The measurable payoff shows up in two places. First, faster oriented-box dataset bootstrapping — the annotation throughput on angled targets rises when the angle is derived rather than drawn (observed pattern across our CV data-preparation work; not a benchmarked rate, and it depends heavily on object shape regularity). Second, tighter box-to-object area ratios on rotated targets, because a mask-derived rectangle hugs the object rather than the loose enclosure a hurried human produces. Against that upside, quantify the costs honestly before committing: Inference cost. The original SAM image encoder is a heavy ViT. Running it per frame at line speed is often infeasible; this is a real reason the offline-bootstrap architecture wins for production. Lighter variants such as MobileSAM and FastSAM exist and trade mask quality for speed, and they change this calculation — but you validate the mask quality on your parts before assuming the trade is acceptable. Prompt-engineering overhead. Automatic mode is not free. Getting one mask per object — not per screw-thread, not per shadow — takes tuning of the point grid, the confidence threshold, and post-filtering. This is engineering time, not a config toggle. The class-agnostic gap. Say it again because it is the load-bearing constraint: SAM does not name what it segments, so a classification stage is still required. Any cost model that forgets this is comparing SAM against a detector it cannot replace. The honest summary is a conditional. SAM-bootstrapped annotation pays off when your targets are rotated, roughly convex, and expensive to angle-label by hand, and when you plan to train a fast production detector on the result rather than run SAM live. It does not pay off when parts are axis-aligned (an AABB detector is simpler), when shapes are concave enough to break the rectangle fit, or when you expected SAM to be the whole pipeline. FAQ What’s worth understanding about SAM first? SAM takes a prompt — a point, a box, or an automatic grid — and returns pixel-precise binary masks for the object being pointed at, ranked by an internal confidence estimate. In practice it means you get accurate object geometry with no class label: the model finds the region, but something else must decide what that region is and turn it into the box a downstream stage needs. What is the Segment Anything Model (SAM) and what does “promptable, class-agnostic segmentation” mean? SAM is Meta AI’s foundation segmentation model. “Promptable” means you steer what it segments with a point, box, or grid rather than it deciding autonomously. “Class-agnostic” means it never names what it segments — it identifies a coherent object without assigning a category — which is why it generalises well but cannot replace a classifier. How do you turn a SAM mask into an oriented (rotated) bounding box for detection? Take the binary mask, extract its contour (cv2.findContours), and fit the minimum-area rotated rectangle to it (cv2.minAreaRect). The result gives a centre, width, height, and angle — the angle comes for free because the mask already traced the object’s true outline. Validate against concave masks, multi-blob masks, and angle-wraparound conventions before trusting the box. Can SAM replace an object detector, or does it still need a classification stage? It cannot replace a detector. SAM supplies orientation-bearing masks but no labels, so a production pipeline still needs a classification stage to name the masked region before any tolerance decision. The correct pattern places SAM upstream of a classifier, or uses it offline to bootstrap a dataset for a purpose-built detector. When does using SAM to bootstrap oriented-box annotations save cost versus manual angle labelling? It pays off when targets are rotated, roughly convex, and costly to angle-label by hand, and when you plan to train a fast production detector on the derived dataset rather than run SAM live. It does not pay off for axis-aligned parts, concave shapes that break the rectangle fit, or when SAM was expected to be the whole pipeline. What are the inference and prompt-engineering costs of putting SAM in a production inspection pipeline? The original SAM image encoder is a heavy ViT that is often infeasible per frame at line speed, which is why offline bootstrapping usually wins; lighter variants like MobileSAM and FastSAM trade mask quality for speed. Automatic mode also needs prompt tuning — point grid, confidence threshold, and post-filtering — to yield one mask per object, and the class-agnostic gap means a classification stage is always an added cost. Where this leaves the build decision SAM’s value in an inspection pipeline is not that it detects objects — it does not — but that it hands you orientation you would otherwise pay a human to draw. The moment you internalise that SAM produces masks and a rotated rectangle produces the box, the architecture designs itself: segment, fit, classify, decide. The question that separates a working deployment from a stalled one is not “can SAM do detection?” but “does my production geometry actually diverge from what an off-the-shelf detector assumes?” — and if the answer is yes, whether that divergence is best served by a SAM-bootstrapped dataset or by SAM in the live loop. Our work on computer vision systems starts from exactly that assessment: whether the geometry your parts demand is the geometry your detector was trained to produce.
