“We can drop SAM into the imaging pipeline and skip most of the annotation work.” That sentence is where a lot of medical-imaging projects quietly acquire a regulatory problem. The Segment Anything Model is a genuinely useful tool, but the assumption baked into that sentence — that a foundation model which “segments anything” can stand in for validated clinical inference — is where the naive read and the expert read part ways. Here is the correct framing up front: SAM is a promptable, class-agnostic segmentation model trained largely on natural images. It turns point or box prompts into masks. It does not know clinical anatomy, it does not attach pathology labels, and its raw zero-shot output on a CT slice or an ultrasound frame is not validation evidence in an FDA Software-as-a-Medical-Device (SaMD) context. Used as an annotation accelerator with expert review, SAM sits on a defensible path. Treated as a diagnostic result, it becomes an unvalidated, undocumented model change inside a regulated pipeline. This piece walks through what SAM actually does, where it fits in a medical-imaging workflow, and where the regulatory line falls — because the technical capability and the regulatory meaning are two separate questions, and conflating them is the failure we see most often. What does working with SAM (the Segment Anything Model) involve in practice? SAM has three parts that matter for understanding its behaviour. An image encoder — a Vision Transformer backbone — runs once per image and produces a dense embedding. A prompt encoder turns a user’s input (a click point, a bounding box, or a coarse mask) into a compact representation. A lightweight mask decoder then fuses the image embedding and the prompt embedding to produce one or more candidate masks, each with a predicted quality score. The design choice that made SAM useful is the separation between the heavy image encoder and the cheap decoder. Once you have run the encoder on an image, you can throw many prompts at the decoder interactively without re-encoding — which is exactly why it feels responsive as an annotation tool. It was trained on the SA-1B dataset, on the order of a billion masks across roughly eleven million images (per Meta AI’s published specifications for the release). Those images are overwhelmingly natural scenes, not medical scans. What that means in practice: SAM answers the question “what is the object boundary around this point?” It does not answer “is this a tumour?” or “is this the left ventricle?” The model has no concept of the labels a clinician cares about. It produces geometry, and a human — or a downstream classifier — supplies meaning. That distinction is the whole article. What is promptable, class-agnostic segmentation, and how do point and box prompts drive the mask? “Promptable” means the model’s output is conditioned on an input you provide at inference time rather than on a fixed set of learned categories. You click a point inside a structure, SAM proposes the mask it thinks that point belongs to. You draw a bounding box, SAM proposes the mask that best fills it. Add a second point tagged as “foreground” or “background” and the mask refines. This interactive loop is what separates SAM from a conventional trained segmentation network like a U-Net, which outputs a fixed label map for classes it was trained on. “Class-agnostic” is the other half. A U-Net trained on liver, spleen, and kidney will output exactly those classes and nothing else. SAM outputs a mask for whatever the prompt points at, with no class attached. This is a strength for annotation — you are not limited to a predefined ontology — and a liability for diagnosis, because the model is not asserting what the region is. It is a segmentation model without semantics. A useful mental model: SAM is closer to an intelligent lasso tool than to a diagnostic classifier. It is very good at “trace the boundary of the thing I am pointing at.” It is silent on “and here is what the thing is, and here is my confidence that it is pathological.” If you need the second sentence, SAM alone does not give it to you — you need a trained, validated model downstream, and the way its promptable masks can feed a detection or tracking stage is a design question we cover in how promptable masks feed segmentation-derived detection. How well does SAM generalise to CT, MRI, and ultrasound? This is the question that decides whether SAM is safe to use, and the honest answer is: it depends heavily on the modality, and you must characterise it rather than assume it. SAM’s training distribution is natural photographs — well-lit, high-contrast, RGB scenes with clear object boundaries. Medical modalities violate almost every one of those assumptions. CT is single-channel with Hounsfield-unit intensity ranges that carry clinical meaning; MRI contrast depends on the acquisition sequence; ultrasound is speckle-dominated with low signal-to-noise and operator-dependent framing. When SAM meets these, it is operating out-of-distribution, and out-of-distribution behaviour is not a single number — it is a per-modality, per-structure characterisation. In the patterns we see, SAM’s zero-shot masks are often reasonable on structures with strong, contiguous boundaries — a well-delineated organ on a contrast-enhanced CT, for instance — and unreliable on low-contrast lesions, fine vascular structures, or anything where the boundary is ambiguous even to a radiologist (observed across medical-imaging engagements; not a published benchmark on your data). The practical consequence: a mask that looks plausible on inspection can still be systematically wrong on the exact cases that matter clinically. That is why federated, generalisability-focused evaluation matters here, and why we point teams toward the kind of cross-site testing described in MedPerf’s federated benchmarking for medical AI generalisability before trusting any zero-shot behaviour. Annotation accelerator or diagnostic model? A decision rubric The single most consequential decision with SAM in medical imaging is what role you assign it. The two roles carry completely different evidence burdens. Dimension SAM as annotation accelerator SAM as diagnostic inference What its output is A draft mask a human reviews and corrects A result acted on clinically Regulatory status Tooling used to build a dataset A model claim requiring validation Evidence burden Inter-rater agreement on final labels Full SaMD validation on locked model Failure mode if wrong Caught in expert review Reaches the patient Defensible today? Yes, with human-in-the-loop No, not from zero-shot output Where the ROI lives Faster dataset preparation There is no shortcut here Read the table as a fork, not a spectrum. The moment SAM’s output influences a clinical decision without a validated, locked model between it and the clinician, you have crossed from tooling into inference — and everything about your regulatory obligations changes. Keeping SAM firmly in the left column is what preserves the defensible path. The ROI is real and worth naming precisely. Using SAM as a promptable pre-annotation tool can cut manual segmentation labelling effort by roughly 40–70% on suitable modalities (observed range across annotation-acceleration engagements; not a benchmarked rate on your data), which shortens the path to a validation-ready training set. It does not remove expert review and it does not reduce the FDA validation evidence burden. The saving is in dataset preparation speed, not in bypassing clearance. What does using SAM mean under FDA SaMD rules, and why is its zero-shot output not validation evidence? Under a SaMD framing, the cleared device is a specific, locked model with documented performance on defined patient populations and modalities. Validation evidence is what demonstrates that this locked model performs as claimed. A zero-shot mask from a promptable foundation model is none of those things: it is not locked (the output depends on the prompt a user supplies), it is not characterised on your patient population, and it carries no documented performance envelope. Feed SAM’s raw output into a diagnostic pathway and you have effectively introduced an undocumented model whose behaviour changes with every click. From a regulatory standpoint that is an unvalidated model change with no traceability — the opposite of what a submission needs. The correct place for SAM is inside the dataset-preparation stage: it helps you build the training and test sets faster, and the model you actually validate and submit is the downstream one trained and evaluated on those sets under a fixed protocol. This is why SAM adoption is a regulatory-pathway decision, not just an engineering one. The choice of where SAM sits in the pipeline should be made during the pathway assessment — where its zero-shot outputs are explicitly scoped as annotation tooling rather than cleared inference — and it connects directly to how you build defensible validation evidence, a subject we treat in depth in A/B testing statistics for clinical CV models that hold up to FDA review and in how detection accuracy maps to FDA validation evidence via [email protected] in medical CV. SAM decisions belong inside our computer vision practice’s regulatory-pathway work, not bolted on after the fact. How do medical-specialised variants like MedSAM differ from base SAM, and when are they worth adopting? Medical-specialised variants — MedSAM being the best-known — take SAM’s architecture and fine-tune it on large collections of medical images across modalities. The intent is to close the out-of-distribution gap: a model that has seen CT, MRI, ultrasound, and pathology slides during training produces more reliable masks on those modalities than base SAM does. The important caveat is that fine-tuning improves the segmentation geometry on medical data; it does not change the regulatory status of the output. MedSAM is still promptable and still class-agnostic in the clinically relevant sense — it draws better boundaries, but it does not turn a zero-shot mask into validation evidence. It is a better annotation accelerator, not a shortcut past validation. When is it worth adopting? When base SAM’s zero-shot masks on your target modality are unreliable enough that annotators spend more time correcting than they would drawing from scratch, a fine-tuned variant usually pays for itself. When you have a domain gap that base SAM cannot bridge — low-contrast lesions, unusual acquisition protocols — fine-tuning on representative data is the lever. The mechanics of adapting SAM to a specific medical segmentation task, including the data and compute trade-offs, are the subject of fine-tuning the Segment Anything Model for medical segmentation. What human-in-the-loop review is required before SAM masks enter a training set? The review is not a formality — it is the mechanism that keeps SAM on the defensible side of the line. A workable pattern has three checkpoints: Boundary correction. A qualified annotator reviews every SAM-generated mask and corrects it. The mask is a draft, never a final label. This is where SAM’s speed advantage is real and where it is safe. Semantic labelling. Because SAM is class-agnostic, a human or a validated downstream step must attach the clinical label. SAM tells you where; the reviewer tells you what. Agreement measurement. Track inter-rater agreement on the corrected labels. If SAM’s drafts systematically bias annotators toward a particular boundary, that shows up as reduced independence between raters — a real risk with any pre-annotation tool, and one worth measuring rather than assuming away. The engineering tooling — versioning the masks, tracking which were SAM-drafted versus hand-drawn, recording the prompt used — matters because reproducibility of the dataset is part of the evidence trail. If a reviewer later asks how a given label was produced, “SAM drafted it, annotator X corrected it on this date under this protocol” is an answerable question only if you instrumented for it. FAQ What should you know about SAM (the Segment Anything Model) in practice? SAM uses a heavy Vision Transformer image encoder that runs once per image, a lightweight prompt encoder for your clicks or boxes, and a fast mask decoder that fuses the two into candidate masks with quality scores. In practice it answers “what is the boundary around this point?” — it produces geometry, not clinical meaning. A human or downstream model must supply what the region actually is. What is promptable, class-agnostic segmentation, and how do point and box prompts drive SAM’s mask output? Promptable means the output is conditioned on the input you give at inference time — a point, a box, or a coarse mask — rather than on fixed learned categories. Class-agnostic means SAM masks whatever the prompt points at without attaching a label. A point proposes the mask that point belongs to; a box proposes the mask filling it; adding foreground/background points refines the result. How well does SAM generalise to medical modalities like CT, MRI, and ultrasound compared to natural images? SAM was trained on natural photographs, so CT, MRI, and ultrasound are all out-of-distribution and its behaviour must be characterised per modality rather than assumed. Zero-shot masks tend to be reasonable on high-contrast, contiguous structures and unreliable on low-contrast lesions, fine vasculature, and ambiguous boundaries — often exactly the clinically important cases. Federated, generalisability-focused evaluation is the right way to test this. Where does SAM fit as an annotation accelerator versus a diagnostic inference model in a medical-imaging pipeline? SAM belongs in the dataset-preparation stage as a promptable pre-annotation tool whose drafts a human reviews and corrects. That role is defensible today. The moment its output influences a clinical decision without a validated, locked model in between, it crosses into diagnostic inference — and inherits the full SaMD validation burden. Keep it firmly in the annotation role. What does using SAM mean under FDA SaMD rules, and why is its raw zero-shot output not validation evidence? Under SaMD, the cleared device is a specific locked model with documented performance on defined populations and modalities. A zero-shot SAM mask is not locked (it varies with the prompt), not characterised on your patient population, and carries no documented performance envelope — so it is not validation evidence. Feeding raw SAM output into a diagnostic pathway introduces an unvalidated, untraceable model change. How do medical-specialised variants (e.g. MedSAM) differ from base SAM, and when are they worth adopting? MedSAM and similar variants fine-tune SAM’s architecture on large medical-image collections, producing more reliable boundaries on CT, MRI, ultrasound, and pathology than base SAM. Fine-tuning improves segmentation geometry but does not change regulatory status — a MedSAM mask is still a promptable draft, not validation evidence. Adopt it when base SAM’s masks on your modality are unreliable enough that annotators spend more time correcting than drawing. What human-in-the-loop review is required before SAM-generated masks enter a training set or clinical workflow? Every SAM mask needs boundary correction by a qualified annotator (the mask is always a draft), semantic labelling by a human or validated step (since SAM is class-agnostic), and inter-rater agreement measurement to catch pre-annotation bias. Instrument the tooling so each label’s provenance — SAM-drafted, corrected by whom, under which protocol and prompt — is recoverable, because dataset reproducibility is part of the evidence trail. The question worth carrying out of this piece is not “can SAM segment my scans?” — it usually can, well enough to be useful — but “which stage of my pipeline is SAM allowed to touch, and what documentation follows it there?” Answer that inside the regulatory-pathway assessment, keep the model firmly in the annotation role, and the ROI lands without importing an unvalidated model change. The failure class to name in a review is zero-shot output treated as cleared inference; the artifact that prevents it is a pathway assessment that scopes SAM as tooling from the start.
