SAM Models Explained: Segment Anything in Medical Imaging Pipelines

What the SAM model is, how SAM, SAM 2 and SAM 3 differ, where zero-shot SAM fails on medical images, and why it is a labelling aid, not a diagnostic tool.

SAM Models Explained: Segment Anything in Medical Imaging Pipelines
Written by TechnoLynx Published on 11 Jul 2026

The SAM model (Segment Anything Model) is Meta’s promptable segmentation foundation model. You give it an image and a point, a box or a rough mask, and it returns a mask for the object you indicated, even for objects and image types it never saw in training. Used zero-shot on medical images, SAM helps annotation but is unstable as a stand-alone segmenter. It handles targets with distinct boundaries and struggles with the weak boundaries and low contrast that are typical of medical targets. No SAM-family model, medical fine-tunes included, turns a labelling tool into a validated diagnostic component.

Misreading that distinction is the most common way SAM disappoints in medical imaging. Our companion post describes the pattern: a team drops SAM into a CT pipeline in place of a task-specific U-Net, and the masks come back plausible-looking but clinically wrong at the organ boundaries that matter, because a natural-image foundation model was treated as a validated diagnostic component. The rest of this piece covers what the model family is, where it breaks on medical data and what it takes to use it safely.

What is a SAM model, and what does promptable, zero-shot segmentation mean in practice?

A SAM model treats segmentation as a prompting problem. It does not assign pixels to a fixed list of classes. It produces a mask for whatever object the user points at. Meta introduced the project in April 2023, training SAM on the SA-1B dataset of over 1.1 billion masks. The original paper describes those masks as drawn from 11 million licensed, privacy-respecting images.

“Zero-shot” means the model is applied to new image distributions and tasks without retraining. On natural-image benchmarks the authors found that zero-shot performance was often competitive with fully supervised results, and sometimes better.

The catch for medicine is the training data. According to Huang et al. (Medical Image Analysis, 2024), SA-1B contains natural photographs of places, objects and scenes but no medical images. Medical images are mostly greyscale, with unclear boundaries, similar foreground and background, and a wide range of object sizes, including some very small objects. SAM can also run without any prompt in an “Everything” mode, but the same study found it performed better with manual hints, especially boxes, than in that automatic mode.

How SAM’s image encoder, prompt encoder and mask decoder produce a mask

The SAM model architecture has three parts. The image encoder is a Vision Transformer pre-trained as a masked autoencoder. In the ViT-Base configuration that MedSAM inherits, it has 12 ViT blocks, each with multi-head self-attention, an MLP and normalisation layers. The prompt encoder turns user prompts into embeddings. The mask decoder fuses those embeddings with the image features through cross-attention to produce the mask.

Prompts come in two kinds. Sparse prompts are points, boxes and text. Dense prompts are masks. A point is encoded with learnable tokens that mark it as foreground or background. A box is encoded from its top-left and bottom-right corners. Free-form text goes through CLIP’s pre-trained text encoder. A mask prompt is embedded with convolutions and added element-wise to the image embedding. The decoder returns several candidate masks per prompt, and not all of them contain the intended object, so evaluation and tooling need a step that chooses one.

SAM’s image embedding can be computed once per image and reused across prompts. Huang et al. found that SAM’s original code re-encoded the same image for every prompt, and caching the embeddings made their multi-prompt evaluation nearly n times faster. Once the embedding exists, manual prompt encoding and mask decoding each took under 0.01 seconds, fast enough for real-time interaction.

ViT-B, ViT-L or ViT-H: choosing a SAM model type

Huang et al. compared SAM’s small ViT-B encoder with its large ViT-H encoder. On medical images, they found that ViT-H performed better overall than ViT-B and was more stable, with smaller standard deviations on targets such as kidney CT and prostate MRI. ViT-H also took nearly four times as long as ViT-B to compute an image embedding.

The accuracy gap depends on the prompt. With point prompts, ViT-H’s advantage grew as more points were added. With box prompts, the two encoders differed by only 0.37% and 0.06% Dice. Our sources do not measure ViT-L separately, so we cannot give a figure for it.

For scale, MedSAM’s ViT-B image encoder has 89,670,912 trainable parameters. MedSAM2 went smaller still and used the SAM2.1-Tiny variant, which its authors found competitive with fewer parameters. If boxes are your main prompt, we would start with the smallest encoder and pay for ViT-H only if point-driven interaction is central to the workflow, because with boxes the measured Dice gap was under 0.4%, and ViT-H’s gains came with point prompts and in stability.

What did SAM 2 and SAM 3 add over the original SAM?

Version Released What it added Prompts
SAM April 2023 Promptable image segmentation trained on SA-1B Points, boxes, rough masks
SAM 2 29 July 2024 Hierarchical Hiera backbone, plus a streaming memory (memory encoder, memory bank, memory attention) for tracking objects through video. An image is treated as a single-frame video Geometric prompts on any frame
SAM 3 19 November 2025 Promptable Concept Segmentation: detect, segment and track every instance of a concept. An image-level detector and a memory-based tracker share one backbone, with a presence head separating recognition from localisation Short noun phrases, image exemplars, points, boxes, masks
SAM 3.1 Described by Meta as a drop-in replacement for SAM 3 Object multiplexing that tracks up to 16 objects in one forward pass As SAM 3

On speed and accuracy, the SAM 2 paper reports better video segmentation accuracy with three times fewer interactions than earlier approaches, and image segmentation that is more accurate and six times faster than SAM. Meta reported roughly 44 frames per second for SAM 2 on image segmentation. For SAM 3, the paper reports double the accuracy of existing systems on concept segmentation in images and video, trained on a data engine that produced 4 million unique concept labels. Meta says SAM 3.1 doubles throughput from 16 to 32 frames per second on a single H100 for videos with a medium number of objects. That is Meta’s own figure.

Can SAM do semantic segmentation, or does it only produce class-agnostic masks?

The original SAM produces class-agnostic masks. Huang et al. note that SAM finds objects but outputs no category: each mask is simply “object or not”. To get semantic segmentation you add a component that supplies the class. One option crops SAM’s region proposals and classifies them with CLIP. Another runs an open-vocabulary detector such as Grounding DINO, which outputs labelled boxes that SAM then turns into masks (Grounded-SAM).

SAM 3 changes the picture by adding concept prompts. A concept prompt returns masks and unique identities for every matching instance. Fine-grained medical concepts are where it struggles. Meta itself says SAM 3 struggles to generalise zero-shot to fine-grained, out-of-domain concepts such as “platelet” in medical or scientific imagery. The MedSAM3 authors found that the prompt “liver” on the LiTS dataset often segmented the lungs and ignored the liver. On a nucleus segmentation dataset, “nucleus” was not recognised at all while the generic “cell” worked. Even after fine-tuning on medical concept labels, text-only inference stayed unreliable, and adding a bounding box gave the best results on all four benchmarks.

In a medical pipeline, then, we would source the class label outside SAM: a trained detector, a clinician’s box or a concept model fine-tuned on your terminology. SAM supplies the boundary.

How well does SAM segment CT, MRI, ultrasound and histology out of the box?

Results vary widely, tracking the target’s boundary and contrast rather than the modality. Mazurowski et al. tested SAM on 19 public medical imaging datasets. With single prompts, IoU ranged from 0.1135 for spine MRI to 0.8650 for hip X-ray. In the MedSAM evaluation, SAM scored lowest on most tasks. It did well on some RGB tasks, such as polyp segmentation in endoscopy (Dice 91.3%), which the authors attribute to its training on RGB photographs.

Huang et al. measured which object characteristics drive SAM’s performance. Across 191,779 anatomical structures, Dice correlated moderately with boundary complexity and with the intensity difference between foreground and background, weakly with size, and not at all with modality or aspect ratio. In practice, SAM does well on some structures and fails outright on others. Weak boundaries produce under-segmentation and over-segmentation. Dense instances are another hard case: in digital pathology whole-slide images, SAM did not reach satisfactory performance even with 20 clicks or boxes per image. On the SegThy thyroid and neck-vessel ultrasound dataset, both SAM 2 and SAM 3 failed almost completely with click prompts, often scoring single-digit Dice.

The counterweight is real too. A zero-shot study of X-ray, ultrasound, dermatoscopy and colonoscopy found that SAM’s performance was comparable to the state of the art and in some cases better. SAM’s point-based multi-round iteration on medical images is also weak. As clicks accumulate, its performance rises slowly or even drops, while dedicated interactive methods keep improving. One review names the domain gap between natural and medical images as the most critical factor limiting SAM. Our companion post goes further into how SAM fails at organ boundaries in CT.

MedSAM, Medical SAM Adapter, SAM-Med3D and MedSAM2: what each changes

Variant What it changes versus SAM Evidence reported Known limits
MedSAM Fine-tunes SAM ViT-B’s image encoder and mask decoder with box prompts; the prompt encoder stays frozen 1,570,263 image-mask pairs, 10 modalities, over 30 cancer types; 86 internal and 60 external validation tasks Modality imbalance toward CT, MRI and endoscopy; branching vessels where a box is ambiguous
Medical SAM Adapter (Med-SA) Adds adapters rather than fine-tuning the whole model: Space-Depth Transpose for 3D images and a Hyper-Prompting Adapter Beats several state-of-the-art methods on 17 tasks while updating 2% of parameters Not stated in the source
SAM-Med3D A fully learnable 3D architecture trained from scratch on volumes SA-Med3D-140K: 22K images, 143K masks, 28 modalities; 60.12% Dice improvement over SAM with one point per volume On unseen ultrasound, comparable only with enough prompts
MedSAM2 Full fine-tune of SAM2.1-Tiny for 3D and video Over 455,000 3D image-mask pairs and 76,000 frames Box prompts limit thin branching vessels; fixed eight-frame memory
MedSAM-3 Fine-tunes SAM 3’s detector on medical concept phrases; image and text encoders frozen Text plus box beats text only on all four benchmarks Text-only prompting remains insufficient

In the MedSAM study, MedSAM also generalised more consistently than the task-specific U-Net models. Task-specific U-Nets did well on internal validation and dropped sharply on external sets, while MedSAM held steady across both without extra fine-tuning. Its authors say its two weaknesses (less-represented modalities such as mammography, and vessel-like structures) can be addressed by further fine-tuning. SAM-Med3D’s authors tried three routes to 3D: adapters on a frozen 2D encoder were weaker on seen targets, full fine-tuning generalised poorly to unseen ones, and training from scratch gave the best average.

Why box prompts beat point prompts for organs, tumours and lesions

Boxes give the model an unambiguous region. Points can be ambiguous, particularly when nearby structures look alike. The MedSAM authors add that boxes are also quick to draw when several objects need segmenting. Huang et al. measured the effect. Adding points improved Dice steadily (ViT-B from 56.02% to 63.81%, ViT-H from 56.78% to 71.61%), but a single box performed best, and adding a point to the box changed almost nothing. A survey of SAM in medical imaging reports the same pattern: point prompting fell short of the state of the art by 20.3% to 40.9% average Dice, while box prompts stayed highly competitive even with moderate jitter.

Boxes are not free of risk. Box offsets of 20 to 30 pixels reduced SAM’s performance by 24.11%, so box quality from a tired annotator or an upstream detector matters. For branching vessels the box itself becomes ambiguous: in eye fundus images, arteries and veins share the same box. Scribbles are useful for those irregular, non-compact targets. SAM 3 narrows the click gap, giving much stronger single-click initialisation than SAM 2 on most structures, yet boxes still produced the highest absolute accuracy for both models.

The MedSAM human study shows a workflow-friendly compromise. Radiologists drew the long and short tumour axes every 3 to 10 slices, as they already do for tumour response evaluation. Those lines were converted into boxes, interpolated for the slices in between and passed to MedSAM. We would design annotation tooling the same way, building prompts from marks clinicians already make rather than asking them to learn a new gesture.

Segmenting 3D volumes and medical video by treating slices as frames

Running 2D SAM slice by slice works, but it misses correlations between slices, gives inconsistent 3D results and needs a prompt on every slice. SAM 2’s memory module gives stable slice-to-slice propagation instead. A memory bank stores features and predictions from past frames, and memory attention uses them to carry an initial mask through a volume or video. MedSAM2 places a box on the middle slice and propagates the mask towards both ends of the volume. The SegmentWithSAM extension for 3D Slicer does the same from point prompts, propagating in one or both directions.

High initialisation accuracy does not guarantee propagation success. In a November 2025 comparison across 16 datasets, SAM 3 segmented the MRI bladder well on the prompt frame, then collapsed to 7.29% Dice across the full volume, while SAM 2 held 76.91%. On SegThy carotid ultrasound the pattern reversed. SAM 2 initialised better and then failed during propagation (5.98% Dice on the left carotid), while SAM 3 held 56.65%. The authors recommend SAM 3 for click-driven work. With boxes or masks they recommend SAM 2 for compact organs and SAM 3 for vessels and irregular soft tissue. They disabled SAM 3’s concept prompts for this test, so it says nothing about text prompting.

Video brings its own limit. MedSAM2’s eight-frame memory bank can lose a colonoscopy polyp that leaves the field of view and later reappears. Because of failures like these, reviewers should check the whole propagated volume or clip, not only the slice that was prompted.

When should you fine-tune SAM instead of using it zero-shot?

We would use SAM zero-shot, with boxes, when the target has clear boundaries and your held-out images show it meets the accuracy you need. Fine-tune when the target has weak boundaries or low contrast, the modality is under-represented, or the structure is dense, branching or volumetric, and the zero-shot numbers on your images fall short. Running that check before anything enters an annotation workflow is the point. TechnoLynx tests SAM’s fit for a specific modality and structure before it goes into an annotation or augmentation workflow, through its Computer Vision R&D work.

Situation Starting point Reason
Clear-boundary targets, box prompts available Zero-shot SAM or MedSAM Strong results where boundaries are distinct
Limited compute, moderate domain gap Parameter-efficient fine-tuning (adapters, LoRA, decoder-only) Much lower compute and fewer trainable parameters, better boundary IoU, IoU close to CNN baselines
Very little labelled data Lightweight adaptation of a medical base model MedSAM-CA fine-tuned 7M parameters and reached 94.43% Dice on melanoma with 2% of the training data, against 90.27% for U-Net
Accuracy first, compute available Full fine-tuning Pronounced gains, but substantial parameter and compute demands
Volumetric targets A native 3D model such as SAM-Med3D Training 3D from scratch beat 2D-to-3D adapters on average

Even decoder-only tuning helps. Fine-tuning only SAM’s mask decoder raised average Dice by 4.39% for ViT-B and 6.68% for ViT-H. Specialising also costs breadth. SAM-Med3D’s authors found that fine-tuning on specific datasets improved the targeted structures but reduced performance on others. Gu et al. compared 18 fine-tuning combinations on 17 radiology datasets and released MRI-specific weights that consistently beat the original SAM. For SAM 3, Meta reports that the model adapts quickly to new visual domains with small amounts of annotated data and has released fine-tuning code.

Measuring “falls short” needs care. Compute metrics per image before aggregating, pair an overlap metric such as Dice with a boundary metric such as normalised surface distance, and keep the test images independent of anything used for tuning.

How does SAM speed up annotation and dataset augmentation for medical image and video datasets?

SAM speeds up annotation by turning drawing into correction. A human gives a coarse prompt, the model proposes a mask and the human edits it. The reported savings are large but come from different study setups. In the MedSAM study, two radiologists annotating an adrenal tumour type that MedSAM had never seen (10 cases, 733 slices) cut their time by 82.37% and 82.95%. Three doctors using SAM with box prompts in Huang et al. annotated about 25% faster, with higher quality.

The largest reported reductions came from closing the loop. CT lesion annotation fell from 525.9 seconds per lesion by hand to 74.3 seconds by the third round. Echocardiogram annotation fell from 102.3 to 8.4 seconds per frame, a 92% reduction. Meta reports that its SAM 3 data engine was about five times faster than humans on negative prompts and 36% faster on positive ones.

The review loop that keeps masks trustworthy

The MedSAM2 lesion pipeline shows where people stay in the loop:

  1. An annotator draws a 2D box on the slice where the lesion is widest.
  2. The model proposes a 2D mask, and the annotator corrects it and marks the top and bottom slices.
  3. The model propagates the corrected mask through the volume.
  4. The annotator corrects the 3D mask.
  5. Once dozens of new annotations are done, the model is fine-tuned for six to fifteen epochs and the next round starts.

SAM masks need human review before they become training labels. Near-100% accuracy is almost never a plausible expectation, so the workflow has to tolerate faults and the team has to be willing to keep people in the loop. The research literature already includes frameworks for detecting and correcting noisy SAM-derived pseudo-labels. SAM can also triage: it produces a first-pass segmentation and flags the complex cases for closer clinical scrutiny. We would treat correction effort per structure as a quality signal. A structure that keeps needing heavy edits is telling you to fine-tune or to stop using SAM for it.

How does SAM fit alongside diffusion and GAN-based synthetic imaging in an augmentation workflow?

SAM produces masks, not images. Diffusion models and GANs synthesise images, and SAM draws masks on images that already exist. In a dataset-augmentation programme they do different jobs. SAM-assisted labelling grows the pool of annotated scans: the MedSAM2 pipeline enabled dataset expansion by more than four times. Synthetic imaging adds variety on top of that pool. We would give masks on synthetic images, or checks that a generated lesion lands where it should, the same human review as masks on real scans. A plausible mask on an implausible image is still a defect. Where generative imaging already earns its place in life sciences is covered in our piece on generative AI in drug discovery and medical imaging, and the wider synthesis work sits in our generative and agentic AI R&D.

Running SAM in a production pipeline: weights, inference cost and edge limits

The SAM models are released under the Apache 2.0 licence, and several medical fine-tunes publish their code and weights openly, Gu et al.’s MRI weights among them. Cost depends on how the encoder is used. Cache one embedding per image and the interactive prompt-and-decode step becomes cheap. The automatic Everything mode is expensive, because it processes hundreds of sampled points and then heavy post-processing. Lightweight adaptations can add very little: MedSAM-CA added 0.0211 seconds per image over MedSAM.

Edge deployment is where it hits a limit. MedSAM2’s authors note that even on the SAM2.1-Tiny model with reduced input size, inference still needs a GPU. That limits its use on point-of-care ultrasound machines and low-power workstations until compression, quantisation or distillation make CPU inference workable. Their 3D Slicer plugin uses a client-server design so inference can run locally or on a remote cluster. We cover FastSAM for medical imaging and what it means for a validation pack separately, including the per-stratum metrics a faster model has to pass. Whatever the variant, we would deploy exactly the checkpoint that was validated. The FDA’s 2025 draft guidance asks sponsors to account for any difference between the tested and released model versions. Pinning the deployed checkpoint to the validated version is part of MLOps engineering.

Labelling accelerator or diagnostic tool: where the validation line falls

A SAM-family model is a segmentation accelerator and a labelling force-multiplier; its masks become clinical evidence only once the device that uses them has been validated for that purpose. The two uses need different evidence.

As a labelling aid, a person corrects each proposed mask, and the output is a training label. We would check it with engineering measures: correction effort, agreement with expert masks on a held-out set, and a provenance record per label. Those records still matter if the model trained on them later becomes a device. The FDA’s 2025 draft guidance treats data management, including how data was annotated, as critical to understanding how an AI-enabled device was developed and validated. It also names segmentation contours as one of the clinical tasks a reference standard must reflect.

As a clinical output, the bar is the one for software as a medical device. The IMDRF clinical evaluation framework (2017) asks for three things: a valid clinical association between output and condition, analytical validation that the software turns inputs into accurate, reliable output, and clinical validation that the output achieves its purpose in the target population. Independent review of that evidence should scale with risk. Where a human interprets the output, the FDA draft points to the performance of the human-AI team rather than the model alone, and reader studies are typically the primary evaluation for imaging aids. After deployment, performance can drift as populations and input data change, so it needs ongoing monitoring. Metrics Reloaded makes the same ordering explicit: technical validation comes first and may be followed by clinical validation against patient outcomes. TechnoLynx can build the engineering side of that evidence. Clinical validation is the manufacturer’s responsibility, and the full checklist is in our guide to what validation SAM needs in clinical imaging.

The question to settle before choosing a SAM variant is narrower than “is SAM good enough”. It is which structure, in which modality, at what correction effort per case, and whether that use stays on the labelling side of the line.

FAQ

Is the Segment Anything Model open source?

Yes. Meta releases the SAM models under the Apache 2.0 licence, and they are used in research and industry, including medical imaging.

What is the difference between SAM and YOLO?

YOLO is a real-time object detector that outputs class-labelled bounding boxes. SAM outputs masks without categories for whatever it is prompted with. SAM can be paired with a detector: in Grounded-SAM, an open-vocabulary detector’s labelled boxes become SAM prompts.

What is segmentation in medical image processing?

It means identifying the region of interest in an image, typically an anatomical structure or a tumour, and it underpins computer-aided diagnosis.

Back See Blogs
arrow icon