Computer aided detection (CADe) software takes a radiology image, cleans it up, finds candidate regions, scores each one and puts marks in front of a clinician, who makes the decision. That is the mechanism. The number on the vendor’s datasheet is a different thing: it describes how the software performed on one test set, read under one set of conditions, and it only predicts your experience to the extent that your scanners, protocols, patients and readers resemble that test set. Whether the tool holds up in your department is settled by validation on a distribution like your own, and by monitoring once it is live, not by the published figure.
How does computer-aided diagnosis software work, from DICOM to the reader’s screen?
In a typical deployment, image data are copied to a CAD server in DICOM format and analysed in several steps: preprocessing, segmentation, analysis of each region of interest, and evaluation or classification. Preprocessing reduces artefacts and noise and harmonises image quality across different exposure conditions. Segmentation separates the structures in the image (organs, vessels, candidate lesions), and each detected region is then measured for shape, size, location, grey-level statistics and its relation to neighbouring structures. Finally, every region is scored individually for the probability that it is a true positive finding.
The FDA’s CADe guidance describes the same anatomy in its own vocabulary: processing (filtering, segmentation, normalisation, registration, artefact or motion correction), features computed from the image and sometimes from patient characteristics, and a model or classifier that combines them, whose parameters and thresholds are set during training.
Where each stage of a CAD pipeline breaks
| Stage | What it does | How it typically fails |
|---|---|---|
| Acquisition | Scanner produces the image under a given protocol | Protocols and pre-processing vary between sites, and images from a different scanner or protocol are a covariate shift the model never saw |
| Pre-processing | Noise reduction, harmonisation, normalisation | A storage quirk (one study found one scanner’s images stored in an inverted colour scheme) can become a feature the model relies on while overlooking pathology |
| Detection / classification | Candidate regions scored against a threshold | Models can fail silently on out-of-distribution inputs, leading to outcomes such as misdiagnosis |
| Reviewer presentation | Marks and scores shown to the clinician | Low specificity (many false positives) reduces acceptance of the system, because the reader has to identify all the wrong hits |
The silent failure in the third row is the one to design around. A model can still return a high softmax score for an input far from its training distribution, so the output looks normal while being meaningless.
Computer aided detection vs computer-aided diagnosis: marking versus characterising
Computer-aided detection marks conspicuous structures; computer-aided diagnosis (CADx) evaluates them. In the FDA’s terms, a CADe device identifies, marks, highlights or otherwise directs the clinician’s attention to parts of an image that may reveal abnormalities. A CADx device goes further, giving an assessment of the likelihood that disease is present or absent, or specifying disease type, severity, stage or recommended intervention.
The distinction carries a regulatory consequence in the US. Medical image analysers, including CADe for mammography, breast ultrasound, lung nodules on radiographs and dental caries, are classified under 21 CFR 892.2070. Software that characterises lesions as suspicious for cancer sits under 21 CFR 892.2060, and its definition keeps diagnostic and patient-management decisions with the clinical user. Detection/diagnosis combinations, triage tools and acquisition-optimisation software each have classifications of their own. The full mapping of intended use to classification is covered in our piece on intended use and risk classification for AI/ML SaMD validation.
Why the CADe/CADx split changes how you evaluate
A detection tool is judged on whether its marks land on the real abnormalities and stay off normal anatomy, which is why the FDA asks for a defined scoring process for deciding when a mark corresponds to the location and extent of a lesion. It also asks for the standalone false positive rate to be derived from normal and abnormal cases separately. A characterisation tool is judged on whether its likelihood or category is right for a lesion already found, so the reference standard for disease type and stage matters more than localisation. Treat a CADe sensitivity figure and a CADx AUC as comparable and you end up comparing different questions under one name.
Where does CAD sit in the reading workflow, and when does it become a regulated device?
CAD is a second opinion: the radiologist uses the computer output and makes the final decision, and the computer’s performance needs to complement the physician’s rather than match it. A current randomised trial protocol shows what that looks like at the workstation: the AI report with an exam-level risk category and scored bounding boxes opens first, the radiologist can toggle the marks on and off, and keeps complete autonomy over the final call. The same protocol requires radiologists to train on external validation cases, including AI false alarms and missed cancers, specifically to counter automation bias.
Keeping the human in charge does not take image analysis out of device territory. The FDA’s 2022 clinical decision support guidance states that software that processes or analyses a medical image, including functions within CADe or CADx systems, fails the first criterion for non-device CDS and remains a device. Non-device CDS works on medical information (a radiology report, or a summary such as “twelve CAD annotations are present” from legally marketed CAD software), not on the pixels. In practice, the boundary is drawn at the input: once your software reads the image itself, adding a review step does not move it out of the device category. Where engineering evidence ends and clearance work begins is covered in our post on FDA medical device regulation for imaging AI.
What did mammography CAD show about premarket results versus real readers?
Mammography CAD is a documented case of a tool performing worse in clinics than its premarket evidence suggested. Computer aided detection for mammography was FDA-cleared in 1998 on the basis of small reader studies, and by 2016 it was in use at 92% of US mammography facilities, according to the 2025 protocol of a pragmatic trial registered as NCT06934239. The same protocol’s authors state that their later work linked historical mammography CAD to lower diagnostic accuracy, more false positives and higher costs, risks that were not evident in the premarket reader studies.
The community-practice data are specific. Fenton and colleagues (NEJM, 2007) analysed 429,345 mammograms at 43 facilities in three US states and found that after CAD implementation specificity fell from 90.2% to 87.2%, positive predictive value fell from 4.1% to 3.2%, and biopsy rates rose by 19.7%, while the change in cancer detection was not significant. Across all facilities, CAD use was associated with lower overall accuracy, an area under the ROC curve of 0.871 versus 0.919 without it. Lehman and colleagues (JAMA Internal Medicine, 2015) compared 495,818 digital screening mammograms read with CAD against 129,807 without, from 2003 to 2009, and found no accuracy improvement; among radiologists who read both ways, sensitivity was lower with CAD (odds ratio 0.53).
A controlled trial in England gave a different picture. In the CADET II equivalence trial of film mammography in England (NEJM, 2008), 31,057 women were randomised, and single reading with CAD detected 87.2% of cancers against 87.7% for double reading, with a small but significant rise in recall rate from 3.4% to 3.9%. Where the comparison was a second human reader in a controlled trial, CAD looked roughly equivalent at some cost in recalls; spread across community practice with its own readers and habits, CAD looked worse than reading without it. Study results depend on design, image quality, examination conditions, reader experience and the type and size of lesion, so each result has to be read on its own terms.
Why a vendor’s accuracy figure won’t hold on your scanner mix and cohort
A published accuracy figure assumes that your data are drawn from the same distribution as the test set, and supervised models rest on exactly that assumption. Data drift is the name for the gap between training data and the data met in clinical practice, and it can be immediate (a different scanner fleet) or gradual (changing demographics, new diseases). Different scanner manufacturers and acquisition protocols are a standard source of that shift.
Zech and colleagues (PLOS Medicine, 2018) tested pneumonia-screening models across three hospital systems. A pneumonia model trained and tested on pooled data from two hospital systems reached an internal AUC of 0.931, but only 0.815 on a third, held-out hospital system. A trivial model that ignored the images and ranked cases by each hospital’s pneumonia prevalence scored an AUC of 0.861 on the same joint test set, so much of the apparent performance came from telling sites apart. The networks identified the source hospital system with near-perfect accuracy, and hospital system was a larger source of variation in the image data than pneumonia. The authors conclude that performance estimated on test data from the training hospitals may overstate real-world performance.
A systematic review by Yu, Mohajer and Eng (Radiology: AI, 2022) of 86 deep-learning algorithms for radiologic diagnosis found that 70 (81%) performed worse on external data than internal data, and 21 (24%) dropped by 0.10 or more on the unit scale. Even a training set representative of deployment does not guarantee performance on every subgroup, because models that minimise average error can still fail specific subpopulations. Why AUC itself is a weak predictor of clinical value is argued separately in why benchmark AUC doesn’t predict clinical performance.
What does CAD validation on a distribution like your own involve?
Validating on a distribution like your own means testing the tool on cases representative of your department’s acquisition, patients and disease, with enough cases per subgroup to estimate performance for each one. The US special controls for both CADe and CADx require a manufacturer’s test dataset to contain enough cases from important cohorts (clinically relevant confounders, effect modifiers, concomitant diseases and image acquisition characteristics) that performance and confidence intervals can be characterised for the intended population and imaging equipment.
The variables worth representing fall into four groups:
- Acquisition: scanner make and model, scanning characteristics, and protocol parameters such as contrast agent or probe positioning.
- Site and pipeline: site-specific acquisition protocols and pre-processing, image processing and compression along the distribution pipeline.
- Lesion and disease: lesion size, type and location, and disease stage.
- Cohort: demographic groups, and populations absent from training altogether (a paediatric chest X-ray is out of distribution for a model trained on adults).
Prevalence belongs on the list as well, because models trained at one hospital’s pneumonia prevalence were poorly calibrated to another’s. Real-world datasets, local ones included, also carry bias from cohort selection, annotation and scanner differences. Sampling and sizing the set itself is the subject of our guide to stratified validation set design.
Questions that tell you whether a vendor validated on data like yours
The FDA’s CDS guidance gives a telling example: a clinician told only that an algorithm was trained on a large dataset, without the distribution of cases by site, breast density or ethnicity, or the independence of development and validation data, cannot judge whether the results generalise. Those gaps turn into questions a buyer can ask:
- Was the test data independent of and sequestered from development data, and was it reused across algorithm versions? Reuse can tune an algorithm to the test set.
- Was the test set enriched with abnormal cases? Enrichment is permitted but may stop estimates generalising to the screening population.
- What was the reference standard (pathology or a standard-of-care determination), and how was reader disagreement in the truthing handled? Our post on ground-truth disagreement in imaging validation goes deeper on this one.
- Are results reported per stratum (lesion type and size, protocol, scanner), with false positive rates for normal and abnormal cases separately?
- Which scanner models and acquisition techniques were tested, and are they the ones in the labelling?
- If the device allows several operating points, which one suits your practice?
- Was any external validation done at sites the developer did not train on, ideally across several hospitals?
If the honest answer to the fifth or seventh question is “not your scanners” or “no”, treat the datasheet figure as a hypothesis to test locally. The specific documentation the agency expects is walked through in our reading of FDA CADe guidance requirements.
Does validation authorise a CAD tool for clinical use?
No. The IMDRF defines validation as confirmation, through objective evidence, that the requirements for a specific intended use have been fulfilled; whether a tool is cleared for clinical use is a separate question. Validation also does not carry over across versions: the FDA treats changes to image processing, features, classifiers, training methods, training data or parameters as potentially significant to safety and effectiveness.
For a buyer the practical reading is two-sided. A cleared AI-enabled medical device comes with performance testing for its labelled population and imaging equipment; local validation tells you whether your department matches that population and equipment.
What does it mean to validate a CAD system properly?
Proper validation answers three separate questions, which the IMDRF sets out as valid clinical association, analytical validation and clinical validation: is the output linked to the clinical condition, does the software turn inputs into accurate and reliable outputs, and does using those outputs achieve the intended purpose in the target population. For computer aided detection that means both standalone performance (the device alone, without a clinician) and, where the device cannot be compared directly with its predicate, a reader study, because the reader is part of the diagnostic process. Mammography is the warning about relying on small reader studies: its risks were not evident in the premarket reader studies.
The third part is ongoing. The IMDRF describes clinical evaluation as iterative and continuous, and encourages manufacturers to use software’s connectivity to monitor real-world performance. Deployed algorithms should be monitored for drift so that providers can decide whether a model should be retrained, taken offline, replaced or re-evaluated. Drift detection has limits: methods are mostly evaluated on simulated drift, and once drift is flagged, expert labelling, retraining and revalidation are needed to confirm and fix a real decline. Watching the statistics of model inputs and outputs is a useful early signal while labels are unavailable.
TechnoLynx’s Production AI Reliability work sits in that production layer. We treat production AI reliability as an engineering discipline distinct from model accuracy: eval harnesses, regression suites, drift detection and validation packages for medical-imaging workloads, not a substitute for regulatory sign-off. What that engagement contains is laid out in our description of clinical-grade imaging AI validation, and our wider AI engineering for life sciences and healthcare covers the build side.
FAQ
What is the purpose of a validation dataset in developing a machine learning model?
Training, hyperparameter tuning and evaluation are run on three separate parts of the data, and the validation part is used for tuning, including setting score thresholds. The FDA discourages repeated use of the same test data, since reuse risks tuning the algorithm to it.
What are examples of computer-aided detection and diagnosis in radiology?
US regulations list CADe for mammography, breast ultrasound lesions, lung nodules on radiographs and dental caries. Other uses include detecting significant coronary stenosis on CT angiography and CADx for bone metastases on whole-body bone scans.
