A detection model that reports 95% sensitivity on a public angiography dataset is not the same model once it runs on your hospital’s scanners. That gap is where aneurysm detection stops being a benchmark exercise and becomes a clinical-safety problem. The number that looked reassuring in a paper describes a curated distribution — a specific set of scanners, contrast protocols, and aneurysm morphologies. Your emergency department’s overnight CT angiograms are a different distribution, and the model does not know that.

This is the recurring failure in medical-imaging AI: a team selects a detector because it topped a leaderboard, deploys it against production scans, and discovers — often on a missed case — that the sensitivity they trusted never applied to their patients. The naive assumption is that benchmark performance transfers. The correct framing is that detection is a recall-critical task whose only meaningful measurement is on the site’s own imaging distribution, where a missed small or atypically located aneurysm is a clinical harm, not a rounding error.

Why does benchmark sensitivity not survive contact with production scans?

The structural problem is that the training distribution and the deployment distribution differ in ways that directly move the decision boundary of a detector. Public aneurysm datasets are curated: they over-represent clearly visible saccular aneurysms at common locations (the anterior communicating artery, the middle cerebral artery bifurcation) because those are the cases that get labelled cleanly and reach a shared dataset. A production scanner sees the full clinical tail — fusiform aneurysms, aneurysms below 3 mm, aneurysms in atypical posterior-circulation locations, and images degraded by motion or suboptimal contrast timing.

When a model has seen few examples of a morphology, its confidence on that morphology collapses. That is not a bug in the model; it is the expected behaviour of a function fit to a distribution and then queried outside it. The same mechanism that makes an object detector reliable inside its training distribution makes it unreliable outside it, and the precision, recall, mAP, and IoU that describe detector performance all shift when the input distribution shifts. A benchmark sensitivity is a conditional statement — sensitivity given this distribution — and clinicians reading the deployment as unconditional are reading it wrong.

There is a second layer. Benchmark sensitivity is usually reported per-lesion on a test set that has been enriched with positive cases, so the false-positive rate per study is either not reported or reported against an artificial prevalence. In real screening, aneurysm prevalence per study is low, which means the operating point that looked balanced on the benchmark can generate a heavy false-positive load in production. That load is not free: every false positive is a study a radiologist must adjudicate, and alert fatigue erodes the sensitivity benefit the tool was bought for.

Which imaging conditions move detection sensitivity the most?

Three variables dominate in the CT and MR angiography settings we see most often. Naming them matters, because each one is measurable before deployment and each one has a different remedy.

Contrast timing. Aneurysm conspicuity depends on arterial opacification. A scan acquired slightly late — venous contamination, or peak arterial enhancement missed — changes the appearance of vessel walls and small outpouchings. A model trained on well-timed studies will silently lose sensitivity on the fraction of your studies where timing drifted.

Slice thickness and reconstruction kernel. Sub-3 mm aneurysms occupy a handful of voxels. At 1 mm slices they are resolvable; at 3–5 mm slices they partial-volume into the parent vessel and disappear. A model validated on thin-slice research data will underperform on a site that reconstructs thicker slices for routine reads.

Scanner and vendor variation. Noise characteristics, reconstruction algorithms, and default windowing differ across scanner generations and vendors. A detector fit to one vendor’s texture statistics treats another vendor’s noise as signal or vice versa. This is the distribution shift that a multi-site design is meant to expose — the reason federated benchmarking approaches like MedPerf exist to measure generalisability across institutions rather than on a single curated split.

Imaging variable	Effect on detection sensitivity	How to measure it before deployment
Contrast timing drift	Lower conspicuity of small/wall lesions; false negatives on late-phase studies	Stratify site validation set by acquisition phase; report sensitivity per stratum
Slice thickness ≥3 mm	Partial-volume loss of sub-3 mm aneurysms	Report sensitivity by lesion-size bin against thin-slice ground truth
Scanner/vendor variation	Texture-statistic mismatch; unpredictable FP rate	Include every production scanner in the validation set; report per-scanner FP/study
Rare morphology (fusiform, atypical location)	Confidence collapse; systematic misses	Curate a rare-morphology subset; report recall on it separately

The evidence class here matters: the direction of these effects is an observed pattern across medical-imaging engagements, not a single published benchmark figure. The magnitude for your site is unknown until you measure it on your own studies — which is precisely the point.

How do you validate against site-representative data?

Validation that predicts clinical behaviour has to be built from the deploying site’s own imaging distribution, not a vendor’s held-out split. In practice that means assembling a validation set that reflects the real prevalence, the real scanner mix, the real slice-thickness distribution, and — critically — a deliberately enriched subset of the hard cases: small aneurysms, atypical locations, and the fusiform morphologies that public datasets under-represent. Ground truth comes from expert consensus reads, ideally with the reference standard (DSA or expert-adjudicated CTA/MRA) that the clinical workflow already trusts.

Then two numbers get reported side by side: per-lesion sensitivity stratified by size and location, and false positives per study at the chosen operating point. Reporting sensitivity alone is how tools get deployed with unacceptable review burden; reporting a single blended sensitivity is how the sub-3 mm miss rate gets hidden inside an average dominated by easy cases. The detection metric conventions used in medical-device CV validation give the vocabulary, but the discipline is in the stratification.

A worked validation frame (illustrative)

Assume a site validation set of representative CTA studies with expert-adjudicated ground truth. The reporting frame that makes the decision legible looks like this:

• Overall per-lesion sensitivity, e.g. if the model detected 88 of 100 confirmed aneurysms, that is the headline recall — but it is meaningless without the strata below.
• Sensitivity by size bin: <3 mm, 3–7 mm, >7 mm. The >7 mm bin often looks like the benchmark; the <3 mm bin is where the real gap lives.
• Sensitivity by location: anterior circulation vs posterior circulation, common vs atypical sites.
• False positives per study, reported at the same operating point, because that number sets the radiologist review cost.

These figures are illustrative of the structure of a defensible report, not measured values — the actual numbers are what a site validation produces. Any vendor claim of a fixed sensitivity that cannot be reproduced on this frame against your own studies should be treated as a benchmark statement, not a clinical one.

When is fine-tuning enough, and when do you need a different approach?

Not every gap is fixable by fine-tuning on local data, and treating fine-tuning as a universal remedy is its own failure mode. The decision hinges on where the model is failing.

If validation shows the model detects the target morphologies but is mis-calibrated to your scanner’s noise and windowing, fine-tuning on a modest set of local studies often closes the gap, because the underlying capability exists and only the operating point and low-level features need adaptation. If validation shows systematic misses on a morphology class — fusiform aneurysms the model was never trained to represent, or sub-3 mm lesions below its effective resolution — no amount of fine-tuning on the same architecture recovers a capability that was never learned. That is a case for a different model, a different input resolution, or a redesigned detection stage.

Signal from site validation	Likely cause	Reasonable response
Sensitivity holds on morphologies but FP/study is high	Operating point / low-level feature mismatch	Recalibrate threshold; fine-tune on local studies
Sensitivity drops only on a specific scanner	Vendor texture-statistic mismatch	Add that scanner’s studies to fine-tuning set
Systematic misses on a morphology class	Capability absent from training	Different model / higher input resolution; do not fine-tune the same net and hope
Small-lesion misses at production slice thickness	Resolution below detectable limit	Change reconstruction protocol or model input resolution, not the weights

Distinguishing these cases is the entire value of a structured readiness assessment: it tells you which problem you have before you spend a quarter fine-tuning a model that was never going to detect the lesions you care about.

FAQ

How should you think about aneurysm detection in practice?

An aneurysm detection model scans angiographic imaging (CT, MR, or DSA) and flags regions that resemble the abnormal vascular outpouchings it was trained to recognise. In practice it functions as a recall-oriented second reader: its job is to surface candidate lesions for a radiologist, not to make the diagnosis. What it “means” operationally is defined by two numbers — how many true aneurysms it catches (sensitivity) and how many false alarms it raises per study — measured on the imaging it will actually run against.

Why do benchmark-validated aneurysm detection models miss cases on a hospital’s own scans?

Because benchmark sensitivity is a conditional figure tied to a curated dataset, and a hospital’s scans come from a different distribution. Public datasets over-represent clearly visible, common-location aneurysms; production imaging includes the full tail of small, fusiform, and atypically located lesions plus scanner and contrast-timing variation the model never saw. When a detector is queried outside its training distribution its confidence on unfamiliar cases collapses, so cases the benchmark never contained get missed.

What imaging conditions most affect detection sensitivity?

Three dominate: contrast timing (late or venous-contaminated studies reduce lesion conspicuity), slice thickness and reconstruction kernel (sub-3 mm aneurysms partial-volume away at thick slices), and scanner or vendor variation (different noise and reconstruction statistics that a model can misread). Each is measurable before deployment by stratifying a site validation set along that variable and reporting sensitivity per stratum.

How do you validate an aneurysm detection model against site-representative data before clinical use?

Build a validation set from the deploying site’s own studies that reflects real prevalence, the full scanner mix, and the real slice-thickness distribution, then deliberately enrich it with hard cases — small, atypical-location, and fusiform aneurysms. Establish ground truth by expert consensus against the reference standard the workflow already trusts, and report per-lesion sensitivity stratified by size and location alongside false positives per study at the chosen operating point.

What miss rate and false-positive-per-study thresholds should be agreed before deployment?

The specific numbers are a clinical and operational decision for the site, but the discipline is to agree them explicitly before deployment rather than discover them on a missed case. Sensitivity thresholds should be set per size/location stratum — especially the sub-3 mm bin — not as a single blended figure, and the false-positive-per-study ceiling should be set against the radiologist review burden the workflow can absorb without alert fatigue.

When is fine-tuning on local imaging enough versus needing a different model or approach?

Fine-tuning is enough when validation shows the model can detect the target morphologies but is mis-calibrated to local scanner noise, windowing, or operating point — the capability exists and only adaptation is needed. When validation shows systematic misses on a morphology class the model was never trained to represent, or lesions below its effective input resolution, fine-tuning the same architecture will not recover a capability that was never learned; that calls for a different model, higher input resolution, or a protocol change.

The honest position on aneurysm detection is that the benchmark number tells you almost nothing about clinical behaviour, and the only figure worth trusting is the one your own scans produce under a stratified report. Before any detection tool touches a live workflow, we characterise its failure modes against the deploying site’s own imaging — a production CV readiness assessment whose entire purpose is to make the miss rate known in advance, not discovered on the case that mattered.