A multi-reader multi-case (MRMC) study is a clinical performance assessment study. Several intended users read several cases under multiple reading conditions, for example unaided versus aided by the device. Readers and cases are both treated as random samples, so the reader-averaged result generalises to the wider population of readers and of cases, not just to the people and images in the study. For an AI tool that assists a clinician’s read, the FDA’s guidance on computer-assisted detection treats the MRMC design as the usual way to assess clinical performance. A defensible multi-reader multi-case study has four properties. It is a single design fixed in advance. It is stratified by scanner vendor and patient subgroup. It is powered for every claim it makes. And it is kept separate from the analytical validation that sits beside it.

The protocol, the statistical analysis plan, the audit trail and the reports are evidence that an auditor or compliance owner reads, defends and signs against. Producing them is engineering work, not paperwork and not certification. This is where statistical design, data handling and software discipline meet in AI engineering for life sciences and healthcare. It also helps to keep the difference between validation evidence and regulatory clearance in view: a well-built study supports a submission, and the decision still belongs to the regulator.

When a multi-reader multi-case study is the right design for an AI device

The MRMC design fits when the claim is about what clinicians do with the device: that readers aided by the AI detect or classify better, or no worse, than readers without it. The FDA recommends comparing against a control arm of conventional reading without the device, using the same reading method and the same cases (or the same population of cases) in every arm. An automated classifier whose output is acted on without a reader in the loop is a different case. We would assess it with a diagnostic-accuracy study that estimates sensitivity and specificity against a reference standard.

Device and claim	Study design	What the evidence must show
Reader-assist (CADe), concurrent or second read; claim about clinician performance	MRMC reader study with an unaided control arm, plus a standalone assessment	Change in reader performance, with readers and cases as random effects
Automated or triage classifier, no reader in the loop	Diagnostic-accuracy study against a reference standard	Sensitivity and specificity pairs with two-sided 95% confidence intervals
Comparison with a non-reference comparator only	Agreement study	Positive and negative percent agreement, not sensitivity and specificity
Substantial equivalence to a predicate CADe	Standalone comparison on the same dataset where possible; reader study when indications or mark characteristics change	Comparable standalone performance, or a clinical effect matching the predicate’s

FDA’s CADe 510(k) guidance explicitly excludes computer-assisted diagnosis (CADx) and computer-assisted triage (CADt) devices. A triage product therefore cannot borrow those recommendations wholesale, although the design logic carries over well. A 2019 American Statistical Association conference presentation on MRMC studies makes the broader point: design and analysis methods differ with the device’s intended use and outcome type.

Fully crossed, split-plot or mixed

In a fully crossed design every reader independently reads every case, and the FDA describes it as offering the greatest statistical power for a given number of cases. A fully crossed design is cost-effective in the total number of readers and cases, but study length and each reader’s workload can be large. In a paired split-plot design, readers still read the same cases under both modalities, so readings stay paired across conditions, but within each modality not every reader reads every case. A newer “mixed” design makes a different trade. That design suits conditions that are not rare and where patients routinely get the gold-standard assessment.

We would start from fully crossed and move away from it only when reader time is the constraint that really binds. The mixed design’s saving in reads is paid for with more truth-verified cases.

A sequential design (unaided read, then the AI output shown) keeps intra-reader variability low. It can, however, confuse the AI’s effect with the extra reading time, or encourage readers to undercall the unaided read. A cross-over design reads the cases unaided and aided in separate sessions separated by a washout period. Session effects cancel, at the price of extra variability. The FDA recommends at least four weeks between sessions that reuse the same cases. It also recommends randomising readers, cases and reading scenarios, for example with Latin squares.

How many readers and cases an MRMC study needs

The number of readers needed depends on reader variability. Case variability is often substantial too, so the number of cases must also be large enough to detect the difference the study is after. The MRMCsamplesize R package (Putha and colleagues, 2023) asks for these inputs: the effect size, the between-reader variability range (best reader minus worst) and the within-reader variability range (one reader reading the same images twice). Its documentation recommends at least 5 readers in any MRMC study and gives 0.04, 0.05 and 0.06 as typical effect sizes. The Multi-Reader Sample Size Program for Diagnostic Studies returns combinations of readers and cases for a target power, or the power for a given combination. Its parameters come from pilot data analysed with the Obuchowski–Rockette or Dorfman–Berbaum–Metz methods, or are conjectured. The iMRMC package can both size and analyse studies that compare reader-averaged AUC across two modalities, and its Roe–Metz simulation generates MRMC data sets for testing analysis methods.

Pilot data is worth getting before you lock the design. A sizing calculation built on guessed variability ranges is only as defensible as the guess.

Worked example: how one pivotal MRMC study was sized

The statistical analysis plan for Fujifilm’s pivotal MRMC study of digital breast tomosynthesis plus synthesised view (DBT plus S-View) against full-field digital mammography (FFDM) shows the pieces fitting together:

• Primary endpoint: non-inferior per-subject average AUC, with margin 0.05 at alpha 0.05.
• Success rule: the lower limit of the two-sided 95% confidence interval for the AUC difference lies entirely above −0.05.
• Sizing: Obuchowski’s method for a fully crossed design, giving 80% power for the MRMC F*-test with 60 cancer and 240 non-cancer cases, and arriving at 18 readers.
• Reading: a counterbalanced crossover over two visits about four weeks apart. In each visit, every reader reads half the cases under each modality.
• Analysis: Obuchowski–Rockette ANOVA, so the result generalises to readers and cases.

Working from those figures: 60 + 240 gives 300 cases. That workload is what split-plot and mixed designs aim to reduce. The 2019 presentation describes a DBT CAD study with 20 readers and 240 cases, and a pathology AI study with 20 readers and 600 cases, both fully crossed.

Secondary endpoints need protection against Type 1 error inflation. The Fujifilm plan uses the graphical approach of Bretz and colleagues: secondary hypotheses are tested only if the primary is rejected. Its path weights were finalised before any reading data arrived.

Stratifying by scanner vendor and subgroup so each stratum has power

Stratification does two jobs, and they need different sample sizes. The first is to describe performance in each important cohort. The FDA says the test set should hold enough cases from cohorts defined by confounders, effect modifiers and image acquisition characteristics that each subset’s performance and confidence interval can be characterised. The second is to support a claim about a subset. The FDA says powering a cohort for statistical significance is not necessary unless you make a specific subset performance claim. If you make claims for individual subsets, you size those subsets to detect them, and multiple claims need a pre-specified multiplicity adjustment.

One design that reports a national estimate and per-vendor results

A pooled result can be reported alongside per-subgroup breakdowns, provided each stratum carrying a claim is sized for it and multiplicity adjustment is pre-specified. The FDA asks for case distributions stratified by factors such as imaging hardware make and model, imaging protocols, collection sites, lesion type and size, and organ characteristics such as breast composition. The FDA also asks sponsors to justify how far results generalise across the acquisition technologies in the labelling, including stratified analysis by make and model.

A nationally representative estimate does not need every site to look like the nation. In an FDA CDRH Learn session, Greg Campbell said no single site may represent the target population. The goal is a varied set of sites (urban and suburban, academic and community) that is representative as a group. Results should then be reported by clinical site and by relevant clinical and demographic subgroup, as fractions and as percentages, with two-sided 95% confidence intervals. Pooling across sites is not automatic. The FDA expects poolability to be justified both statistically and clinically.

Powering by vendor and subgroup, not just sampling in proportion

Proportional enrolment is the route to an unbiased overall estimate. FDA guidance on both sex-specific and age-, race- and ethnicity-specific data starts there. The same guidance adds that when prior evidence suggests a clinically meaningful difference in a subgroup, sponsors should enrol enough of that subgroup to support claims about it, which can mean a stratified design. Representative proportions alone may not give a subgroup enough cases to support claims about it. Enrolling enough of that subgroup to support its own analysis is the fix.

None of our sources gives a worked method for powering a three-vendor study. Here is how we would do it. Treat each vendor carrying a claim as a subset claim. Run a separate sample size calculation for each, using that vendor’s expected sensitivity and the smallest clinically meaningful gap. Then apply the multiplicity adjustment across the vendor claims. Two effects push the required numbers up: an expected sensitivity close to 50%, and a narrower confidence interval target. For small strata, or proportions near 0 or 1, the Fujifilm plan allows exact binomial intervals instead of normal approximations.

We treat uneven case volumes across sites or vendors as a recruitment problem, not a reason to change the statistics. Adding sites or extending recruitment is the honest fix, and it is better decided before the design is locked. Enrolment by subgroup should be checked at each stage (screening, eligibility, consent) to see where an underrepresented group drops out. If one site enrols mostly one demographic group, differences between sites can be wrongly read as differences between subgroups.

Pre-specified confidence intervals by vendor and subgroup

The FDA wants the subgroup analysis plan in the statistical analysis plan, with significance levels for heterogeneity tests set in advance. A test of interaction can serve as the condition for pooling. The FDA notes, though, that such tests often have unspecified power, so a non-significant interaction is weak evidence that no clinically relevant difference exists. Our reading is that the interaction test itself should be sized to detect a difference that matters, as the guidance suggests, and not left as an afterthought.

External validation on a distinct cohort

Generalisation across hardware and populations needs appropriately designed external validation data. The test set must differ from the data used for training, calibration and tuning. Kim and colleagues found that only 6% of 516 eligible published imaging AI studies (31 studies) used external validation. None of those 31 combined a diagnostic cohort design, multiple institutions and prospective data collection. They recommend external data from newly recruited patients or from institutions other than those that supplied the training data, representative of the target population’s full spectrum. Multiple external institutions test whether the algorithm holds up across hospital systems.

Whether “distinct” means a different site, a different time period or both is not settled by these sources. Kim and colleagues accept either new patients or other institutions. Our view is that a claim about unseen scanner vendors or populations needs a different site, while a later time period at the same site mostly tests stability over time. For sizing, authors of an external-validation series note that the rule of thumb of at least 100 events and 100 non-events is only a starting point. They recommend tailored calculations, and a dataset representative of the target population and setting.

Can one MRMC reader panel cover several equipment vendors?

Our judgement is that one panel can cover several vendors if vendor is handled as a case-level stratum inside a fully crossed design. Every reader then reads cases from every vendor, so vendor differences are not confused with reader differences. Separate panels per vendor tie each vendor’s result to a different set of readers, which makes per-vendor comparisons harder to defend and the overall estimate harder to pool. Our sources do not address this question directly. The constraint they do impose is that readers should represent the intended clinical users and the imaging hardware should match current clinical practice.

Enrichment, stress tests and spectrum bias

A retrospective reader study enriched with diseased cases changes what the numbers mean. The FDA notes that enriching with diseased cases will likely change reader performance compared with clinical practice, because prevalence and case difficulty differ. Stress testing with harder cases can inflate the AI’s measured effect if the AI helps most on exactly those cases. The FDA still encourages stress testing because it informs important subgroups and allows smaller samples. Spectrum bias, where only clear-cut healthy and diseased subjects are included, gives an overly optimistic picture. The FDA’s remedy is to include the full range of disease states, relevant confounding conditions and demographic groups. More subjects do not fix bias. Selecting the right subjects does.

If one subgroup’s confidence interval is too wide, can the pooled result still support the claim?

It can support the overall claim, but not a claim about that subgroup. The FDA’s position is that unplanned subgroup analyses, or analyses with inadequate sample size, are generally not adequate to support labelling statements about that subgroup. Pre-specified subgroup analyses with multiplicity adjustment may carry inferential statistics. Exploratory ones should use descriptive statistics only. Whether pooling is valid depends on the size and clinical importance of any observed interaction, not just its p-value. The evidence from the study determines what the labelling can claim. So a subgroup whose interval stays too wide needs its own dedicated evidence, or the claim is narrowed to exclude it. A study design answer like this one cannot promise that any particular model will meet its targets in every stratum.

Adjudicating ground truth so results from every site compare

Ground truth in a multi-site MRMC study should come from a described truthing methodology, fixed before the evaluation starts. The FDA asks for the basis of the reference standard: another device, an established clinical determination such as biopsy, follow-up imaging or examination, or interpretation by clinical truthers. Fujifilm’s plan, for example, used biopsy for cancer and benign cases and one-year follow-up imaging (320 to 455 days) for recall and normal cases. Where clinicians adjudicate, the FDA wants their number, qualifications, instructions and the information they saw. It also wants to know how disagreements between them were combined. Truthers must not be the readers in the performance study, because that introduces bias. The scoring process must also be scientifically justified and fixed in advance.

Expert panels carry their own risks. At least 8% of studies did not verify all participants the same way (differential verification). The same review advises strongly against dropping hard-to-diagnose cases, because that overestimates accuracy. FDA guidance adds that discrepant resolution does not solve the bias problem either.

When several CROs or sites each ran part of the reading, consistency is a fact you check, not one you assume. Compare each protocol’s adjudication method and reader-panel make-up. Document the differences, and report separately where they undermine pooling. That check is a natural place to bring in the wider guidance on designing defensible multi-site reader studies for AI/ML SaMD submissions, which covers the site and governance side in more depth.

Analytical validation vs clinical validation in the technical file

Analytical validation and clinical validation answer different questions. In the International Medical Device Regulators Forum (IMDRF) SaMD clinical evaluation framework, analytical validation asks whether the software correctly processes input data to produce accurate, reliable and precise output. Clinical validation asks whether using that output achieves the intended purpose in the target population in clinical care. A third pillar, valid clinical association, asks whether the output is linked to the target condition at all. The EU’s MDCG 2020-1 guidance takes three components into the MDR and IVDR. In the MRMC setting the split is concrete. Standalone performance is the device alone. Clinical performance is the device in intended use by intended users.

What each workstream is, and why a retrospective result is not enough

Analytical evidence can come from verification and system testing, curated databases, reference databases or previously collected patient data. Clinical performance evidence comes from testing the software, or an equivalent device, in the target population for the intended use, through pre-clinical testing, a clinical investigation or a performance study. Under MDCG 2020-1, a prospective study may be required when the software’s output affects clinical outcomes or patient management. Retrospective data suits cases with no such impact. In MDCG’s segmentation example, gaps where retrospective data misses input variability are filled by a prospective clinical investigation. Our own example: a bench or phantom test says little about conditions it never reproduced, such as cardiac or respiratory motion.

A technical file that keeps the two apart

The FDA recommends a standalone assessment in every CADe submission, whether or not a reader study is included. For a reader-assist device, a standalone assessment alone may not be enough. In the file we would give each its own section, with its own dataset description, ground-truth method, endpoints and results. Where a method is shared (for example, how the operating threshold was calibrated), it would be cross-referenced, not merged. Section names should follow the target jurisdiction’s own terms. In our view, a single report that mixes archive accuracy with reader-study results risks being read as a methodological flaw. The FDA recommends a “firewall” so that nobody linked to algorithm development sees test data, along with an access log, a cap on reusing cases, and new readers in each new reader study. An audit trail should also show that there was no leakage between training and test sets and that the algorithm was fixed before testing.

Separate acceptance criteria for each workstream

Each workstream needs its own acceptance criteria, set in advance and justified clinically, not chosen because they are statistically convenient. One national framework for radiology AI, from Russia, grades analytical-stage metrics below 0.6 as unsuitable, 0.61–0.8 as needing revision, and 0.81 or above as admissible to proceed to clinical validation. It also asks developers to calculate metrics with 95% confidence intervals. Clinical criteria are about clinically meaningful, patient-relevant outcomes. MDCG 2020-1 adds that clinical performance validation should be considered at each new software release. This kind of package discipline is distinct from model accuracy. It is the discipline behind a clinical AI validation pack.

Pre-specification checklist before the first read

Everything below should be fixed before the evaluation starts and before anyone sees outcome data:

1. Intended use, target population and the device’s role in the read (concurrent or second reader).
2. Design (fully crossed, split-plot or mixed), reading scheme, washout of at least four weeks, and randomisation plan.
3. Control arm and comparator.
4. Primary and secondary endpoints, and multiplicity control.
5. Sample size calculation with its variability inputs, including separate calculations for every stratum that carries a claim.
6. Strata (vendor, site, protocol, lesion type, demographics) and the poolability and heterogeneity tests.
7. Reference standard, truthing process and scoring rules, with truthers separate from readers.
8. Standalone performance endpoints and analysis methodology, set before standalone data are collected.
9. Data firewall, access log and audit trail.
10. The final statistical analysis plan, submitted before outcome data exist.

Changes that were not planned in advance can severely weaken a pivotal study. Analyses added later belong in sensitivity analyses, reported alongside the planned ones.

What the design still cannot settle

The open issue in a multi-vendor programme is not the MRMC statistics. It is per-vendor and per-subgroup powering: which strata carry claims, and whether enough cases exist to support them. Every stratum carrying a claim must be sized for it. It belongs at the start of the programme, not after the first interim look. Kim and colleagues also recommend registering diagnostic accuracy studies in advance, with their analysis plans, because AI performance varies between institutions and selective reporting is a real risk.

FAQ

What is the difference between sensitivity and specificity?

Sensitivity is the share of subjects with the target condition in whom the test is positive, TP/(TP+FN). Specificity is the share of subjects without the condition in whom the test is negative, TN/(FP+TN).

What is proportional stratified sampling?

Proportional stratified sampling draws from each stratum in proportion to its share of the population, so the overall estimate reflects the population mix. Its weakness for device validation is that small strata end up with too few cases to estimate their own performance precisely.