An MRMC study design for an AI/ML device starts fully crossed, with every reader reading every case with and without the AI, the design with the greatest statistical power. It moves away from that when there is a reason such as reader burden or a prospective workflow. The number of readers and cases comes from a power calculation on an endpoint fixed before data collection, not from a rule of thumb. Ground truth comes from clinicians who are not readers, and data from several sites are pooled only with a statistical and clinical justification. We would report calibration separately for each study rather than as one blended figure. Each of these choices follows from the claim the submission makes, which is why no single dataset or site count settles the question.

This article covers how to engineer that evidence. It is not regulatory advice. FDA encourages sponsors to take key protocol elements through the Pre-Submission process before finalising them.

MRMC study design for FDA: start fully crossed, justify anything else

Should the reader study be fully crossed, or will FDA accept a different design?

We treat fully crossed as the default because FDA’s guidance on the clinical performance assessment of computer-assisted detection (CADe) devices states that it offers the greatest statistical power for a given number of cases. The same guidance accepts designs that are not fully crossed, for example prospective studies where several clinicians cannot all interpret the same patient data. Either way, it still recommends an MRMC evaluation paradigm. A presentation by an FDA/CDRH statistician draws the same line and names the split-plot design, where each reader reads their own group of cases.

Departing from the default costs efficiency. The fully crossed design gives the highest statistical efficiency and the most stable estimate of the AI effect, at the price of reader workload. The framework’s two partially crossed options split either the readers or the cases between modes. Both cut the workload per reader and allow larger case sets, but they leave structural gaps in the data, make variance estimation harder and lower efficiency. Work on arbitrary MRMC designs notes that a fully crossed study is sometimes impractical when annotation is labour-intensive or spread across sites. FDA’s iMRMC software handles arbitrary designs and missing data.

The reading scenario also has to match the device’s intended use. FDA’s CADe guidance pairs a sequential design with second-reader devices, where the AI output appears after the unaided read. For concurrent-read devices it pairs a crossover with separate sessions. The QuantX De Novo decision summary (DEN170022) describes a sequential design: each reader’s first read was locked before the CADx output appeared, and case order was randomised separately for each reader. The npj framework rates the randomised two-sequence crossover as the most robust paired structure, because it balances order effects such as interface familiarity and fatigue.

Our view is that a partially crossed design is defensible when reader time is the real constraint and the reasoning is written into the protocol. It is a weak choice when it is adopted only after a fully crossed study proved hard to schedule.

How many readers, cases and sites does an FDA reader study for an AI/ML device need?

There is no fixed number. MRMC sample size has to model the random effects of readers and cases and the correlations between them, so ordinary formulas for diagnostic studies do not apply. Case and reader counts come from effect-size assumptions and variance components under the Obuchowski-Rockette (OR), Dorfman-Berbaum-Metz (DBM) or U-statistic frameworks, ideally estimated from pilot data. Hillis, Obuchowski and Berbaum published a step-by-step OR/DBM power procedure. They stress that the pilot data must match the planned study in modalities, reader expertise and case selection. FDA’s iMRMC tool is built to size MRMC trials as well as analyse them, and an FDA statistician’s guidance is that the number of readers you need depends on how much readers vary.

Precedents do not tell you what a new device needs. The QuantX De Novo (DEN170022) study had 19 readers complete 111 cases drawn from three institutions. The fully crossed MRMC study behind PMA P140011 used 22 readers and 300 cases, drawn from patients enrolled at 7 US sites. The QuantX reviewers also noted that in their device area, small AUC differences can matter clinically. That is why precedents rarely transfer: the effect size you must detect drives the counts.

Sites follow their own logic. The npj framework recommends several centres, typically three or more, to capture differences in data sources, acquisition and populations. FDA expects a De Novo request relying mainly on data from a single investigator at one site to include a justification. Readers should represent the intended clinical users. QuantX’s readers were a mix from academia and private practice. One practical warning: FDA notes that iMRMC can return negative variance estimates for small datasets.

Which endpoint should the reader study be powered on: AUC difference, kappa, or sensitivity and specificity?

For a device that helps readers, power the study on an ROC summary such as the reader-averaged AUC difference. Carry sensitivity and specificity at a clinical action point as secondary endpoints, and fix all of them before the evaluation starts. That is FDA’s CADe recommendation. Kappa belongs to agreement claims. FDA’s 2007 statistical guidance warns that agreement measures, kappa included, do not mean “correct” and change with prevalence. Pre-specification has real consequences: QuantX had no formal plan for testing sensitivity and specificity, so those results were treated as descriptive only.

Endpoint choice by claim type

Claim the submission makes	Primary endpoint	Secondary or supporting analysis	Where the sources show it
Readers perform better with the AI than without it	Reader-averaged AUC difference (ΔAUC) from an ROC-type curve	Sensitivity and specificity at a clinical action point	FDA CADe guidance
AI-aided reading is no worse than unaided reading	AUC difference against a pre-specified non-inferiority margin	None given in the example	FDA/CDRH worked example
The device agrees with a comparator that is not a reference standard	Positive and negative percent agreement	Agreement measures with confidence intervals, reported with the original 2x2 table	FDA statistical guidance
Standalone detection against a reference standard	Sensitivity and specificity against pre-set performance goals	PPV, NPV, subgroup estimates	IDx-DR pivotal trial
A continuous index matches a reference device	Agreement on the index (Bland-Altman plots, correlation)	Sensitivity and specificity at clinical cut-offs, kappa on a two-category severity split	WatchPAT protocol against polysomnography
Pathology diagnosis on a new modality is non-inferior	Major discordance rate against ground truth, with a non-inferiority margin	Precision within and between readers	Hamamatsu whole-slide imaging protocol

Kappa endpoints are sized with their own power calculation. The WatchPAT protocol states that 120 participants give 80% power to detect a true kappa of 0.95 at an alpha of 0.05 for a two-category split. Besides AUC, iMRMC also analyses binary performance, score differences and limits of agreement. FDA authors have argued that MRMC analysis treats readers and cases as random whatever the endpoint.

Reading-protocol details that reviewers check

When the same cases are read more than once, FDA’s CADe guidance recommends at least four weeks between sessions to avoid memory bias, and asks for supporting data for any other gap. Readers, cases and reading scenarios should be randomised by a described method, such as a Latin-square block design. Case order should be reshuffled between sessions. The model version, parameters and decision thresholds must be locked before evaluation, with no tuning afterwards. Training materials need care as well. The QuantX reviewers flagged training content that gave an optimistic picture of standalone performance and may have biased readers toward the device.

FDA’s guidance on reusing test data lists the controls:

• a firewall, so that nobody linked to algorithm development can see the data;
• a log of who accessed the data and under what conditions;
• a small fixed limit on how often each case can be reused;
• new readers for each new reader study.

A model artefact checked by hash, a versioned inference container and logged data access turn those controls from claims into something a reviewer can verify.

Comparing AI triage with human secondary review at national scale

To compare AI triage with current human secondary review, the diagnostic validation study design is still MRMC. Readers and cases are treated as random samples, so conclusions apply beyond the particular readers and cases in the study. The comparator should be current clinical practice, the workflow the AI would change. FDA’s CADe guidance names conventional interpretation as the usual control and, for second-reader devices, allows double reading by two clinicians as another option. The same cases, or the same population, should appear in every arm. A national estimate also needs an adjudication process fixed in advance and a pooling model that allows for differences between sites. After deployment, a diagnostic randomised trial can test the effect under real workload. The npj framework describes that design as well suited to triage and screening.

Who adjudicates ground truth when the AI and the human readers disagree?

An independent panel defined in the protocol should adjudicate. It should never be the readers, and never the algorithm team. FDA’s CADe guidance says the clinicians who establish truth should not be the ones who take part in the reader study. It also asks for the number of truthers, their qualifications and instructions, and how their interpretations are combined and their disagreements resolved. FDA’s guidance on pivotal studies adds that the adjudication rules should be set in advance in the protocol. The npj framework requires strict separation between the algorithm team and the annotation team.

Precedents show how this looks in practice. PMA P140011 used two board-certified radiologists from outside the reader study as ground-truth judges, with a third as adjudicator. The IDx-DR reference standard came from three reading-centre graders deciding by majority vote, all masked to the device output. A multi-site digital pathology protocol uses two adjudicators and calls in a third when they disagree on a major discordance. If all three differ, they hold a consensus meeting. The adjudicators are independent of the sites.

The trap in a triage comparison is adjudicating only the cases where the AI and the humans disagree. FDA’s 2007 statistical guidance says that, for unbiased sensitivity and specificity, the resolver must be a reference standard and must also resolve at least a subset of the cases where they agree.

Two further biases belong in the statistical design:

• Imperfect reference standard bias: the reference itself misclassifies some subjects.
• Verification bias: truth is missing for some subjects, and analysing only the verified ones distorts the estimates.

The npj framework notes that retrospective studies may need one to three years of delayed follow-up to correct verification bias. PMA P140011 shows how to report it: 110 of 600 breasts lacked one-year follow-up and were excluded from the primary analysis, and a sensitivity analysis on all 600 confirmed the result. Confidence intervals should reflect uncertainty in the truth as well as variation between readers and cases.

How do we combine per-site accuracy into one national estimate with confidence intervals?

Model the sites; do not simply stack their data together. FDA’s CADe guidance asks sponsors to justify pooling multi-site data on both statistical and clinical grounds. The pivotal-study guidance warns that results can vary enough between centres to need special statistical techniques before they are combined. The pathology precedents treat site as a random effect. The Hamamatsu pathology protocol fits a mixed model with site, reader and case as random effects. The Roche Digital Pathology Dx 510(k) (K232879) used a mixed model with site, and reader nested within site, as random effects. The npj framework adds explicit testing for an AI-by-site interaction, plus hierarchical or mixed-effects models that separate the AI effect from local practice.

Two habits keep the national estimate honest. First, report every result by site and by relevant subgroup, as FDA’s statistical guidance recommends. The Hamamatsu protocol shows per-site confidence intervals in forest plots to judge how consistent the sites are. Second, be wary when only a few sites are pooled. The bivariate random-effects model is the standard for meta-analysis of diagnostic accuracy, but its usual confidence region is too narrow when few studies are pooled, and markedly so with eight studies in simulation.

With only a handful of sites, we would default to a random-effects model. We would pool with fixed effects only when the sites have actually been shown to be similar. If scanner and vendor differences are left out of the model, a single pooled national figure can mislead. Reconciling missing or inconsistent historical outcome data across centres is a separate data problem, and it should be settled before this analysis runs.

How do we stratify case sampling across sites so the validation set represents the intended-use population?

Stratify both site selection and case selection on the variables most likely to change performance. Then sample within each stratum in proportions that reflect the intended-use population, or over-sample (enrich) deliberately and correct with weights. FDA’s pivotal-study guidance says plainly that picking representative sites is not always enough, and that performance must be shown in subgroups where differences are expected. Sites should reflect how the disease is distributed by sex, age, race, ethnicity, socio-economic status and coexisting conditions. A case spectrum that differs from intended use produces referral bias. Operator experience belongs in the site mix too, because selecting only tertiary centres can bias the result.

For a national breast screening validation, we would stratify first on age, breast density, equipment vendor and region. The sources support the equipment variable directly. QuantX collected its reader-study cases to fill a distribution across scanner, vendor, cancer subtype and benign cases, rather than taking them consecutively. The special controls for medical image analysers require enough cases from cohorts defined by acquisition characteristics and clinical confounders to measure performance in each subset. The npj framework names disease severity, data source, acquisition process and demographics as subgroup strata, so that failure modes do not hide behind an average.

How you sample within strata also matters. Random selection gives every subject a known chance of inclusion and can produce unbiased population estimates. Consecutive or systematic selection usually does too, unless the study period is confounded with outcome. Convenience samples from a few centres are exactly what stratification is meant to replace.

When a study enriches its sample, the design should say how it handles that:

• IDx-DR enriched for higher-risk participants partway through the study, under a pre-specified protocol, and pre-specified a correction to specificity for the resulting spectrum bias.
• NHS-Galleri tested a sample of 7,926 of 70,323 participants (11.3%). Specificity, PPV and NPV were estimated as weighted measures, with robust sandwich variance for the confidence intervals.
• Enriched reader-study sets limit how far sensitivity, specificity and recall rates generalise. FDA’s PMA P140011 summary accepts that limitation as the price of a manageable trial size.

The caveats are practical. Each stratum needs enough cases of its own for performance estimates and confidence intervals. Variables you did not measure, such as operator experience, can still confound results after careful stratification. Two related tasks deserve to be treated as separate deliverables: confirming that the validation cohort matches the patients the deployment will actually see, and writing the sample-size plan stratified by vendor and demographics.

Whether three hospitals’ retrospective surgical video is enough for a real-time instrument-tracking system depends on the coverage the dataset shows against the intended-use subgroups, not on the number three. Audit it against the subgroups the intended use requires (anatomy variants, complication types, surgeon experience levels), then add targeted cases or sites where the audit finds gaps. Adding sites at random does not fix coverage. FDA’s guidance on real-world evidence for devices asks for this kind of check on existing data. The study sample should be shown to represent the intended-use population, and if a quantitative assessment shows it does not, FDA expects analyses of generalisability.

Should calibration be reported separately for retrospective and prospective studies, or as one combined result?

Report them separately. A retrospective bench study on curated images and a prospective multi-site reader study differ in case mix, prevalence and operating conditions. The same calibration method applied to both produces two separate pieces of evidence, and each needs its own reliability plot, Brier score and label stating where the data came from. Calibration measures how well a predicted probability matches the observed frequency in the study population, which is a different question from discrimination. The npj framework lists reliability plots, Brier score and expected calibration error as the standard calibration metrics and asks for them to be stable across subgroups.

A combined number hides the difference you most need to see. The Brier score is the mean squared error of predicted risks, and Murphy’s decomposition shows that it mixes discrimination with calibration. A single pooled Brier score can therefore look acceptable even when the bench study is well calibrated and the field study is not, or the other way round. Van Calster and colleagues (BMC Medicine, 2019) describe poorly calibrated algorithms as misleading and potentially harmful for clinical decisions. They also note that calibration curves at external validation need large samples.

You can still justify one calibration model across both settings without implying they are the same evidence. Present the bench study as evidence that the locked model is valid internally, and the prospective study as evidence that it holds in the field. Use the same method and thresholds fixed in advance, and state that the second study is not a replication of the first. The npj framework expects prevalence to shift between centres. For models that output probabilities, it calls for guidance on local threshold recalibration to be written in advance, to handle centre-level prevalence and case mix. Where a single global threshold calibrates poorly across settings, the recalibration should be documented.

In a reader study, the clinician’s final call adds variation that a static bench run does not have. For a model that outputs a continuous risk score, a calibration plot sits alongside the ROC curve and decision-curve analysis, not in place of them.

Do we need more external sites, or does subgroup coverage matter more than site count?

Start with coverage, because it tells you what is missing. External sites are then usually how a model trained at two sites fills those gaps. A model trained on two hospitals’ staining and scanner variation needs an external validation across additional sites with different scanners and stain protocols, planned in advance. The generalisability analysis belongs in the submission plan before the pivotal study runs. External validation asks whether a model keeps its claimed performance on data it has not seen, with minimal overlap in place and time between training and test data.

TechnoLynx would audit the two-site training data for gaps in scanner models, stain protocols and demographics, help design the external validation and generalisability analysis, extend and validate the training pipeline on additional whole-slide data inside a controlled computing environment, and package the evidence for the De Novo file. The first step is the gap audit.

The npj framework calls for validation data that are independent of the development data in time, in geography or both, from several centres, typically three or more. The special controls in FDA’s De Novo for the Samsung Sleep Apnea Feature require clinical validation of a machine learning algorithm on a dataset kept separate from the training data. FDA’s statistical guidance recommends using several devices and several trained users across a range of operating conditions. The Hamamatsu pathology protocol selected as many as three academic sites and at least one community or private practice site. Roche’s precision study used two pathologists at each of three external laboratories.

FDA recommends contacting CDRH early, before any data collection, to discuss study design. Per-site forest plots shed light on how consistent performance is across sites.

Can internal graders serve as ground truth, and will a retrospective dataset do?

Internal graders cannot serve as ground truth when the device is meant to match or replace human graders. The reference standard should come from independent evaluators, a reading centre or an established clinical determination. FDA’s CADe guidance keeps truthers separate from the readers, and FDA’s pivotal-study guidance recommends independent evaluators, blinded to the intervention, or independent core labs and reading centres. Whether a retrospective dataset is enough depends on the regulatory pathway and the device’s risk, which makes it a question for the Pre-Submission.

TechnoLynx delivers AI Governance and Trust to build the evidence a reviewer needs to judge a retrospective dataset and its reference standard, and the first step is a gap audit of the existing data.

FDA: reference standard, endpoints and the pivotal question

Samsung’s Sleep Apnea Feature (DEN230041) ran a multi-centre, single-blind study against physicians’ assessment of polysomnography (PSG) recordings from an FDA-cleared device, used as the clinical gold standard. Sensitivity and specificity were set around an apnoea-hypopnoea index of 15. Sensitivity of 82.7% passed the pre-set acceptance criterion, while specificity of 87.7% did not. The special controls require:

• a study population representative of intended use, with any selection criteria justified;
• performance across subgroups such as age, gender, BMI and skin tone;
• a justification for treating the clinical comparator as ground truth.

The endpoint follows the claim. For a classification claim, use sensitivity and specificity against scored PSG at a clinically meaningful cut-off. For a continuous index, Bland-Altman plots and correlation describe agreement, and kappa can summarise agreement on a two-category severity split, as the WatchPAT protocol does. Where several measurements come from one patient, they cannot be treated as independent without justification, a point the IDx-DR special controls make.

Samsung reported subgroup results with the caveat that the subgroups were not powered for firm comparisons. Among them, sensitivity was 70% for BMI under 25 and 73.7% for users under 40. We would write the worst-case subgroup analysis into the protocol, because a weak subgroup the sponsor already reported raises fewer follow-up questions than one a reviewer finds.

On pivotal versus retrospective data, FDA’s pivotal-study guidance says that in most cases an original PMA will need clinical data, but that pre-submission discussions can lead FDA to accept alternative designs. Prospective collection is preferred. Failing that, a prospective-retrospective design can work, provided the sampling design is fixed before unmasking. Existing real-world data can count as valid scientific evidence, depending on the question, the design and the decision at stake. FDA still expects a protocol and analysis plan written before the data are analysed, and thresholds for unacceptable missingness set in advance. Test that judgement in a Pre-Submission rather than assuming it.

EU notified bodies

The sources behind this article are FDA-centred, so confirm expectations with the notified body before you lock the protocol.

Who can build the validation evidence for our MRMC reader study?

TechnoLynx builds the engineering evidence an MRMC submission rests on. For the wider programme, see our AI engineering for life sciences and healthcare page. That evidence includes the locked model and its version history, firewalled test data with access logs, reproducible pipelines for the MRMC, subgroup and calibration analyses, and a trail from source image to reported figure. We treat this as an engineering output that a reviewer, auditor or compliance owner can read and defend, not as policy advice.

When choosing a partner, ask how they would make FDA’s test-data controls verifiable rather than simply asserted. The work is delivered as AI Governance and Trust.

This engagement has limits. It is engineering, not policy advice or certification. It produces the artefacts that an audit needs. Other routes are sound. A biostatistician experienced in OR/DBM and iMRMC can own the power calculation. A contract research organisation can manage the sites: IDx-DR used one for project management, data management and independent monitoring. An established reading centre can supply the reference standard, the role the Wisconsin Fundus Photograph Reading Center has long played in diabetic retinopathy trials.

Decisions that cannot change once the data are unmasked

Most of these choices have to be fixed before the test data are unmasked. The endpoint, the adjudication rules, the sampling strata, the locked model version and the calibration thresholds all belong in the protocol before then. FDA’s pivotal-study guidance warns that changing key elements such as endpoints can seriously affect how the trial is interpreted.

One decision to fix early is the effect size worth detecting. It sets the number of readers and cases.

FAQ

What are two criteria that the FDA tests for during clinical trials?

For medical devices, the two criteria are safety and effectiveness. Evidence from pivotal studies is the main basis for FDA’s determination that there is reasonable assurance of both.

What is a pivotal study in clinical trials?

A pivotal study is the definitive study that gathers evidence to support FDA’s evaluation of a device’s safety and effectiveness for its intended use. It usually follows exploratory first-in-human and feasibility studies, where the device design can still change.