A multicentre reader study validation needs four decisions fixed before anyone reads a case. They are how reads are sequenced (sequential, or crossed over with a washout), how readers and cases are laid out (fully crossed or split-plot), how centre enters the analysis, and how missing or inconsistently coded outcomes will be handled. All four belong in a statistical analysis plan that is locked before unblinding. FDA’s guidance on computer-assisted detection (CADe) devices states that pre-specification of the statistical analysis is a key factor for obtaining consistent and convincing scientific evidence. Any of these decisions made after the results are visible becomes a question the reviewer has to ask, and the study team can rarely answer it cleanly.

The rest of this page takes the decisions in the order a study team meets them. The reading scheme and reader layout come first because they set what the data can show. Centre effects and missing data come next because they decide what the pooled number means. The plan lock comes last, and it is what makes the other decisions defensible.

Sequential reading or a crossover with washout?

FDA’s CADe guidance describes two basic schemes. In a sequential design, readers make a conventional unaided read and then see the CADe output immediately after it. In a cross-over design, unaided and aided reads happen in separate sessions divided by a washout period meant to erase reader memory of the images. The guidance matches the scheme to the device’s intended use: sequential for second-reader devices, cross-over for concurrent-read devices.

Each scheme has its own weak point. Because the two reads sit so close together, the sequential design keeps intra-reader variability low. However, FDA notes that the CADe effect can be confounded with the extra reading time per case, and that readers may undercall the unaided read and so inflate the apparent benefit. FDA’s suggested mitigation is to randomise a fraction of cases, separately for each reader, to be read only unaided, with the randomisation revealed after the unaided read. The cross-over design cancels session effects in the comparison, but the two sessions add variability to the estimate of the CADe effect.

The washout rule is explicit. When the same cases are read in more than one session, FDA recommends separating the sessions by at least four weeks to avoid memory bias, notes that longer gaps may be advisable, and asks for supporting data whenever the proposed gap is shorter or longer. Four weeks is FDA’s recommended minimum, not a proven threshold. A study comparing sequential and independent reading notes that the minimum time lag between sessions is still unknown.

How do counterbalanced case ordering and washout periods reduce reading-order and memory bias?

Counterbalancing randomises the order of conditions so carryover is meant to cancel, and washout shrinks the carryover itself. FDA names reading-order bias as the case where a reader’s interpretation is shaped by memory of the competing test, and recommends randomising readers, cases and reading scenarios. Latin squares within a block design are one route.

Counterbalancing is often assumed to remove carryover: if half the cases are read aided first and half unaided first, the order effects should cancel. That argument rests on an assumption. A 2025 analysis of counterbalanced within-subjects designs shows that counterbalancing works only if carryover effects are symmetric, and that this symmetry cannot be verified in advance. When carryover is asymmetric, the treatment-effect estimate can be biased even with proper counterbalancing.

The same authors propose three remedies: a washout period, a heuristic check that compares the estimated effect in the first period with the effect in the second, and adjustment for covariates that shift between periods. They describe a marked gap between the period-one and period-two estimates as a warning sign of carryover or time drift. They also say washout length should come from subject-matter knowledge of how long the effect takes to fade. For a reader study, we would write the period comparison into the plan as a pre-specified diagnostic, with an agreed response if it fails (for example, reporting the period-one comparison as a sensitivity analysis). Writing the check in advance keeps it from turning into a post hoc rescue.

Fully crossed or split-plot for a multicentre reader study?

Choose fully crossed when readers can carry the workload. Choose a paired split-plot when they cannot. FDA describes the fully crossed multi-reader multi-case (MRMC) design, where every reader reads every case, as offering the greatest statistical power for a given number of cases. FDA also accepts non-fully crossed designs, for example in prospective studies where several clinicians cannot all interpret the same patient data.

The catch with full crossing is reader time. Chen, Gong and Gallas (2018) quote Obuchowski’s point that, in a fully crossed design, study length and the time commitment of each reader can be great. In their paired split-plot design, readers and cases are paired across modalities, and a split-plot design is used within each modality. The authors show theoretically that with a fixed number of readers and a fixed workload per reader, the paired split-plot design is more efficient than the fully crossed one, both for measuring a single modality and for comparing two.

In a multicentre study the decision often comes down to logistics. Fully crossed works when cases can be centralised and every reader can reach them. Split-plot fits when reader time is the binding constraint, or when readers are tied to their own site’s cases. In that second case, the split-plot blocks can follow site boundaries, provided the analysis accounts for it. We cover sample sizing, endpoints and the MRMC variance analysis that goes with each layout in designing a defensible MRMC validation study for AI-assisted diagnostics.

How do you reduce reader variability and disagreement across sites?

Reader variability is wide, both between and within readers. It is sampled and measured, and the design decides whether the result generalises beyond the readers who took part. The same study of sequential versus independent reading lists experience, visual skill, fatigue, motivation and differing decision aggressiveness as major drivers of interreader variability. It argues that MRMC studies, especially fully crossed ones, let that variability be measured and extrapolated to wider reader and case populations. A prospective single-centre study of liver metastasis detection at CT examined variability linked to reader experience, image navigation patterns and eye gaze time at missed lesions. It separates search errors, where the eye never fixates on the lesion, from classification errors, where the lesion is seen but not reported.

Several design levers follow from FDA’s CADe guidance:

• Reader sampling. Readers should be selected to represent the intended population of clinical users, and the protocol should record their qualifications, experience and training.
• Fresh readers. FDA recommends new readers for each new clinical reader study.
• Separate truthers. Clinicians who establish the reference standard should not be the readers in the core assessment, and the method for combining several truthers’ interpretations, including how their disagreements are resolved, should be described.
• Fixed scoring. The scoring process should be described and fixed before the evaluation starts.
• Representative hardware. Imaging hardware should match current clinical practice.

In a multicentre setting we would add one rule of our own: hold the reading environment (display, hanging protocol, viewer tools) as constant across sites as practice allows, and record it where it can’t be held constant. The navigation findings above suggest that how readers use the workstation is part of reader variability, not noise around it.

Should centre be a random or a stratifying effect?

Centre should be in the analysis model whenever it was used to stratify the design. When centres are small or unbalanced, random centre effects with a degrees-of-freedom correction are the safer default. Kahan and Morris (Statistics in Medicine) show that when randomisation is stratified by centre, the analysis has to adjust for centre to get correct inference, and they recommend pre-specifying that adjustment in the protocol or analysis plan. Kahan’s companion paper on binary outcomes makes the same recommendation as a default position, regardless of the expected intraclass correlation.

Fixed centre effects use only within-centre comparisons, while random centre effects combine within-centre and between-centre information. Across 378 simulated scenarios, Kahan and Morris found random centre effects superior or equivalent to fixed effects in every one. Across 108 scenarios, random effects gave a median power gain of 13.9% over fixed effects. The weakness of random effects is small-sample inference: they rely on asymptotic results that can produce intervals that are too narrow. A degrees-of-freedom correction restored nominal coverage in the scenarios with few patients per centre. The authors recommend random centre effects as the default when centres have fewer than 10 patients or when assignment within centres is likely to be unbalanced.

ICH E9 adds two rules that matter here. First, where the meaning of “centre” is in doubt, the protocol should define it (by investigator, location or region), and any rules for combining centres should be specified in advance and always decided blind to treatment. Second, the main effect is investigated in a model that allows for centre differences without a treatment-by-centre interaction term, and Kahan and Morris note that ICH E9 treats that interaction as exploratory. ICH E9 also describes mixed models with random centre and treatment-by-centre effects as especially relevant when the number of sites is large.

This evidence comes from multicentre randomised trials of patients. In a reader study, the unit is the case and the comparison is aided versus unaided reading. We think the logic carries over: centre still marks a cluster of cases that share acquisition protocol, case mix and coding practice. The small-centre warnings apply to sites that contribute only a handful of cases. FDA asks for the same discipline from its own side, recommending that poolability of multi-site data be justified both statistically and clinically.

Handling missing or inconsistently coded outcome data across centres

To handle missing and inconsistently coded outcomes across centres, characterise the missingness for each field and each centre first, and only then choose between imputation, exclusion or stratification. Keep centre in the model whichever you choose, and document the choice and its sensitivity analysis before unblinding.

TechnoLynx can do the engineering behind this for a validation dataset. That means cataloguing how practice and missingness vary across centres and years of historical data, building the harmonisation and missing-data pipeline with its diagnostics, comparing pooled with centre-adjusted validation metrics across the candidate handling choices, and packaging the method and rationale so a governance board can review it. The first step is the catalogue, because every later choice is justified against it. Our wider AI engineering work for life sciences and healthcare is where this sits. The work would produce documentation suitable for board or governance review. It is not regulatory sign-off, and it does not replace the sponsor’s statistician or the governance body’s own judgement.

Why the missingness mechanism decides the method

The right method depends on the missingness mechanism. Groenwold and colleagues, citing earlier work, state that complete case analysis is not valid in non-randomised studies when data are missing at random, and recommend multiple imputation in that situation. They also note that the simpler missing-indicator method typically gives biased estimates in non-randomised studies, whether data are missing at random or completely at random. A 2026 review of bias in clinical research (Communications Medicine) makes the same point: complete case analysis and mean imputation can bias estimates and produce artificially narrow confidence intervals.

Validation adds a twist that development work doesn’t have. One methods paper argues that the right handling of missing values in validation data depends on how the model will be used. If the model has to cope with missing inputs in practice, validation should use methods that rely only on the development data and that can be applied to a single new patient. Imputing independently within the validation set uses information a deployed model would not have.

For a reader study, a key gap is missing ground truth, not only a missing covariate. FDA defines verification bias as ground truth being missing for some subjects, and warns that analysing only the subjects with established truth can bias performance estimates.

A decision path for impute, exclude or stratify

Missingness pattern (per field, per centre)	Preferred handling	What must be documented
Low rate, plausibly missing at random given observed variables	Multiple imputation, with centre in the imputation model	Imputation model, variables used, number of imputations, sensitivity analysis under less favourable assumptions
Systematic, concentrated in particular centres or fields	Exclude the field or records, or stratify and report results by missingness pattern	Counts included in each analysis, so the impact can be assessed
Missing ground truth for some cases	Treat as verification bias; do not silently restrict to verified cases	How truth was established and for whom
Coding changes across years or sites	Map codes to a common definition, with a flag separating genuine practice change from entry error	The mapping table and the reasoning behind each flag

Four caveats come with this table. Imputation can hide a real data-quality problem if the mechanism has been misclassified. Centre-stratified analysis needs enough cases per centre, so small centres may need to be pooled or shrunk through a hierarchical model. Coding drift across a decade can reflect real changes in clinical practice rather than entry error, and treating both as noise biases the interpretation. Finally, a board needs to see the residual uncertainty, not just the chosen method. ICH E9(R1) asks for pre-specified sensitivity analyses that test how far the assumptions can be stretched before the conclusion changes, tipping-point analysis being one example.

Worked example: site poolability and small-site pooling in a locked plan

The statistical analysis plan for Medtronic’s PERIGON pivotal trial is an example of centre and missing-data rules written in advance. It applies to a heart valve, not an imaging device, but the rules are concrete enough to adapt:

1. Sites that implanted fewer than five subjects are classed as small sites and pooled into one site.
2. A Cox proportional hazards model tests the site and geography effect. A p-value above 0.15 for site means the data are treated as poolable.
3. If site p ≤ 0.15, clinically meaningful baseline risk factors are added and pared down by backwards elimination, keeping site in the model.
4. If site becomes non-significant after adjustment, the data are poolable. If not, results are presented by centre and the clinical significance of the differences is assessed.
5. Missing data are not imputed unless an objective says otherwise. Subjects with missing data drop out of that analysis, and the number included in each analysis is reported.

Translated to a reader study, the same skeleton would set a minimum case count per centre, a pre-stated poolability test on reader performance with centre as a term, a fallback to reporting by centre, and a missing-truth rule. The thresholds themselves would need their own justification. The 0.15 cut-off is that plan’s own choice. The value of the example is that every branch was written down before the outcome was known.

Why the analysis plan and missing-data rules must be locked before unblinding

Locking the plan before unblinding shows that the analysis was fixed before the results could influence it. ICH E9 says the statistical methods should be fully specified before information on outcomes and treatment assignments is available. The analysis plan may be updated after a blind review of the data, but it should be finalised before the blind is broken. ICH E9 also requires the reasons for excluding subjects from the per-protocol set to be fully defined and documented before unblinding. A trial can still be regarded as valid despite missing values if the methods for handling them are sensible, particularly when those methods were predefined in the protocol. Redefining the primary variable after unblinding is described as almost always unacceptable.

The ICH E9(R1) addendum covers sensitivity analysis as well. The protocol and analysis plan should pre-specify the main estimator together with a sensitivity analysis that tests robustness to departures from its assumptions. Results should be reported stating whether each analysis was pre-specified, introduced while the study was still blinded, or run post hoc. CASRAI’s analysis plan guide explains why the timing has to be provable: a plan amended after unblinding cannot rule out that the change was influenced by the results, which is why version history and sign-off dates serve as the audit trail. Analysis firewalls are the AI-study analogue: locked holdout sets evaluated by an independent analyst, with model development kept apart from final performance assessment.

What the locked plan for a multicentre reader study should fix

• Reading scheme (sequential or cross-over), washout length and the evidence for it
• Randomisation of readers, cases and reading scenarios, with a flowchart
• Reader-case layout (fully crossed or split-plot) and the justification for reader and case numbers
• Period-comparison diagnostic for carryover, and what happens if it fails
• Definition of centre, rules for combining small centres, and the fixed or random centre model, with the degrees-of-freedom correction where centres are small
• Treatment-by-centre interaction designated as exploratory
• Missingness taxonomy per field and centre, the chosen handling for each, and sensitivity analyses under less favourable assumptions
• Reference-standard process, truther separation and scoring, all fixed before reading begins
• Version history and sign-off dates showing when each item was fixed

The question a team still has to settle for itself is which centres are genuinely comparable. A statistical test alone does not answer that. It takes clinical judgement about acquisition, case mix and coding at each site, written down before anyone knows which answer would help.