A multi-centre validation of a donor-recipient matching model holds up when the whole analysis is fixed before anyone sees results. That means fixing which centres and transplants enter, how large the sample must be, which subgroups are reported with which metrics, what counts as non-inferior to the current allocation score, who runs the analysis and which survival model carries the primary endpoint. We would write that once as a pre-registered protocol, so the same study can answer an ethics board and supply the evidence the OPTN policy process asks for. OPTN approval requires evidence that a proposal addresses its stated problem and complies with NOTA and the Final Rule, and only the Board adopts policy. We would expect an FDA submission built on the same study to need adapting, not just relabelling.
This page sets out the design piece by piece. The aim is a single package that serves generalisability, subgroup fairness and governance together, rather than three studies run one after another.
The validation protocol at a glance
Each row below is something to write into the protocol and timestamp before the held-out data is opened. Each one maps to a question a reviewer will ask.
| Design element | What to fix in advance | Who it answers |
|---|---|---|
| Cohort and centres | Inclusion window, exclusion criteria, centres included, how coding changes over the years are reconciled | Ethics board, OPTN committee |
| Sample size | A calculation per performance measure, with the largest result adopted | Ethics board, statistical reviewers |
| Generalisability | Internal-external cross-validation by centre, with between-centre heterogeneity reported | OPTN committee, FDA |
| Subgroups | Named subgroups, metrics per subgroup, and the minimum acceptable performance for each | Ethics board, OPTN Minority Affairs review |
| Comparator test | Non-inferiority margin against the existing score, then superiority, with subgroup interaction tests | OPTN committee, FDA |
| Independence | Who executes or audits the analysis, data access route, monitoring committee if prospective | Ethics board, conflict-of-interest review |
| Survival model | Primary model, competing-risks secondary analysis, proportional-hazards checks by centre | Statistical reviewers |
| Post-deployment review | Performance indicators, baseline values, review schedule | OPTN policy evaluation |
How do you validate an allocation model across transplant centres so the results generalise?
You hold centres out, not just random patients, and you report how much performance varies between them as well as the average. Internal-external cross-validation (IECV) is built for this kind of clustered data. Takada and colleagues (J Clin Epidemiol, 2021) show that IECV can quantify generalisability across settings and populations in a way traditional internal validation cannot. In their example, adding predictors slightly improved discrimination but increased between-practice heterogeneity, which hurt generalisability. That trade-off matters for allocation models, where a richer feature set can look better pooled and travel worse.
Case mix is the second thing to plan for. Debray et al. (BMJ, 2017) note that the C statistic can vary substantially across validation settings because of differences in the distribution of patient characteristics, even when the regression coefficients are correct. They also recommend extracting and reporting case-mix information, such as the standard deviation of the linear predictor, for each validation study. Borrowing Debray’s meta-analysis method, pooling performance across centres with a random-effects model gives an estimate of between-centre standard deviation alongside the average. Summaries should be presented with confidence intervals and, where appropriate, prediction intervals. A prediction interval answers the reviewer’s real question: what performance should a new centre expect?
A decade of retrospective registry data brings its own problem. Coding practice, protocols and case mix can drift over that period, and no amount of design rigour fully repairs a generalisability claim built on inconsistent outcome definitions. Clinical practice also changes: the SRTR team that refit the kidney donor risk index chose a four-year cohort over the original eleven-year window, starting after HCV-positive donor to HCV-negative recipient transplants became more common. Reconciling multi-site heterogeneity and missing data before evaluation is its own piece of work. The protocol should name the method, for example the multiple imputation by chained equations that SRTR uses in its risk models, rather than leave it to the analyst after unblinding.
How many centres, and how many cases per centre, before the result is not one population’s artefact?
Any fixed threshold is a judgement, and the protocol should say so and justify it. What the evidence does offer is tailored sample size calculations for external validation, in place of rules of thumb. Riley et al. (BMJ, 2024) note that many published external validations are too small, which produces wide confidence intervals and potentially misleading claims about reliability. They propose separate calculations targeting precise estimates of each performance measure. In their binary-outcome worked example, at least 949 participants (408 events) were needed to estimate all four measures precisely, with calibration the binding constraint. A narrower case-mix distribution pushed that requirement higher. For time-to-event outcomes such as graft survival, they suggest a simulation-based approach instead of closed-form calculation.
For the number of centres, our view is that it follows from what you want to claim. If the claim is “this model generalises to US transplant programmes”, enough centres are needed to estimate between-centre heterogeneity with some precision. Per-centre volume matters less for the pooled estimate than for any per-centre statement. SRTR’s own reporting makes the point: smaller programmes generate less information, so their intervals are wider. For some subgroups SRTR reports nothing at all because there are too few transplants or events. We would pre-register a rule for that case, such as pooling small centres by region or reporting them only within the IECV loop, rather than drop them quietly.
Which subgroup fairness metrics and thresholds should be pre-registered?
Pre-register named subgroups, a discrimination metric and a calibration measure for each subgroup, calibration plots by subgroup, and a minimum acceptable performance per group with its uncertainty, all fixed before results are seen. The reason is that population-level evaluation masks differential behaviour within the population. A critical appraisal of fairness metrics in clinical predictive AI (2025) recommends AUROC parity only when paired with a calibration parity metric. It recommends calibration plots stratified by sensitive attributes. Rather than aiming for statistical parity at all costs, it frames fairness as minimum acceptable performance for all groups. Our wider argument for stratified validation across sites and subgroups rather than one blended metric applies here directly.
Worked example: the SRTR plan for refitting KDRI without race and HCV
One public template comes from the OPTN Minority Affairs Committee’s proposal to refit the Kidney Donor Profile Index without donor race and hepatitis C status. SRTR analysts recalculated the index on 50,769 kidney transplants from 1 January 2018 to 31 December 2021, using the original variables, exclusion criteria and modelling method. Models were fit on an 80% training sample and assessed on a 20% held-out test sample. The analysis plan named the subgroups in advance:
- male and female recipients;
- recipients in each age quartile;
- recipients of kidneys from Black, White, non-Black and non-White donors;
- recipients of kidneys with original allocation KDPI of 85 or more, and of 20 or less;
- recipients of kidneys from HCV-positive donors, stratified by test type.
For each subgroup the plan specified concordance, the integrated Brier score and calibration plots comparing observed and predicted graft failure up to five years. Observed survival in those plots was estimated by Kaplan-Meier. The dated record shows the request submitted on 11 March 2024, the analysis plan on 12 March and the final report on 29 March. So the plan was dated before the final report. The finding was that removing race and HCV had no substantial impact on concordance or Brier score, overall or in any subgroup. The calibration plots still exposed a weakness of the KDRI/KDPI measure itself: the model tends to overestimate risk in the highest-risk donors.
Two lessons carry over to a new matching model. Calibration plots by subgroup surface problems that summary statistics hide, and that is their value. A weak absolute baseline also shapes what “not worse” means: internal concordance in the HCV-era cohort overall was about 0.6.
Small subgroups need uncertainty, not just point estimates
The same appraisal notes that most fairness metrics do not provide confidence intervals. That matters because sensitive groups typically have very different and often limited sample sizes. A transplant subgroup analysis (2025) used a one-sided nonparametric bootstrap with 1,000 resamples to test whether performance differed between pairs of subgroups. The authors note that this helps tell real bias from chance in small subgroups. In their transplant task, performance was lower for older recipients, women and recipients of deceased-donor organs. Intersectional subgroups shrink faster still. Our judgement is that thresholds set too strictly on rare subgroups will fail for reasons of sample size, not model behaviour. The protocol should therefore state the decision rule as “the interval excludes unacceptable performance” rather than “the point estimate clears a line”.
Showing non-inferiority or superiority against the existing allocation score, subgroup by subgroup
The test that shows a new matching model is no worse than the current score, and possibly better, is a pre-specified non-inferiority comparison with a fixed margin. It is followed by a superiority test only if non-inferiority holds, and it adds treatment-by-subgroup interaction tests so the conclusion is not purely aggregate. A multicentre randomised resuscitation trial protocol, for example, specifies non-inferiority first against a fixed limit, then superiority with two-sided 95% confidence intervals. The same protocol tests treatment-subgroup interaction for each predefined subgroup variable on the primary outcome, using logistic regression. That design comes from resuscitation, a different clinical field. What transfers is the structure of the analysis plan, not the margin.
Choosing the margin is the hard part, and it is a clinical and policy decision rather than a statistical one. The KDRI refit reported “no substantial impact” on fit statistics. That served the committee’s purpose, but an explicit margin stated in the plan would make the same conclusion harder to contest. For a matching model, the comparison should run on the same held-out centres and subgroups as the fairness analysis, so one table answers both questions.
Can an outside biostatistics group independently replicate the validation?
Yes. For US registry data the route is defined, and it takes planning time. SRTR releases standard analysis files for bona fide research under a signed data use agreement, renewed every three years. Any request needing a DUA must include a data security plan and a research plan. Linkage requests also need approval from the requestor’s Institutional Review Board and authorisation from HRSA, a review that can take up to 30 days. A group with a bona fide research purpose can request these files under its own data use agreement. That is what makes replication meaningful.
Independence can take one of two forms. In the first, the independent group executes the pre-registered analysis plan. In the second, the developer executes it and the independent group re-runs it from the locked protocol and the raw extract. If any part of the study is prospective, an independent data monitoring committee receiving unblinded interim summaries is the pattern that resuscitation trial used. Independent replication adds time and cost. It can also surface disagreements about cohort definitions that delay submission. We would still treat it as a feature, because it is better to find those disagreements before a reviewer does.
Mapping one package to ethics review, OPTN governance and FDA
The ethics board and the OPTN committee ask different questions of the same evidence. For an ethics board, we would put three questions first: was the analysis fixed in advance, were subgroup harms looked for, and did the analysts have a stake. OPTN governance asks what a policy does to the organ allocation system. Allocation policy proposals must show how they meet the OPTN Final Rule. They must address access and equity, including ethnic disparities on the waiting list and paediatric access. The data analysis should include, among other items, effects on programmes with different transplant volumes, risk-adjusted post-transplant patient and graft survival, and risk-adjusted waiting time and transplant rates. Proposals must also name performance indicators, baseline data for current policy, projected benefit, and a plan for periodic review.
That last requirement turns validation into the first step of monitoring. The KDPI proposal committed to compare metrics before and after implementation, including transplant rates and kidney non-use by KDPI sequence, race and ethnicity, and HCV status. It scheduled reviews at about six months, one year and two years. A matching-model protocol that reports subgroup performance at baseline already holds the reference values that review will need. Live per-site monitoring after deployment is a separate engineering problem, and the protocol only needs to point to it.
For an FDA route, the discussion paper on AI/ML-based software as a medical device expects changes to follow pre-specified performance objectives and defined validation processes, with real-world performance monitoring. It ties the rigour of performance evaluation to the quality and applicability of the test dataset. The same locked protocol helps here. The intended-use and risk framing, though, is organised around the significance of the information to the healthcare decision and the severity of the condition. That is a different structure from OPTN policy analysis, so the package needs rewriting for that audience.
For appraisal, PROBAST+AI (BMJ, 2025) gives reviewers 18 signalling questions for model evaluation across four domains: participants and data sources, predictors, outcome, and analysis. Writing the protocol against those questions in advance is a cheap way to anticipate a risk-of-bias critique.
Audit-ready validation documentation of this kind is itself an engineering output. It is not a policy memo written afterwards. TechnoLynx builds those artefacts as engineering work within its AI engineering for life sciences and healthcare, not as policy advice or certification.
Cox, competing risks, or both, for graft and patient survival endpoints?
We would make Cox proportional hazards the primary endpoint model, because SRTR’s own expected-survival models use it. SRTR fits Cox models to national data for time to graft failure and time to death to derive expected survival. The original KDRI hazard ratios were also expressed against a reference donor. A competing-risks analysis belongs alongside it as a secondary analysis wherever death and graft loss compete. For its own transplant and death-rate models, SRTR notes that these are cause-specific hazard functions because the events compete.
The two competing-risks approaches, cause-specific Cox and Fine-Gray, answer different questions, so the choice depends on the question of primary interest. A kidney allograft study notes that failing to account for competing risks can bias the Kaplan-Meier cumulative incidence of the event upwards.
What follows is our judgement. We would test the assumption within each centre and in the pooled model before accepting pooled hazard ratios, using the standard residual-based diagnostics. Where a covariate effect changes over follow-up, we would add a time-varying term or split the time axis. A violation confined to a few centres does not necessarily invalidate the pooled estimate if the analysis is stratified by centre. Apparent effect modification between centres may also reflect case mix rather than a real difference in how the model works.
Where a pre-registered study can still mislead
Pre-registration does not protect against a good analysis of the wrong outcome. The fairness appraisal notes that parity metrics do not capture downstream impact on patient outcomes. SRTR warns that its death and graft-failure counts may be under-counts because the data sources are not designed to capture every event. A matching model can pass every subgroup threshold on registry outcomes and still shift who receives which organs in ways the validation never measured. The KDPI committee recognised this when it judged changes in transplant rates by race to be an expected outcome of the policy. The question a protocol cannot settle is which of those shifts the transplant community accepts, which is why the OPTN review plan is part of the design rather than an afterthought.
