Confidence score calibration in healthcare should be validated one site at a time before any results are pooled. Take a tumour classifier deployed across hospitals. Compute expected calibration error (ECE), Brier score and a reliability diagram on each hospital’s own outcomes. Report all twelve beside the network figure, and send any site that diverges for root-cause work before sign-off. A pooled number answers a different question: whether the network is calibrated on average. That is compatible with one or two hospitals being badly over- or under-confident.
Two separate problems get folded into that one pooled figure, and each needs its own fix. The first is site-level calibration drift, where the probabilities no longer match outcomes at a particular hospital. A simple post-hoc recalibration can often improve it. The second is heterogeneity in assays, staining protocols or scanners. That can bias what the model sees, and correcting the outputs does not always repair it.
Why pooled confidence score calibration hides a miscalibrated hospital
One measured case is an ICU mortality model developed on MIMIC-IV and validated externally on the eICU database. Internally the model was almost perfectly calibrated: calibration slope 0.982, intercept 0.001, ECE 0.010. Externally the slope barely moved (0.980). The intercept, however, fell to −0.678 and ECE rose fivefold to 0.053, a systematic overestimation of risk that the authors attribute to eICU’s lower event rate. AUROC-based validation could not have caught this, because the model still ranked patients correctly while its absolute probabilities had shifted. A post hoc intercept update, without retraining, cut ECE by 26% (from 0.053 to 0.039). Calibration intercepts also ranged from −0.400 to −0.783 across subgroups, an unevenness the authors flag for subgroup-specific recalibration.
A study of pneumonia-screening CNNs tested them across three hospital systems. Calibration slopes between training and test hospitals ranged from 0.047 to 10.4. A model trained at one system was poorly calibrated at another. A CNN trained for that task identified the hospital system of a radiograph with 99.95% and 99.98% accuracy on two of the systems. Hospital system was also a larger source of variation in the image data than pneumonia itself.
The masking comes from the averaging itself. Riley and colleagues (BMJ, 2016) report a 12-study example in which the summary expected/observed ratio was 1.02, almost perfect agreement, yet 97% of the variation between study estimates came from between-study heterogeneity. Debray and colleagues found a summary observed:expected ratio of 1.11 for EuroSCORE II with an approximate 95% prediction interval of 0.43 to 2.85. Predicted risks were systematically too high or too low in particular populations. The same thing happens across classes within one dataset. A 2026 analysis of multimodal clinical condition classifiers found aggregate ECE “deceptively low” while it concealed severe miscalibration on individual conditions.
A pooled calibration figure is an average, and an average can look excellent while one hospital’s probabilities are consistently wrong in the same direction.
How do we break ECE, Brier score and reliability diagrams out by hospital site?
Compute every calibration metric on each site’s own predictions and outcomes, using exactly the procedure the pooled analysis uses. Then show the per-site results next to the pooled one, not in an appendix. Riley and colleagues (BMJ, 2016) recommend summarising performance per cluster, for example in a forest plot. They also recommend quantifying between-cluster heterogeneity with a random-effects meta-analysis and deriving 95% prediction intervals for calibration in a new cluster. They also call for calibration plots for the whole dataset and for each cluster separately.
Per-site calibration procedure for a 12-site classifier
- Freeze the model and its operating threshold, and define the outcome label the same way at every site before computing anything.
- Split the held-out predictions by site, so each site’s metrics use only that site’s cases and that site’s ground truth.
-
For each site, compute ECE as the bin-weighted average of the gap between accuracy and confidence across M bins: ECE = Σ ( B_m / n) · acc(B_m) − conf(B_m) . - Recompute ECE with several bin counts and with adaptive (equal-frequency) bins, to check the result is not a binning artefact.
- For each site, compute the Brier score: the mean squared difference between predicted probability and observed outcome, which needs no binning.
- For each site, draw a reliability diagram of observed event rate against predicted probability, where departures from the diagonal indicate miscalibration. Prefer a smoothed calibration curve with a confidence interval over a ten-bin plot, keep the axes square and undistorted, and show the distribution of predictions underneath.
- Add calibration-in-the-large (ideal value 0) and calibration slope (ideal value 1) for each site, with confidence intervals.
- Only then compute the pooled figures, and place a forest plot of per-site intercepts and slopes beside them.
- Flag for investigation any site that crosses a threshold agreed before the analysis, or whose interval excludes the pooled estimate.
Why ECE alone is too fragile for a per-site report
ECE depends on how you bin. Coarse bins can hide localised miscalibration, and finer bins raise the variance of the estimate, especially for low-prevalence outcomes. The 2023 “triptych” paper on evaluating probabilistic classifiers quotes the warning that binned calibration errors “vary significantly based on the choice of bins”, a choice that “enables the analyst to fudge results”. A guide to evaluating clinical prediction models makes the same point about grouped calibration plots. Grouping can be gamed by changing the number of groups, and it loses information in the high-risk range, which in their example is where miscalibration was most pronounced. The triptych authors recommend CORP reliability diagrams, which estimate the curve with isotonic regression instead of fixed bins, and propose the miscalibration component of the Brier score as the calibration metric.
No single summary number tells the whole story either. In that guide’s worked examples, one model had a calibration slope near 1 but an observed/expected ratio of 1.27. Another had a ratio of 1.01 but a slope of 0.72, because it overestimated high risks. Plain ECE is also unsigned, so it cannot tell an overconfident site from an underconfident one. A signed variant can, although over- and under-confident bins can partly cancel each other out. For a tumour classifier, where overconfidence on a wrong call is the costly failure, we would report the direction of miscalibration explicitly for each site.
Presenting twelve curves without burying the one that matters
A model risk report does not need twelve full-page plots up front. We would lead with a forest plot of per-site intercepts and slopes against the pooled estimate and its prediction interval, so an outlying hospital stands out at a glance. Behind it goes a grid of reliability diagrams on identical axes. Each flagged site then gets its own page with its curve, its confidence band and the status of its investigation.
Small sites and wide intervals
Sites with few cases give less certain per-site estimates. Van Calster and colleagues (2016) conclude that flexible calibration assessment in small validation datasets is problematic. A 2021 scoping review of healthcare AI guidelines notes that subgroup analysis may be limited by small sample sizes. The 2026 Nature Biomedical Engineering analysis of whole-slide models recommends per-stratum power calculations rather than aggregate targets alone. A 2024 drift-monitoring study makes the matching operational adjustment: sites with fewer scans may need a longer detection window to reduce uncertainty. A wide interval at a small hospital is a reason to collect more outcomes before judging it, not evidence that the site is fine.
Should EHR platform or lab assay vendor be a stratification variable?
Yes, if there is a documented reason to expect it changes the inputs or the labels. Without such a reason it adds comparisons and no information. The Nature Biomedical Engineering authors (2026) recommend prespecifying stratification factors such as grade, tumour mutational burden, site and key co-mutations, and running prospective subgroup analyses with subgroup confidence intervals. In their data, one model predicted colorectal MSI status with a cohort-level AUROC of 0.88, which fell to 0.72 within hypermutated and non-hypermutated subgroups.
EHR platform belongs on the list when coding practice differs. All six encoded vancomycin injectable solution with different sets of RxNorm codes, and only the training site’s encoding produced a signal the model recognised. Lab assay vendor belongs on the list for a related reason. Luijken and colleagues show in simulation that predictors measured under different protocols, or by tests from different manufacturers, can induce miscalibration to the point where a model is no longer clinically useful. For a histopathology classifier the equivalent axes are fixation method, stain formulation and scanner.
Put the axes in the analysis plan before the data comes in: site, scanner model, stain protocol, and any assay or EHR difference with a documented mechanism, each with its reason stated. Adding every available variable multiplies the number of comparisons, and at small sample sizes some strata will look miscalibrated by chance.
Site drift or assay bias: two problems with different fixes
When one site is miscalibrated, first decide which kind of problem it is, because that decides the fix. A 2025 review of degradation in medical AI states that each type of shift needs its own detection and correction strategy. Label shift often calls for output reweighting, and covariate shift for feature-level adjustment. The same review notes that calibration methods are lightweight and effective when the model’s ranking is still reliable but its confidence has drifted. They cannot address changes in decision boundaries or in the underlying concept.
Telling a calibration error apart from an accuracy error
Check discrimination and calibration separately at each site. If a site’s AUROC holds but its calibration intercept moves, the model still ranks cases correctly and only its probabilities have shifted. That is the eICU pattern, and a post hoc correction repaired it. If discrimination itself has dropped, recalibration alone cannot restore it. In the triptych authors’ worked example, one forecast’s miscalibration was remediable by recalibration, while another’s lack of discrimination was not.
Separating staining, scanner and population causes
A calibration curve tells you a site is off. It cannot tell you why. Howard and colleagues (Nature Communications, 2021) trace site-specific histology signatures to fixation artefacts, stain formulation and timing, scanner calibration and resolution, and genuine biological differences between the patients each centre treats. Their deep learning models predicted the submitting site with one-versus-rest AUROC between 0.964 and 0.998. Stain normalisation reduced this only modestly: site prediction stayed above 0.850 for every cancer type. Normalisation removed some perceptible variation but left second-order image characteristics unaffected.
In practice we would do three things. Split the flagged site’s calibration by scanner and stain batch where the site runs more than one. Compare input feature and output distributions against the reference sites. Check case mix (grade, stage, subtype) against the development cohort. The 2025 review gives a worked example of the same sequence: a subgroup drop, a shift in X-ray device type, weakened internal features, then correction. Population differences are the hardest to separate, because Howard and colleagues found that clinical and demographic variables differ significantly across submitting sites. If a site runs one scanner and one stain protocol across its whole population, those causes are confounded, and no analysis of that site alone will separate them.
During development, preserved-site cross-validation is a useful check. Of 56 predictable features in Howard and colleagues’ study, 51 (91.1%) lost AUROC when each site was confined to a single fold. The authors suggest the method can identify features unlikely to survive external validation before more resources are committed.
What the original validation assumed about standardisation
Every validation study rests on assumptions about standardisation: which assays, which stain protocol, which scanners, which reference ranges. These hold whether or not anyone wrote them down. A new site or vendor can break them without any visible sign, because the model still produces a confidence score. The practical first step is to write those assumptions out and check each new site against them.
A new lab assay vendor does not automatically need a full re-validation. One is warranted when a heterogeneity check shows the vendor’s data shifts model inputs or outputs beyond a tolerance set in advance. Inside that tolerance, the per-site calibration report is the evidence. The team has to set and justify the threshold for the specific model. Separating an assay effect from a genuine change in patient risk means comparing input and output distributions across vendors while controlling for known confounders. A one-time check will not catch gradual protocol drift, so the check belongs in ongoing monitoring and not only in the pre-submission package. Song and colleagues show how this kind of artefact surfaces. Haemoglobin was associated with AKI risk in opposite directions at two sites, and the authors’ candidate cause was systematically higher recorded haemoglobin among moderate-to-severe cases at one site.
Two neighbouring questions deserve their own treatment and are only flagged here. One is building these checks into a repeatable, automated pipeline that reruns as part of model governance. The other is re-validating accuracy when the classifier moves to a new sequencing or scanning platform.
When per-site recalibration is enough and when re-validation is needed
| What the per-site report shows | Evidence | Action |
|---|---|---|
| Discrimination stable, intercept shifted, slope near 1 | A post hoc intercept update cut ECE by 26% at a new institution; re-estimating a breast cancer risk model’s baseline hazard per country cut calibration heterogeneity (I²) from 98% to 35% | Per-site intercept update or baseline re-estimation, then re-check calibration |
| Subgroups within a site miscalibrated by different amounts | A single global intercept adjustment may not restore calibration equally across subpopulations | Subgroup-specific recalibration or monitoring before threshold-based use |
| Discrimination has dropped at the site | Recalibration fixes miscalibration but cannot restore discrimination | Investigate the cause; consider local refitting or model revision |
| Recalibrated site still overconfident on shifted data | Standard post-hoc calibrators produce highly overconfident predictions under domain shift | Tune the calibrator on a perturbed validation set |
Recalibration does a great deal when the problem really is calibration. Barda and colleagues (JAMIA, 2021) found two risk models with good overall calibration but poor calibration in a substantial share of subpopulations. Their recalibration algorithm reduced the variance of calibration-in-the-large across subpopulations by 98.8% and 94.3%. One multi-site AKI study frames the choice as a scale of effort: simple recalibration, refitting on local data, or full model revision with new predictors.
Does per-site temperature scaling count as retraining for validation purposes? Technically, temperature scaling leaves predicted classes unchanged, because dividing the logits by one scalar does not change which class scores highest. It does change the confidence scores your clinical claims rest on. We would settle whether that counts as a modification needing new validation evidence, under the change-control plan you committed to, before the first site is recalibrated.
Choosing a recalibration method for a ViT classifier on whole-slide patches
Temperature scaling is the simplest option: it has a single parameter, leaves predicted classes unchanged, and in Guo and colleagues’ study it outperformed all other methods on the vision tasks. Isotonic regression is more expressive, but unlike temperature scaling it can change the ranking of predictions and so accuracy.
TechnoLynx carries out this model calibration assessment (reliability diagrams and Expected Calibration Error per site and pooled) as part of its AI Governance and Trust work.
Guo and colleagues’ methods assume a hold-out set drawn from the same distribution as the test data. At a new hospital with a different data distribution, that assumption no longer holds. A 2021 CVPR study found that existing post-hoc calibrators still produce highly overconfident predictions under domain shift. The authors suggest that temperature scaling’s single parameter may lack the expressive power needed to stay calibrated across levels of drift. Our decision rule follows from that. If the sites look homogeneous and per-site calibration data is thin, fit one global temperature and still report per-site ECE as a check. If there is meaningful shift and enough local data, fit per-site temperature or isotonic scaling and test each fit on a different site or a later time window.
Pathologist disagreement adds a further layer, because ECE is only as good as the label it is measured against. We would report calibration against the majority-vote or consensus label as the primary result and against each individual rater as a sensitivity analysis. Disagreement then appears as stated label uncertainty and is not silently resolved. Methods also exist that calibrate a model’s estimate of how likely raters are to disagree, evaluated with reliability diagrams on medical images that carry several labels per case. Where the question is whether an outside party should check that a model’s probabilities can be trusted, our piece on independent calibration validation across clinical and maintenance models takes that up.
Calibration is a prerequisite for trusting a classifier’s probabilities, not the whole of validation. It sits inside a wider validation that also covers overall fit and discrimination.
Who can build per-site calibration validation for a multi-site tumour classifier?
TechnoLynx builds the calibration assessment for a multi-site classifier: reliability diagrams and Expected Calibration Error computed per site and pooled, delivered as evidence a reviewer or model-risk committee can read. TechnoLynx treats that evidence as an engineering output, not a policy document. Doing it in-house works when the team has a biostatistician comfortable with clustered validation data and time to build per-site tooling. A specialist earns their place when that capacity is missing or the report has to be defensible to outside reviewers. When choosing a partner, ask to see a per-site report with intervals and a stratification plan, not only a pooled number. The work falls under AI Governance and Trust, within TechnoLynx’s wider AI engineering for life sciences and healthcare.
TechnoLynx delivers this per-site and pooled calibration assessment through LynxBenchAI.
The scope has limits. TechnoLynx produces the evidence artefacts that audit and approval review require. That is not regulatory sign-off, certification, legal advice or a promise of audit-readiness. A team that first needs to choose its regulatory pathway is better served by a regulatory consultant. A team that needs a prospective clinical study run is better served by a clinical research organisation. A team with an experienced in-house biostatistics group and complete site metadata may need no outside help at all.
A clean retrospective report is not a deployment guarantee
Per-site calibration on retrospective data tells you how the model behaved on cases that already happened. It does not tell you how the twelfth hospital’s new scanner will behave next quarter. In one silent trial, a model with an AUROC of 0.90 on a retrospective dataset fell to 0.50 on real-time data. The team traced the drop to shifts in patient age, the laterality of obstructed kidneys and image format. After the fixes, the team checked performance by sex, side, ultrasound machine and postal code before calling the model safe. That is the same discipline this article argues for, applied over time and not just across sites. A per-site calibration report is worth most when it becomes the reference that every later check is compared against.
