Calibrating failure probability models means making predicted risk match observed event rates: among cases given a risk of R%, about R% should go on to fail. Van Calster and colleagues (2016) call this “moderate calibration” and recommend it as the target for development and validation. The choice of method follows the data. Platt scaling beats isotonic regression when the calibration set is small, and isotonic regression does as well or better from about 1,000 cases upward. Sometimes the right answer is no post-hoc calibration at all. What makes the output trustworthy, especially when failures are rare, is a documented chain of evidence rather than the number of events: how calibration was checked, whether the model holds steady across refits, how drift is caught, and how wide the uncertainty really is.

Platt scaling or isotonic regression for failure probabilities?

Pick by calibration-set size and by the shape of the distortion. In Niculescu-Mizil and Caruana’s 2005 study, Platt scaling outperformed isotonic regression with all nine learning methods when the calibration set held fewer than roughly 200 to 1,000 cases. With 1,000 or more points, isotonic regression always performed as well as or better than Platt scaling. The scikit-learn documentation gives the same rule of thumb: isotonic matches or beats sigmoid once there are more than about 1,000 samples, and below that it is more prone to overfitting.

The two methods assume different things. Platt scaling fits a sigmoid, so it works best when the distortion is sigmoid-shaped. Isotonic regression can correct any monotonic distortion, but it pays for that flexibility with overfitting when data is scarce. Boosted trees showed sigmoid-shaped reliability plots on seven of eight test problems, motivating a sigmoid, and the learners showing this shape are maximum-margin methods.

Situation	Lean towards	Why
Calibration set below roughly 1,000 cases	Platt scaling	Isotonic overfits small sets; Platt won with every learner in the 2005 study
Calibration set of 1,000+ cases	Isotonic regression	As good as or better than Platt at this size
Sigmoid-shaped distortion (boosted trees, SVMs)	Platt scaling	The sigmoid matches the distortion
Heavily imbalanced classes	Check the sigmoid fit before trusting it	Platt’s logistic model works best with symmetrical calibration error, a problem where imbalanced outputs have unequal variance
Base model already looks calibrated	Possibly none; measure first	Platt scaling can push probability mass away from 0 and 1 on well-calibrated learners

A 2025 preprint (Calibration Meets Reality) found isotonic regression ahead in 18 of 20 synthetic conditions. On five real datasets, though, the uncalibrated model beat both methods in 36% of classifier-dataset combinations. The same paper reports XGBoost as well calibrated out of the box on synthetic data but poorly calibrated on a real-world dataset (ECE 0.185). So the algorithm alone does not settle whether you need calibration. Measuring does, and the authors themselves recommend empirical evaluation before deploying any calibrator.

Two mechanical rules apply whichever method you pick. Fit the calibrator on data independent of the classifier’s training data, otherwise it maps to probabilities closer to 0 and 1 than it should. Scikit-learn’s CalibratedClassifierCV handles this by fitting each calibrator on a held-out cross-validation fold. Then measure the result rather than assuming it worked: Kumar, Liang and Ma (2019) found that Platt and temperature scaling are less calibrated than reported, and that common techniques cannot estimate how miscalibrated they are.

Checking calibration with a reliability curve and expected calibration error

A reliability diagram plots observed event frequency against predicted probability. A perfectly calibrated model traces the diagonal, and any deviation is miscalibration. Expected calibration error (ECE) condenses this into one number by binning predictions and taking the weighted mean gap between confidence and observed frequency. ECE weights each part of the score distribution according to its local density. Maximum calibration error (MCE) looks only at the worst case.

Report both numbers. The 2025 preprint shows they can disagree: after calibration, ECE improved while MCE got worse than the uncalibrated baseline. MCE is in turn sensitive to the binning scheme and less suited to small test sets.

The estimate needs as much care as the model:

• Bin count. The ECE estimator study tests a square-root-of-sample-size heuristic for the number of bins, alongside adaptive binning where each bin holds the same number of predictions.
• Sample size. In the 2021 ECE estimator study, every estimator’s error was around 10% of the true value in small-data regimes, and the authors advise against evaluating calibration on so little data at all.
• Uncertainty on the curve. Bootstrapping the holdout set and reading percentiles off the resampled reliability curves gives confidence bands instead of a single line. A single-point calibration assessment is not enough.
• Proper scores alongside. Brier score and log loss mix calibration with discrimination and data uncertainty, so a lower Brier score does not on its own mean better calibration.

How the calibration check is then validated independently, and how the approach carries across clinical and maintenance settings, is covered in our piece on calibration assessment across clinical and maintenance models.

Calibrating when failures are rare and the training data is heavily imbalanced

Imbalance biases models towards predicting “no event”. With fewer than about 5% events, plain logistic regression tends to underestimate the event probability (a result that goes back to King and Zeng, 2002). That shift in probability level is a calibration problem.

Undersampling the majority class produces base models whose predictions are severely biased. Platt scaling is often used to correct them, but it was not designed for that job. A study on this question recommends a check. If Platt scaling would have calibrated a model trained on the full data, it may be acceptable after undersampling. If not, they recommend a modified version that fits a logistic generalised additive model to the logit of the base predictions.

The validation split matters too. In a longitudinal rare-event study, leave-one-out cross-validation reported an AUC of 0.84 where a time-respecting validation gave 0.72, because leave-one-out used future information. For equipment or credit histories where events depend on the past, validate the way the model will be used: train on earlier data and score later data. In one rare-event prediction study, the best probability threshold was dataset-dependent, to be chosen on a validation set against the application’s objective.

How do you stop a more severe outcome being predicted as more likely than a milder one?

Use a model that treats severity as ordered. Cumulative link models such as the proportional odds model, the ordinal counterpart of binary logistic regression, project inputs onto one latent scale with ordered thresholds and predict probabilities of contiguous groups of categories. CORAL does the same in a neural network: it is provably rank-consistent, and its output-layer biases were confirmed to be ordered after training.

TechnoLynx can choose between a cumulative-link or CORAL-style loss and a dual-head design, implement and tune it on the target training hardware, check bucket monotonicity and PD-head calibration, and support its move into the existing scoring pipeline.

It helps to be precise about where inversions come from. A plain multi-class softmax ignores the ordering: mistaking a 30-day account for a 90-day one costs the same as mistaking a 60-day one.

Ordinal loss alone, or a dual head?

If the regulatory binary PD must stay as it is and inversions across severity are unacceptable, add an auxiliary ordinal head next to the calibrated binary PD head rather than replace the PD model with a multi-class one. This design has a cost. Weight the two losses deliberately and re-check the primary PD head’s calibration afterwards.

Treat the architecture’s guarantee as something to verify on the holdout set, not something to assume. Count inversions across buckets and re-run the calibration checks on the PD head after the ordinal head is added. Evidence from a related setting supports re-checking: in a 2025 benchmark of monotone-constrained gradient boosting across five credit PD datasets, feature-level monotonicity constraints cost little in AUC but more in Brier score, and the calibration effect could not be predicted from the change in discrimination. That study constrained input features rather than output buckets. The lesson it carries over is that an ordering constraint can move calibration, and only measurement tells you by how much.

Coefficient stability across refits: L1 vs L2 regularisation and the metric to report

Measure stability with a harness that repeats the refit: rolling-window or bootstrap resamples that imitate the scheduled refit cycle, run separately for the L1 and L2 models on the same features. Track four things across refits. Sign flips and coefficient variance show whether individual effects hold. Stability of the selected feature subset shows whether small data changes alter which predictors L1 chooses. PSI on the output scores shows whether the score distribution the business sees has moved.

One practitioner write-up states that L2 gives more stable solutions with correlated features because it spreads the penalty across coefficients, while L1 tends to pick one correlated feature over another arbitrarily. A 2025 study of stable variants of the Lasso found that the standard Lasso had the lowest median selection stability of the methods compared. The write-up itself notes that results vary with dataset and model, so confirm the pattern on your own feature set.

The stability measure proposed by Nogueira, Sechidis and Brown (2018) meets the properties a stability measure should have and, for the first time in the literature, supports confidence intervals and hypothesis tests on stability. Under the Fleiss-based reading those authors adopted, values above 0.75 indicate excellent agreement and values below 0.4 poor agreement. Expect challenge on the choice of measure, since a review of stable feature selection notes that there is still no consensus on the best one.

Which metric goes in front of the reviewers? Lead with score-level PSI and the selection-stability estimate with its confidence interval, because they show whether the score moved and whether the model still rests on the same features. Keep coefficient variance and sign flips as supporting diagnostics, since they explain why PSI moved. If sparsity matters for explainability and you use L1, filter correlated features beforehand (for example with WOE/IV screening) to reduce arbitrary swapping.

Three caveats keep the comparison honest. Stability results belong to one feature set and correlation structure, so re-test on each new feature vintage. Introducing elastic net or WOE binning at the same time as the L1/L2 switch confounds the comparison, so change one thing at a time. And the window length in a rolling or bootstrap scheme shapes how stable things look, so report it next to the result.

When should a failure-probability model be recalibrated?

Calibration drift beyond a pre-agreed threshold is the trigger to consider recalibration, adjustment or redevelopment. Discrimination alone will not reveal this. A nine-year study of acute kidney injury models found discrimination stayed stable while calibration degraded, driven by shifts in event rates and in predictor-outcome relationships. A dashboard that tracks only AUC will miss exactly this failure.

With labels, monitor calibration directly against true outcomes. Davis and colleagues (2020) maintain dynamic calibration curves updated with true outcomes and run an adaptive windowing (Adwin) test for increasing miscalibration. Their alert also points to a window of recent data suitable for the update. Drift detectors are set with two key parameters, a threshold and a window size. A 2025 review suggests setting the threshold so the false-alarm rate stays within an acceptable range (for example 1 to 5%), with rolling windows of hundreds of samples or one to three months of data common in practice.

Repeated checks carry their own risk. Any fixed-sample test applied repeatedly to an open-ended stream will eventually raise a false alarm, even on a stable model. Anytime-valid monitors address this. PITMonitor, from a 2026 preprint, bounds the probability of ever raising a false alarm, and it recorded a 3.8% false-positive rate against a nominal 0.05. It reacts to accumulated evidence, so it suits persistent drift better than brief excursions.

Without labels, score-level PSI gives an early warning of distribution shift. The common bands (under 0.1 stable, 0.1 to 0.25 moderate, above 0.25 large) are starting points, not universal law. Yurdakul and Naranjo point out that they are used without reference to Type I or Type II error rates. Tune them against the false-positive rate on your own history. Remember too that PSI detects distribution change, not a drop in performance, so a firing PSI calls for a labelled evaluation rather than an automatic retrain.

When drift is confirmed, recalibration is often enough. The same 2025 review cites clinical-model studies finding recalibration sufficient in most cases, with full retraining needed only for major shifts. Recalibration cannot fix changed decision boundaries or underlying concepts, though. Supervisors frame the same sequence in policy terms. The Federal Reserve’s SR 26-2 (April 2026) says persistent deviations outside established thresholds may warrant adjustment, recalibration or redevelopment. PRA SS1/23 expects firms to define maximum tolerances for performance exceptions and the controls that apply once triggers are breached.

Treating this as an engineering problem, not a dashboard, is what our production AI reliability work is about: drift detection, recalibration monitoring and release gates that catch calibration decay before the people relying on the numbers do. The pipeline side (alert scoring and dynamic thresholds) is covered in building drift detection, alerting and governance pipelines. What happens after the alarm (revalidation, rollback and change triggers) is covered in keeping deployed models trustworthy through revalidation and rollback.

How much validation does a failure-probability model need for its risk?

Enough to match the model’s use and materiality. SR 26-2 says the nature and rigour of validation generally align with the model’s approach, use and materiality, and that monitoring frequency and scope depend on model materiality. PRA SS1/23 ties the rigour, frequency and intensity of validation and monitoring to the model’s tier. It also expects model risk to rise with complexity and with uncertainty in inputs, including results that cannot easily be reproduced. Where development data does not represent the population the model will score, that limitation should be reflected in the tier.

In practice, a probability that triggers an unscheduled shutdown or sets capital needs the full chain described here. A score that only orders a weekly inspection list can live with lighter monitoring. The depth should be decided once, written down with its reasons, and revisited when the model’s use changes.

Defending a probability built on a handful of failures

An estimate built on few failures is defended by its method and its stated uncertainty, not by its event count. Document how the probability was calibrated, the interval around it, how it compares with whatever historical base rate exists, and the assumptions and data limitations behind it. When the calibration evidence is thin, add sensitivity analysis and recorded expert judgement instead of presenting a point estimate as certain. Whether the challenge comes from a model-risk reviewer or plant management questioning a risk ranking, the question is the same: what would have to be true for this number to be wrong?

Start by being honest about what sparse data can show. The NRC’s parameter estimation handbook for probabilistic risk assessment notes that with few events it is hard or impossible to see departures from the model, and the data cannot support a more complicated one. It also says failure and demand counts should be partitioned by system, plant or year to study whether the failure probability really is constant.

Which interval to put on a rare-event estimate

The textbook normal interval is unreliable when the expected number of failures is small: at an expected count of five, its coverage was 0.858 against a nominal 0.95. Bai and Lam recommend the exact (Clopper-Pearson) interval when the nominal confidence level must be guaranteed, and the Wilson interval otherwise, given its strong empirical performance. A separate 2021 study judged intervals on both coverage and margin of error relative to the size of the probability, and found Wilson gave the best overall performance. It suggests keeping the relative margin of error between 0.1 and 0.5.

For zero observed failures, the classical upper (1 − α) bound is approximately −ln(α)/n. In credit portfolios with very few defaults, Pluto and Tasche’s “most prudent estimation” principle uses upper confidence bounds while keeping the PD ordering consistent across rating grades. A Jeffreys test, using a beta posterior with parameters D + 1/2 and N − D + 1/2, assesses PD estimates per rating grade and at portfolio level.

Worked example: zero failures in 300 demands

Assume 300 comparable demands, no failures, and a 95% one-sided bound, so α = 0.05. The bound is −ln(0.05)/300 ≈ 3.0/300 ≈ 0.01. The defensible statement is therefore “the failure probability is below about 1% per demand at 95% confidence”, not “the failure probability is zero”. The same arithmetic shows the cost of sparse data. Because the bound scales with 1/n, halving it means doubling the demands, so the interval width belongs in the evidence pack next to the estimate.

Evidence pack checklist

• Calibration method, the calibration set’s size and independence, and why that method was chosen.
• Reliability diagram with bootstrap bands, ECE and MCE, with the binning stated.
• Interval method and its result (exact where guaranteed coverage is needed, Wilson otherwise).
• Comparison with historical base rates, with the pooling assumptions checked by subset.
• Sensitivity analysis showing the operating boundaries where performance stays acceptable.
• Any conservatism added, with its justification and documentation.
• Known limitations and data gaps, written for the people using the output.

Two caveats keep the pack credible. Small failure counts limit statistical confidence whatever method you use, so no interval should be presented as overcoming that. Overstating certainty is also the quickest way to lose the argument when someone challenges it: a model that reports “below 1% at 95% confidence, on 300 demands” survives questioning that “0% failure risk” does not.

None of these links in the chain works on its own.