A candidate risk model can replace the incumbent if a paired test shows its Brier score is no worse than the incumbent’s by more than a fixed margin. The margin and the sample size are fixed at the design stage, and the testing approach cannot be switched after seeing results. Non-inferiority testing exists for this situation: it supports a model replacement when strict superiority cannot be shown or is not needed. The margin is justified from the incumbent’s own historical record under matched conditions. If any of these is chosen after the results are visible, the validation becomes a negotiation.
TechnoLynx builds this kind of validation for teams swapping a risk model. TechnoLynx frames the swap as a non-inferiority decision, estimates the incumbent’s Brier variance with within-match correlation taken into account, and implements the paired test (a one-sided paired test or a match-level bootstrap, with TOST where equivalence is needed) with its pre-registration protocol. TechnoLynx then runs the power analysis and tests the whole design by simulation on historical data before the real comparison happens. The first step is agreeing the metric and the resampling unit (see AI engineering for finance).
A model swap is back-testing run head to head
Back-testing compares actual outcomes with model forecasts over a period not used in development, and outcomes that consistently fall outside the organisation’s predetermined acceptability thresholds are a reason to adjust or redevelop the model. A model swap is a head-to-head version of the same idea. Two models are scored on the same out-of-sample outcomes, and the question is whether the new one gives up too much.
The running case here is a win-probability engine that issues a probability at every tick of a match. All the ticks in one match share a single outcome, so they are not independent observations. A credit engine that rescores the same obligor every month has the same structure, with the obligor in place of the match.
Non-inferiority or superiority: which question is the model swap asking?
A superiority test asks whether the candidate is better. A non-inferiority test asks whether it is not worse by more than a pre-specified margin, so its null hypothesis is that the candidate is inferior, not the usual null of no difference. A swap is often motivated by a candidate that is simpler, more interpretable or cheaper, advantages not captured by predictive performance. Non-inferiority is the honest framing in that case.
A lower Brier score is better. Write the difference as d = Brier(candidate) − Brier(incumbent), and let Δ be the margin. The hypotheses for an outcome where lower values are better are then H0: d ≥ Δ against H1: d < Δ. Superiority is the special case with the margin set to zero.
The decision rule works on a confidence interval. The convention is a one-sided type I error of 0.025, half of the two-sided 0.05 used for superiority. In practice that means building a two-sided 95% interval (equivalently a one-sided 97.5% interval) for the difference and requiring it to lie entirely on the acceptable side of the margin. FDA guidance applies the same rule: the upper bound of the 95% two-sided interval must fall below the margin, which corresponds to a one-sided alpha of 0.025. If the same interval also excludes zero, you can additionally conclude superiority.
The two framings are not interchangeable once data is in. EMA guidance shows that a non-inferiority conclusion is like a superiority test at a less stringent significance level, and it does not accept switching to that approach retrospectively after a result fails to reach conventional significance.
How do you choose a non-inferiority margin for the Brier score difference?
The margin is the largest loss in Brier score you would accept as unimportant. Never justify it from the size of the backtest. EMA guidance requires the margin to be justified on both clinical and statistical grounds, and to be independent of power: “a small study is not a justification for a wider non-inferiority margin”.
FDA guidance separates M1, the entire effect of the active control over placebo, from M2, the largest acceptable loss, which is usually a fraction of M1. ICH E10 adds that the margin cannot be larger than the smallest effect the control would reliably be expected to have, and that it should be chosen conservatively.
Adapted to a probability model, this logic treats a naive reference forecast (the historical base rate) as the “placebo”. The incumbent’s skill is then the Brier improvement it delivers over that reference, measured period by period on historical data. In this adaptation, M1 is a conservative lower bound on that skill, and M2 is the part of it the business is prepared to give up. Without that anchor, a margin can quietly allow a candidate that is barely better than the base rate.
The incumbent’s historical variance gives the second constraint. Score the incumbent separately for each historical period (season, quarter, regime) under matched conditions. The spread of those per-period scores shows how big a difference falls within its normal fluctuation. One machine-learning precedent pre-specified an AUROC margin of 0.005 on three grounds: it was below any plausibly meaningful difference, roughly one-tenth of the effect that motivated the study, and of the order of the between-seed standard deviation. Another set a margin of 0.036 as the maximum acceptable loss in macro-F1.
“Matched conditions” carries real weight here. ICH E10 requires a planned non-inferiority trial to share the critical design characteristics of the historical trials behind the margin, and asks that any unavoidable differences be considered explicitly when choosing the margin. For a risk engine, we read that as the same markets or competitions, tick frequency, data feed and outcome definition. If the composition of the backtest differs from the history, FDA guidance expects M1 to be revisited for those differences.
Two variances are easy to confuse. The between-period variability of the incumbent’s Brier score helps justify the margin. The estimated variance of the difference between the models affects the sample size. Using the second to set the margin brings back the dependence on power that EMA rules out.
Paired TOST or match-level bootstrap when ticks within a match are correlated?
Use a paired analysis whose independent unit is the match. The choice between a paired t-test on per-match differences and a match-level bootstrap matters less than getting that unit right. Pairing is required. The Brier-UDAM framework requires both rules’ predictions to be generated for the same evaluation subjects, preferably out of sample. Hoessly (2025) shows that Brier scores depend heavily on the true outcome probabilities, so comparisons across different datasets can mislead, while comparisons on the same data are meaningful.
Treating correlated observations as independent underestimates the variability. In a non-inferiority test on clustered data, Zhang et al. report a cluster-robust standard error of 0.013 against 0.004 from the naive approach that treated every observation as independent. They conclude that ignoring within-cluster dependence can produce seriously misleading inference. The standard false-positive guarantee assumes independence too. The false-positive control of a Welch-based non-inferiority test is proven for independent, normally distributed samples. The Brier-decomposition variance estimators were tested on independently sampled forecast pairs.
Three implementations respect the match as the unit:
- Per-match paired t-test. Average each model’s tick-level squared errors within a match, take the per-match difference, and run a one-sided paired non-inferiority t-test on those differences.
- Cluster-robust variance. A sandwich variance estimator that allows unrestricted dependence within a cluster, assuming independence across clusters, gives asymptotically valid paired non-inferiority tests.
- Match-level bootstrap. Resample whole matches with replacement and recompute the mean Brier difference each time. One published comparison resampled 94 paired squared errors 10,000 times and built percentile intervals. Resampling ticks instead of matches breaks the correlation structure the bootstrap is meant to keep.
The number of matches decides which of these to trust. Cameron, Gelbach and Miller (2008) found that standard cluster-robust tests can over-reject with 5 to 30 clusters, and that cluster bootstrap-t procedures brought a 10% rejection rate back to the nominal 5%. Zhang et al. likewise recommend more conservative type I error control when only a few independent clusters are available.
TOST belongs in the design when the question is equivalence and not just “not worse”. An example is a like-for-like port, where a candidate that is meaningfully better is as suspicious as one that is worse. TOST tests both bounds of the margin, and equivalence is concluded when the confidence interval falls entirely inside (−Δ, +Δ). The AUROC study above reported one-sided non-inferiority and two-sided equivalence tests together. One more detail needs a decision in advance: whether matches of different lengths count equally, or whether long matches weigh more because they have more ticks. Either choice is defensible, but it has to be made beforehand.
How many historical matches does the backtest need?
The required number of matches depends on the size of the margin and the variance of the difference. FDA guidance states that both the size of the margin and the estimated variance of the treatment effect affect the sample size. The standard two-group formula is n = (z_α + z_β)² σ² (1 + 1/r) / (Δ + δ)², where Δ is the margin, δ the expected true difference and r the allocation ratio. In a paired design there is a single sample of per-match differences, so the allocation term drops out. With lower-is-better signs, n = (z_α + z_β)² σ_d² / (Δ − δ)², where n counts matches and σ_d is the standard deviation of per-match Brier differences.
σ_d is not known in advance. It can be estimated from pilot data, such as the development data used to build the models, provided that data represents the backtest population.
Worked example: how matches scale with margin and variance
Assume a one-sided alpha of 0.025 (z ≈ 1.96) and 80% power (z ≈ 0.84), so (z_α + z_β)² ≈ 7.84. With the margin written in units of σ_d, the normal approximation gives:
| Margin Δ | Expected true difference δ | Required matches (normal approximation) |
|---|---|---|
| σ_d | 0 | 7.84 → 8 |
| σ_d / 2 | 0 | 31.4 → 32 |
| σ_d / 4 | 0 | 125.4 → 126 |
| σ_d / 2 | σ_d / 4 (candidate slightly worse) | 125.4 → 126 |
Halving the margin roughly quadruples the matches needed, and so does doubling σ_d. MetricGate’s documentation makes the same point: a smaller margin requires substantially more participants for the same power. A candidate expected to be slightly worse costs as much as a halved margin, because only the gap between δ and Δ is left to detect. Conservative margins can therefore require very large samples, as ICH E10 notes.
The normal approximation is a starting point. For an exact figure with unknown variance, calculate power from the noncentral t distribution. Better still, check it by simulation. Zhang et al. evaluated empirical power by Monte Carlo at 0.8, 1.0 and 1.2 times the formula-based sample size.
If within-match correlation is underestimated, the effective sample size looks larger than it is and the real power falls below the planned power. If the historical archive cannot supply enough matches for an honest margin, widening the margin is not a fix, because that is exactly the power-driven margin EMA rules out. The real options are a longer out-of-sample window or accepting that the comparison may come back inconclusive.
What to lock before unblinding, and why a post-hoc margin is invalid
Pre-registration separates the confirmatory test from the exploration around it. The Center for Open Science describes its central aim as distinguishing confirmatory from exploratory analyses so that statistical inferences stay valid, because the same data cannot both generate and test a hypothesis. The regulatory position on margins is blunt. FDA guidance says a margin set only after a failed superiority result was known would not suffice to show non-inferiority. EMA also requires the margin’s justification to be written into the protocol.
| Item to lock | What the record states | Why it must be fixed first |
|---|---|---|
| Comparison data | Out-of-sample window, matches included, exclusion rules | Back-testing uses a period not used in development |
| Metric | Brier score on the defined outcome, tick or match weighting | Brier comparisons are only meaningful on the same data |
| Unit of analysis | Match (or entity) as the independent cluster | Ignoring within-cluster dependence invalidates inference |
| Margin and derivation | Δ, the reference forecast, the incumbent’s historical skill and variance | The margin must be justified in the protocol |
| Hypothesis and alpha | H0: d ≥ Δ, one-sided 0.025, or TOST if equivalence is needed | One-sided 0.025 is the convention |
| Test procedure | Paired t on match means, cluster-robust variance or match bootstrap (resample count) | Choosing after the results blurs confirmatory and exploratory analysis |
| Sample size | Number of matches, σ_d source, target power | The fixed margin sets the sample size at the design stage |
| Secondary tests | Order of tests, multiplicity handling | The testing order must be planned to control type I error |
| Covariate adjustments | Which adjustments and how | Post-hoc adjustments are not appropriate |
| Decision rule | Pass, fail, and what “inconclusive” triggers | A test that cannot reject inferiority may reflect too little data |
Writing the record into version control is the practical form of a registry. One model-comparison study fixed its full protocol (model roster, tuning budgets, operating rules, analysis plan) in version-controlled configuration files frozen before full-scale execution. Backtests on historical data have a specific weakness: the team may already know the data well. COS warns that intimate prior knowledge of existing data reduces how much bias pre-registration can prevent, and that disclosing what was known is the best available remedy. A reusable evaluation harness that reruns the locked protocol against each future candidate turns this from a one-off into a standing control.
The same discipline applies when the “candidate” is the incumbent ported to new hardware. There the question becomes numerical tolerance, which is covered in governing tolerance when validating GPU-accelerated risk engines against legacy benchmarks.
Where a non-inferiority gate stops protecting you
A passed gate says the data tested exclude the candidate’s Brier score being worse than the incumbent’s by Δ or more. It does not say the candidate is a good model. Four limits matter most.
Constancy. Clinical non-inferiority relies on the “constancy assumption”: the conditions that produced the historical effect must still hold in the new study, and FDA describes this as scientific uncertainty that is largely unquantifiable. For a risk engine, a change in market structure, data feed or rule set can make the historical variance behind the margin obsolete. The margin then needs a fresh derivation, not a reuse.
Bio-creep across successive swaps. After a series of new therapies that are each slightly worse than the last, an ineffective one can eventually be declared acceptable. Everson-Stewart and Emerson (2010) found in simulation that this was rare except when the constancy assumption was violated. Model lineages can drift the same way when each candidate is compared only with the latest incumbent. Anchoring M1 to the original reference forecast, not to the previous model’s score, closes that route.
Averages hide regressions. The Brier score is an aggregate measure. A difference shows that two rules differ in overall error, but not why. In one paired comparison, a small total Brier difference hid much larger, opposing dispersion and alignment contributions. A candidate can pass on the average and still get worse in late-game states or thin markets. A companion gate on per-bucket accuracy, because average metrics hide regressions, catches what the headline test cannot.
Brier is not calibration. Hoessly (2025) shows that a low Brier score does not necessarily mean good calibration, and recommends assessing calibration with additional measures such as calibration curves. Those measures belong in the pre-registered secondary tests, not in a post-hoc search.
There is also an analogue of assay sensitivity. ICH E10 notes that a trial unable to distinguish treatments that really differ provides no useful comparative information. A backtest window can be too quiet for the same reason. Before the real comparison, check that a deliberately degraded copy of the incumbent fails the gate on the same historical data. If it passes, the design cannot detect the losses it claims to rule out.
FAQ
What does 80% power mean in a study?
It is the probability that the planned test reaches the intended conclusion, here non-inferiority, if the true difference equals the value assumed in the sample-size calculation. With 80% power, one such comparison in five would still fail to show non-inferiority by chance.
How do you calculate a confidence interval for a paired t-test?
Take the mean of the paired differences and add and subtract the t critical value with n − 1 degrees of freedom multiplied by s/√n, where s is the standard deviation of the differences. For a non-inferiority decision at one-sided 0.025, use the 95% two-sided interval and require its bound on the inferior side to clear the margin.
