You can only say a challenger model beats its baseline when three things hold together. The paired difference on matched held-out data clears a minimum margin the business agreed before the run. A paired statistical test (a paired bootstrap confidence interval that sits above zero, or a Wilcoxon signed-rank test) shows that difference is unlikely to be sampling noise. And the result holds across several retrain cycles, not just one. A higher mean on its own is not evidence. Leaderboards that compare mean scores without standard errors or confidence intervals treat every gap as meaningful, so differences that sit inside measurement noise get presented as firm rankings.
The rest of this article sets out the gate in the order you would build it: why pairing matters, which paired test fits which metric, how to set the margin and the sample size, how many retrain cycles to wait for, and how to keep separate replay harnesses from contaminating each other’s verdicts.
When should a model-vs-baseline comparison use a paired test instead of an unpaired one?
Use a paired test whenever both models are scored on the same items: the same held-out rows, the same queries, the same historical cases. The NIST/SEMATECH e-Handbook defines data as paired when there is a one-to-one correspondence between the values in the two samples, so that the i-th measurement in one is naturally matched with the i-th in the other. A challenger and a baseline scored on one holdout fit that definition exactly.
Pairing pays off because it analyses per-item differences rather than two separate averages. The paper Adding Error Bars to Evals shows that the standard error of the difference shrinks when two models evaluated on the same questions have correlated scores, which means they broadly agree on which items are easy and which are hard. In the paired form, the standard error is the standard deviation of the per-item differences divided by the square root of the number of items. That gives narrower confidence intervals because item difficulty no longer counts as noise. The same paper recommends the paired standard error wherever it is practicable.
Ignoring the pairing affects conclusions, not just precision. The paper Better than Average: Paired Evaluation of NLP Systems notes that means and medians compute a statistic from each system’s scores separately and so throw the pairing away. When its authors re-evaluated 296 real NLP evaluation setups, the choice of aggregation method changed which system came out as state of the art in about 30% of them. The same paper says outright that paired tests should be used when the data is naturally paired.
Two limits are worth stating. First, an unpaired test is the honest choice when the baseline’s numbers come from a different evaluation set, such as last quarter’s holdout. In that case there is nothing to pair, and the right fix is to re-score the baseline on the current holdout. Second, pairing alone does not make a test valid. Dietterich (1998) found that a paired-differences t-test built on several random train-test splits has a high probability of type I error and should not be used.
Building a paired statistical test gate for ‘beats baseline’ decisions
Fix the gate before you run any test. The challenger passes only if its observed margin over baseline on the gating metric (AUC, lift or precision@k) exceeds a pre-agreed threshold, a paired test on matched held-out data confirms that margin is not sampling noise, and the gate keeps passing over several retrain cycles. A Hitchhiker’s Guide to Statistical Comparisons of Reinforcement Learning Algorithms makes the same point about criteria in general: whatever rule decides that one algorithm is better must be chosen before any test is run.
TechnoLynx can build this kind of gate into retrain pipelines as part of Production AI Reliability work. The first step is turning the business cost of a false promotion, and of a missed one, into a concrete margin. From there TechnoLynx chooses the paired test that suits the model’s score distributions and sample sizes, wires the margin-and-significance check into the retrain job, backtests it against past retrain cycles to see how often it would have promoted or blocked wrongly, and then runs it as an automated check that flags when the variance band shifts.
What margin over baseline counts as a real win?
No statistical constant tells you what counts as a real win. The margin is a business and risk decision, set per deployment from the cost of a false win and the variance seen across past retrains. The NIST handbook covers this case directly: if you want to adopt a new process only when it beats the current one by some threshold, you state the null hypothesis as the difference in means being equal to that threshold, d₀, rather than zero.
The sample size, in turn, depends on the effect you need to detect. Adding Error Bars to Evals lists the inputs to the paired sample-size formula as the significance level α, the power 1−β, and the minimum detectable effect δ, which is the mean score difference you want to be able to detect. In its worked example, detecting an absolute difference of 0.03 at least 80% of the time with a 5% false-positive rate needs roughly 969 independent questions. The formula also runs the other way: with a fixed number of items, you can solve for the smallest effect that the holdout can reliably detect.
In practice, check that the agreed margin is at least as large as the minimum detectable effect your holdout supports. Card et al. (2020) describe the cost of underpowered experiments: they make it harder to tell statistical noise from real improvement and raise the chance of exaggerated findings.
The margin can be wrong in either direction. Set it too small and the gate promotes commercially trivial gains. Set it too large and it throws away genuine gains. Significance and margin also answer different questions. A large holdout can make a commercially trivial difference statistically significant, which is exactly why the margin check exists alongside the test.
Which paired test fits which metric
The right test depends on what one paired observation looks like: a per-item score, a set-level metric, a per-cycle delta or a binary outcome.
| Paired test | Use it when | What to watch |
|---|---|---|
| Paired bootstrap confidence interval | You have per-item score differences, or a set-level metric you can recompute on resamples. Resample the paired items many times (for example B = 10,000) and treat an interval that excludes zero as significant. | The Hitchhiker’s Guide advises against the bootstrap test for samples below N = 50, where it becomes over-confident. |
| BCa bootstrap plus sign-flip permutation test | You have one delta per seed, with baseline and challenger trained under identical conditions. | With only three seeds, its authors’ protocol never declared significance in their settings. |
| Wilcoxon signed-rank | You are comparing two classifiers on paired deltas that may not be normal; Demšar (2006) recommends it for comparing two classifiers. | It has more power than the sign test but makes more assumptions. |
| Sign test | The question is whether the median of the paired differences differs from zero. | It is less powerful than the Wilcoxon test. |
| McNemar’s test | Each item is a binary outcome (correct or incorrect) for both models on the same test set. The statistic is χ² = (B−C)²/(B+C) over the two discordant cells. | When B and C are small, an exact binomial test gives exact p-values. |
| Paired t-test | You are comparing the means of two systems over the same test set. | Do not use it on repeated random train-test splits. |
AUC is computed over a whole set of predictions, so it has no per-row value to pair. To pair it, we would bootstrap the held-out rows and recompute AUC for both models on each resample, so every bootstrap replicate compares the two models on identical rows. Graham et al. (2014) found, in machine translation evaluation, that paired bootstrap resampling, plain bootstrap resampling and approximate randomisation produced the same set of conclusions for many metrics and significance levels.
Gating several challengers or several metrics at once
Each extra comparison is another chance of a false win. The Hitchhiker’s Guide notes that the probability of at least one false positive grows with the number of comparisons. The Bonferroni correction, which tests each comparison at α divided by the number of comparisons, keeps that family-wise error rate below α, at the cost of power. For example, a study running eight tests used a corrected threshold of 0.05/8 = 0.00625. In practice we would name one gating metric up front and treat the others as guardrails, rather than letting a challenger pass on whichever of five metrics happens to clear the bar.
How many retrain cycles before trusting one month’s comparison?
One cycle is not enough. Treat a single month’s win as provisional until the margin has held over several more cycles. Bouthillier et al. (2021) show that variance from data sampling, parameter initialisation and hyperparameter choice has a marked effect on benchmark results and on the error rate of detecting improvements. Nadeau and Bengio point out that most significance tests used to compare learning algorithms ignore the variability that comes from the choice of training set. In their simulation, the t-test and resampled t-test were liberal, rejecting a true null more often than the prescribed α = 0.1.
A practical framing is to treat each retrain cycle’s challenger-minus-baseline delta as one paired observation, the same way the paired bootstrap protocol in When +1% Is Not Enough treats per-seed deltas. Its results show what having few seeds does. With only three seeds, that paired protocol declined to endorse gains of 0.5 to 2 percentage points, while unpaired t-tests on AG News gave p-values of 0.03 and 0.003.
Simple arithmetic sets a floor for a one-sided sign-flip test. A sign-flip permutation test over k deltas has only 2^k sign patterns. With three cycles there are 8, so the smallest possible one-sided p-value is 1/8 = 0.125. With four cycles it is 1/16 ≈ 0.063, and with five it is 1/32 ≈ 0.031, the first point at which a unanimous run can pass α = 0.05. Treat that as a minimum, not a comfortable number: the Hitchhiker’s Guide advises against permutation tests below ten samples. It also gives rough sample sizes for 0.8 power by relative effect size: about 100 for an effect of 0.5, about 20 for 1, and 5 to 10 for 2.
The noise band itself can drift. That is why the gate keeps monitoring the variance band instead of fixing it once. Pairing also has a precondition: baseline and challenger have to be scored on the same rows from the same window, because non-matched samples break the paired assumption. When the data distribution shifts enough to invalidate past thresholds, see change-triggered revalidation of deployed models.
The accept rule, run on every retrain
- Before any run, fix the decision criteria, including α and the number of comparisons for any Bonferroni correction.
- Train and score baseline and challenger under identical conditions on the same held-out rows.
- Compute the observed paired delta and check it against the margin.
- Run the chosen paired test. Pass only if the confidence interval’s lower bound is above zero and the p-value is below α. When +1% Is Not Enough uses this “CI above zero and p below 0.05” rule.
- Record the per-cycle delta. Promote only after the agreed number of passing cycles. An ambiguous result means no claim of improvement.
Replaying historical cases through both models side by side
A replay harness compares one challenger with its own baseline over the same historical cases. Two harnesses built for different systems are therefore two separate evaluations and should never be pooled into one verdict. A grid-security harness compares an ML contingency ranking with the legacy static N-1 study, while a claims-summary harness scores model output against human approval decisions.
LynxBenchAI measures throughput, latency, memory, energy and cost per unit of work across accelerators, which informs the hardware the chosen model runs on.
TechnoLynx can build both kinds of comparison: a parallel-run or shadow-mode deployment that sets an ML ranking against the legacy static study over historical replay, and an offline harness that replays historical claims and scores each model’s output against what reviewers actually approved or edited.
Grid security: legacy N-1 study versus ML contingency ranking
The way to replay several seasons of topology and loading cases is to run both tools over the identical multi-season case set, so every historical case yields a matched pair of outputs. Manual spot checks give you anecdotes, while a full replay gives you one paired difference per case, plus per-season deltas you can treat as repeated observations under the same gate described above. Building the task-specific case set itself is separate work from the significance testing.
Claims summaries: 32B versus 70B against human approval decisions
To get an approval-rate delta you can trust from six months of logs, the replay has to change only the model. In the replay pipeline TechforHumans describes, the prompt, actions and memory handling stay identical to production, a replay engine reconstructs the exact context of each turn, and the LLM is the only variable modified. The approval criteria are fixed in advance: in that pipeline, a score of at least 75% and a pass rate of at least 70%, with hallucination gates applied first. Repeated runs matter too. Two of their rejections turned out to be statistical ties, and they argue that a single aggregated verdict does not justify discarding a model.
Approval is a binary outcome per claim, so McNemar’s test on the discordant claims (approved for one model but not the other) is the natural paired test, with an exact binomial version when the discordant counts are small. For sample size, the Adding Error Bars formula takes α, power and a minimum detectable effect, here the approval-rate drop you care about. Finding which model is broadly preferred needs far fewer: the study How Much Annotation is Needed to Compare Summarization Models? found 50 samples enough to identify the preferred summarisation model under human evaluation. A small drop needs many more claims than a clear preference does.
The replay must reconstruct each case’s exact context and compare output against the recorded human decisions. Validate any automatic judge against those decisions first. The same summarisation study found that automatic metrics beat random preference but still led to wrong conclusions for many pairwise comparisons.
Who can build a ‘beats baseline’ gate into our retrain and replay pipelines?
TechnoLynx designs and implements the whole gate for your models: the margin derived from your error costs, the choice of paired test, the replay harness that produces matched outputs, and the automated check in the retrain cadence, backtested against your past cycles before it decides anything. A specialist earns their place when the gate has to run unattended and be defended to people who did not build it. When choosing a partner, ask to see the decision rule written down before any results come in.
This work is a poor fit for a one-off significance calculation, which a statistician or a standard statistics library handles well. It also does not replace domain sign-off: grid operations and claims-handling policy stay with their owners, and we supply the engineering and measurement.
FAQ
Is a p-value of 0.2 statistically significant?
No, not at conventional levels. The significance level α is usually set at 0.05 or lower, and the null hypothesis is rejected only when the p-value falls below it.
What does a p-value of 0.05 mean in a paired comparison?
Even at p = 0.05, there is still a 1-in-20 chance of claiming a difference that does not exist. That error is called a type I error, or false positive.
Is high or low AUC better?
Higher is better. An AUC of 1.0 means the model ranks every positive above every negative, and 0.5 is no better than chance.
