A clinical computer-vision team runs an A/B test the way a growth team would: split the validation set between the current model and a candidate, read off which arm has higher accuracy, and prepare to ship the winner. That instinct is exactly what turns a promising model into a submission that stalls. Inside a Software as a Medical Device (SaMD) framework, each arm of that comparison is a regulated model version, and “the candidate scored higher” is not validation evidence unless the statistics that produced the number were designed as a regulatory artefact before any data was collected. The gap between those two worlds is not sophistication — plenty of ML teams know their p-values. It is sequencing. A consumer A/B test can afford to ask its statistical questions after the data lands. A clinical model comparison cannot, because the credibility of the result depends on decisions made before the first case is scored: what the primary endpoint is, how large an effect counts as clinically meaningful, how many cases you need, and which comparisons you are allowed to make. Get the sequence wrong and the result is not weak — it is inadmissible. What does A/B testing statistics actually mean for a medical CV model? Strip away the web-experiment framing and an A/B test is a hypothesis test: you are asking whether a measured difference between two model versions is large enough, and reliable enough, that it is unlikely to be an artefact of the particular cases you happened to evaluate. Everything that makes clinical A/B testing hard flows from taking that question seriously. Three quantities carry the argument, and each answers a different question: Significance (the p-value) answers “if the two models were truly equivalent, how often would I see a difference this large by chance?” A small p-value means the observed gap is hard to explain away as sampling noise. It does not measure how big the difference is, or how important it is clinically. Confidence intervals answer “given my data, what range of true differences is compatible with what I observed?” A 95% interval on the sensitivity difference between predicate and candidate tells a reviewer both the direction and the plausible magnitude — far more useful than a bare “p < 0.05”. Statistical power answers “if a clinically meaningful difference really exists, how likely is my study to detect it?” Power is decided before data collection, through the sample size. An underpowered study can miss a real improvement and produce an inconclusive result that no amount of post-hoc analysis can rescue. The reason these matter more here than in a growth experiment is the endpoint. When the metric is a detection score, the same discipline that governs how [email protected] maps to FDA validation evidence applies to the comparison: you are not comparing “accuracy” in the abstract, you are comparing a pre-defined operating point on a pre-defined test population, and the statistical test is built around that operating point. Why the analysis plan has to come first The single decision that separates admissible evidence from a discarded experiment is whether the analysis plan was locked before the validation data was collected. This is the divergence point between the naive and the expert approach, and it is worth being precise about why it bites. If you collect the data first and then choose what to test, you have — often without intending to — searched a large space of possible analyses for the one that looks best. You can pick the metric where the candidate wins, the subgroup where the effect is strongest, the threshold that flatters the candidate’s sensitivity. Each of those choices, made after seeing the data, inflates the chance of declaring a difference that is not real. A reviewer cannot tell a genuinely pre-planned comparison from a lucky one reconstructed after the fact, so the regulatory posture is to treat post-hoc analyses as hypothesis-generating, not confirmatory. A pre-specified statistical analysis plan removes that ambiguity by committing, on paper and before data collection, to: The primary endpoint — the single comparison that decides the study (e.g. difference in sensitivity at a fixed specificity, or a non-inferiority margin against the predicate device). The effect size that counts as clinically meaningful — not a statistical threshold, a clinical one. The target power and resulting sample size, derived from that effect size. The population strata and how subgroup comparisons will be handled. The multiplicity correction for every secondary endpoint. In our experience with regulated-pathway work, teams that lock this plan before validation data collection avoid the most expensive failure in the whole process: an inconclusive primary endpoint that forces a 3–6 month data re-collection cycle (observed pattern across engagements, not a benchmarked rate). That delay compounds directly against the parent hub’s faster path to cleared-device status — the statistics are not paperwork bolted on at the end, they are the schedule. How do you calculate statistical power and required sample size? Power analysis runs the study design in reverse. Instead of collecting data and asking what it shows, you start from the smallest difference you would care to detect and ask how many cases you need to detect it reliably. Four inputs drive the calculation. Input What it is Who sets it Minimum detectable effect The smallest clinically meaningful difference (e.g. +3 points of sensitivity over the predicate) Clinical + regulatory, not ML Significance level (α) Tolerated false-positive rate, conventionally 0.05 Pre-specified in the plan Power (1 − β) Probability of detecting a true effect, conventionally 0.80–0.90 Pre-specified in the plan Outcome variability Spread of the metric across the target population Estimated from prior data or a pilot The interaction that trips teams up: sample size scales roughly with the inverse square of the effect size. Halving the difference you want to detect roughly quadruples the cases you need. A team that quietly hopes to prove a one-point sensitivity gain on a few hundred cases has, without doing the arithmetic, designed a study that cannot succeed. For a worked example: if a validation study targets a 3-point sensitivity improvement at 80% power and α = 0.05, and the metric’s standard deviation across the population is moderate, the required case count runs to the low thousands, not the low hundreds — the exact figure depends on the assumed variability, which is why a pilot estimate of that variability is worth its cost. Two further design choices belong here. First, whether the study is a superiority test (candidate is better than predicate) or a non-inferiority test (candidate is no worse by more than a set margin) — the two have different hypotheses and different sample-size formulas, and the margin in a non-inferiority test is itself a clinical judgement that must be defensible. Second, whether comparisons are paired — running both model versions on the same cases, which removes between-case variability and usually cuts the required sample size substantially. For a CV model comparison, pairing is often the natural design because both versions can score the identical image set, and the analogous reasoning for validating perception model changes with A/B statistics in AV computer vision carries over directly. How multiplicity and subgroups change the statistics Here is where clinical A/B testing diverges hardest from the consumer version. A regulated submission almost never rests on a single number. Reviewers want to know the candidate holds up across the population strata that matter — scanner vendor, patient demographic, disease severity, site. The moment you run more than one comparison, the statistics change, and ignoring that change is one of the most common ways a study becomes indefensible. Every additional comparison is another chance to observe a difference by luck. Run twenty independent tests at α = 0.05 and, under the null, you would expect roughly one to come up “significant” purely by chance. If you then present that one as evidence, you have manufactured a false positive. Multiplicity control is the machinery that prevents this — methods like Bonferroni (simple, conservative), Holm, or Benjamini–Hochberg (controlling the false discovery rate) adjust the thresholds so the overall error rate stays where you claimed it was. Subgroup analysis carries the same hazard with an extra edge: subgroups are smaller, so each is individually underpowered, and the temptation to highlight the flattering one is strong. The disciplined posture is to pre-specify which subgroups are confirmatory (part of the primary evidence, powered and corrected) and which are exploratory (reported honestly as hypothesis-generating). When a candidate model’s overall win hides a regression in one demographic, that is precisely the kind of heterogeneity a pre-specified subgroup plan is meant to surface — and the kind of failure that reasoning tools like causal trees for estimating treatment effects in medical-device CV validation can help characterise before it reaches a reviewer. A diagnostic checklist: is your model comparison submission-ready? Before you collect a single validation case, run the plan against this list. A “no” on any confirmatory item is a schedule risk, not a detail. Is there one pre-specified primary endpoint, and is it a clinical operating point rather than a generic accuracy number? Is the minimum detectable effect set by clinical judgement, not by whatever the model happens to achieve? Was the sample size derived from a power calculation, with the variability estimate documented? Is the design paired where the same images can score both versions? Is superiority vs non-inferiority chosen explicitly, with the margin justified? Are confirmatory subgroups pre-specified and separately powered? Is a multiplicity correction named for every secondary and subgroup comparison? Is the entire plan locked — timestamped, version-controlled — before data collection begins? The failures that make a study inconclusive at submission almost always trace back to a box left unchecked here: an endpoint chosen after the fact, a sample size set by budget rather than power, a subgroup win that survived no multiplicity correction. None of these are exotic statistical errors. They are sequencing errors, and they are avoidable. FAQ What does working with ab testing statistics involve in practice? An A/B test is a hypothesis test comparing two model versions: it asks whether the measured difference between them is large enough and reliable enough that it is unlikely to be sampling noise. In practice, that means fixing a primary endpoint, powering the study to detect a clinically meaningful difference, and locking the analysis plan before any validation data is collected — so the comparison produces evidence a reviewer can trust rather than a number you happened to like. What are significance, p-values, and confidence intervals, and how do they apply to comparing two CV model versions? The p-value tells you how often you would see a difference as large as the one observed if the two models were truly equivalent — it measures reliability, not importance. A confidence interval gives the range of true differences compatible with your data, showing both direction and plausible magnitude. For comparing CV model versions, a confidence interval on the sensitivity difference at a fixed operating point is far more informative to a reviewer than a bare “p < 0.05”. How do you calculate the statistical power and required sample size for a medical-device CV validation study? Power analysis runs the design in reverse: start from the smallest clinically meaningful effect, set your significance level and target power, estimate the metric’s variability, and solve for the number of cases. Sample size scales roughly with the inverse square of the effect size, so detecting a smaller difference requires far more cases. Paired designs — scoring both model versions on the same images — usually cut the required sample size substantially. Why does an A/B test on a clinical CV model need a pre-specified analysis plan rather than post-hoc metric picking? Choosing what to test after seeing the data lets you search a large space of possible analyses for the one that looks best, which inflates the chance of declaring a difference that is not real. A reviewer cannot distinguish a genuinely pre-planned comparison from a lucky one reconstructed afterward, so post-hoc analyses are treated as hypothesis-generating, not confirmatory. Locking the plan before data collection makes the primary result admissible. How do multiplicity and subgroup (population strata) comparisons change the statistics for FDA-facing validation evidence? Every additional comparison is another chance to observe a difference by luck, so running many tests without correction inflates the overall false-positive rate. Multiplicity methods like Bonferroni, Holm, or Benjamini–Hochberg adjust the thresholds to keep that rate where you claimed it was. Subgroups are smaller and individually underpowered, so the disciplined approach is to pre-specify which subgroups are confirmatory and which are exploratory. What common statistical mistakes make a CV model-comparison study inconclusive at submission, and how do you avoid them? The recurring failures are choosing the endpoint after the fact, setting sample size by budget rather than by a power calculation, and presenting an uncorrected subgroup win. These are sequencing errors, not exotic statistical ones. You avoid them by locking a timestamped analysis plan — one primary endpoint, a power-derived sample size, and named multiplicity corrections — before collecting a single validation case. The deeper point is that the statistics are not a hurdle placed at the end of model development. When the analysis plan is designed as a regulatory artefact from the start — the way a medical-device CV regulatory pathway engagement treats it — the A/B comparison becomes the thing that shortens the path to clearance rather than the thing that stalls it. The question worth asking before you split any traffic is not “which model won?” but “would this comparison still count as evidence if a reviewer saw exactly how I designed it?”
