Swap perception model B for model A, watch fleet-average mean-average-precision tick up 1.2%, ship it. That workflow is borrowed straight from web A/B testing, and in an autonomous-vehicle perception context it is the fastest way to ship a safety regression while your dashboard shows a win. The problem is not the statistics. The problem is what you measured and what you compared it against. A web A/B test can pool everything into one conversion number because every user click carries roughly the same weight. An AV perception model does not work that way. The metric that moves — aggregate detection rate across millions of frames of ordinary daylight highway driving — is almost never the metric that decides whether the change is safe. The frames that matter are the rare ones: a pedestrian stepping off a kerb at dusk, a cyclist half-occluded by a parked van, a stopped vehicle in heavy rain. Those live in the long tail, and a pooled average is designed to average them out. Why does a fleet-average metric show a win while a safety subsystem regresses? Consider a concrete illustrative case. Model B raises fleet-average mAP by 1.2% because it detects vehicles and lane markings slightly more consistently in clear conditions — the frames that dominate the dataset by volume. Buried in that same result, pedestrian-detection recall in low-light and adverse-weather frames drops by several points. Those frames are a small fraction of the total, so their regression barely dents the pooled number. But they are exactly where an AV spends its safety budget. The aggregate went up; the thing you actually care about went down. This is Simpson’s paradox with a safety price tag. When a strong effect on a large, low-stakes segment masks a harmful effect on a small, high-stakes segment, the pooled statistic is not just uninformative — it is actively misleading. Any team validating a perception change on a single fleet-wide accuracy number is measuring the wrong quantity with impressive statistical rigour. The fix is not more data or a tighter p-value. It is segmentation defined before data collection, tied to the operating conditions each subsystem is responsible for. A perception stack is not one model; detection, tracking, and multi-sensor fusion each own distinct failure surfaces, as we discuss in sensor fusion engineering. The A/B test has to respect that structure. Statistical significance versus a meaningful improvement Significance and importance are different questions, and conflating them is the second common failure. Statistical significance answers “is this difference likely to be noise?” A meaningful improvement answers “is this difference large enough to matter for the decision I have to make?” With a large enough validation set — and AV fleets generate enormous ones — a 0.1% recall change becomes statistically significant. That does not make it worth shipping, and it certainly does not offset a tail regression. Run the logic in the correct order. First define the effect size a safety review will accept: the minimum improvement in the segment that matters, and the maximum tolerable regression anywhere. Only then does significance testing have meaning, because you are asking whether the observed change clears a threshold you set in advance rather than fishing for any difference that happens to be detectable. This is where confidence intervals earn their keep over bare p-values. A 95% confidence interval on per-subsystem recall tells a reviewer both the direction and the plausible magnitude of the change. An interval of [-0.4%, +2.1%] on pedestrian recall is a very different shipping decision from [+1.5%, +2.0%], even if both are “significant” against a null. Report the interval per subsystem, not a single pooled point estimate. The reviewer needs to see where the uncertainty sits, not just whether a boundary was crossed. For the distinction between the two kinds of uncertainty that show up in these intervals, see our note on aleatoric versus epistemic uncertainty in production ML. Computing sample size and power before you run the test The most common shortcut is to run the model over whatever validation data happens to be available and read off the result. That inverts the correct sequence. Sample size is something you compute before the test, from three inputs you fix in advance: The minimum effect size that matters — for a safety subsystem, the smallest recall change a review will act on. The target statistical power — the probability of detecting that effect if it is real. A common target is 0.8, meaning you accept a 20% chance of missing a genuine change. The false-positive rate (α) — the probability of declaring a difference that is not real, commonly set at 0.05. The subtlety in AV perception is that the events you care about are rare. If pedestrian-in-adverse-weather frames are a small fraction of collected data, the sample size that gives you power 0.8 on fleet-average detection may give you power well below 0.5 on the tail segment — meaning you are very likely to miss a real regression exactly where it is most dangerous. Sample size has to be computed per segment, and the rare segments dictate how much data you actually need. This is often the point where a validation plan collapses: the team has millions of frames and still cannot make a confident statement about the segment that matters, because the tail is thin. Worked example: sizing a pedestrian-recall test Assume a review will act on a 2-percentage-point drop in pedestrian-detection recall in adverse-weather frames (the minimum effect size), a baseline recall around 85%, target power 0.8, and α = 0.05. A two-proportion power calculation puts the required number of adverse-weather pedestrian frames — not total frames — on the order of a few thousand per arm to resolve a 2-point difference. If those frames are, say, 0.5% of your capture, you need on the order of a million total frames per arm just to accumulate enough tail examples. (Illustrative figures — the exact count depends on baseline rate and variance in your data; treat this as the shape of the calculation, not a benchmark.) The lesson holds regardless of the exact numbers: the rare segment, not the fleet aggregate, sets the data budget. A per-subsystem acceptance-gate rubric The point of all this is to turn “the number went up” into an auditable decision. The table below is a decision rubric a reviewer can apply per subsystem before a model change ships. It is self-contained — each row is a gate the change must clear. Gate What to check Ship if Reject / re-test if Operating point fixed Confidence threshold / IoU set before data collection Same operating point for A and B Threshold tuned per-model to flatter B Segment defined a priori Weather, lighting, rare-event, and per-subsystem splits declared before the test All safety segments pre-registered Segments chosen after seeing results Power adequate per segment Sample size gives target power (e.g. 0.8) on each safety segment, not just fleet average Rare-segment power meets target Tail power below target — collect more tail data No tail regression Per-subsystem CI lower bound above the tolerable-regression floor CI excludes a meaningful regression everywhere Any safety segment CI dips below floor Effect exceeds threshold Observed improvement clears the pre-registered minimum effect size Point estimate and CI clear the bar Significant but below the meaningful threshold Result reproduces Re-run on held-out capture reproduces the direction Direction and rough magnitude hold Result flips or collapses on hold-out A change that clears every row is defensible. A change that fails any single row on a safety-critical subsystem should not ship on the strength of a fleet-average gain, however large or however significant. Segmenting across weather, lighting, and rare events Segmentation is only useful if the splits map to the conditions the subsystem is accountable for. In practice that means slicing the validation set along at least three axes: illumination (day, dusk, night, direct-sun glare), weather (clear, rain, fog, snow), and event rarity (routine versus near-miss or edge-case interactions). Each subsystem — object detection, multi-object tracking, fusion — gets its own set of intervals across these slices. The reason to fix the segments before the test is not bureaucratic. Post-hoc segmentation — slicing the data after you have seen the results until you find a split where B wins — is a well-documented route to false positives. It inflates the effective number of comparisons and quietly destroys your α. Pre-registration of the segments and the hypothesis is what keeps the false-positive rate honest. This is the same discipline that survives regulatory scrutiny in the clinical-imaging world; the parallel is worth reading in A/B testing statistics for clinical CV models, where the acceptance evidence has to hold up to FDA review. Test the deployed model, not the training checkpoint One more constraint separates AV perception from a textbook A/B test. The model that ships to the vehicle is not the checkpoint you trained. It has been quantised, compiled, and tuned for an on-device latency budget — decisions that can shift the operating point and change the precision-recall behaviour in ways that matter most in the tail. Significance testing that runs on the full-precision training checkpoint can certify a model that behaves differently once it is on the edge hardware. The statistical gate therefore has to run on the deployed artifact, under the same latency contract the vehicle enforces. The latency and quantisation trade-offs made at the edge are not a separate concern from validation — they change what you are validating. A test that ignores them is measuring a model that will never drive. Broader context on how the perception stack lives inside a latency contract sits in our computer vision practice overview. FAQ How does ab test statistics work? An A/B test compares two variants by measuring an outcome on each and asking whether the observed difference is larger than you would expect from noise. In practice it means fixing a hypothesis, an operating point, and a required sample size before collecting data, then reporting the difference with a confidence interval. For AV perception the outcome is per-subsystem detection or recall, not a single pooled accuracy number. What is the difference between statistical significance and a meaningful improvement when comparing two CV perception models? Statistical significance asks whether a difference is likely to be noise; a meaningful improvement asks whether it is large enough to matter for the decision. With a large validation set almost any difference becomes significant, so significance alone is a weak gate. You fix the minimum effect size a safety review will accept first, then test whether the observed change clears that pre-registered threshold. How do you compute required sample size and statistical power before running an A/B test on an AV perception model? You fix three inputs in advance: the minimum effect size that matters, the target power (commonly 0.8), and the false-positive rate (commonly 0.05), then compute the sample size that resolves that effect. In AV perception the rare safety segments dictate the count, because power on the fleet average can be high while power on the tail is far below target. Compute sample size per segment, and let the thin segments set the data budget. Why can a pooled fleet-average metric show a significant win while a safety-critical subsystem (e.g. pedestrian tracking) regresses? Because the frames that dominate the dataset by volume are ordinary conditions, and a gain there can outweigh a loss in the small, high-stakes tail — a Simpson’s-paradox effect. A 1.2% fleet-average mAP gain can coexist with a pedestrian-recall drop in adverse weather that the pooled number never surfaces. The aggregate is the wrong quantity because it averages out exactly the segment carrying the safety budget. How should A/B results be segmented across operating conditions like weather, lighting, and rare events? Slice the validation set along illumination, weather, and event-rarity axes, and report per-subsystem confidence intervals within each slice. The segments must be declared before the test — post-hoc slicing until a favourable split appears inflates the false-positive rate. Each subsystem gets its own intervals so a reviewer sees where the change helps and where it hurts. What false-positive and false-negative rates are acceptable when the outcome carries a safety budget, and how does that change the test design? When the outcome carries a safety budget, a false negative — missing a real regression — is the expensive error, so power on the safety segment matters more than a tight α on the aggregate. Teams commonly hold α at 0.05 but require power of 0.8 or higher specifically on the tail segments, which drives up the data needed there. The design shifts from one pooled test to per-segment tests sized around the rare, high-stakes conditions. How do you turn A/B test statistics into an auditable acceptance gate for shipping a CV model change? Pre-register the operating point, the segments, the minimum effect size, the target power, and α, then evaluate the deployed model against a fixed rubric of per-subsystem gates. A change ships only if it clears every gate — including no meaningful regression in any safety segment and reproducibility on held-out capture. Because every threshold was set in advance, the decision is auditable rather than eyeballed. The uncomfortable part is that most of the work happens before any data is collected: choosing the operating point, the segments, the effect size, and the power target. Get those right and the test almost runs itself. Get them wrong and no amount of significance testing will tell you that the pedestrian-tracking tail just regressed. That pre-registration discipline — one hypothesis, one operating point, per-subsystem intervals — is what we pressure-test when scoping an AV perception subsystem, because a validation gate that averages away the tail is not a gate at all.
