Average-case metrics hide regressions because a net accuracy delta is only what is left after per-item changes in opposite directions cancel out. When you swap, upgrade or port a model or backend, the new version can fix some cases and break others while the headline number barely moves. A 2026 audit of 1,707 paired model-by-task comparisons from public per-item evaluation data found that median per-item churn was about five times the median net accuracy change, and 145 of those comparisons reported an aggregate accuracy identical to their baseline. The answer is a reusable evaluation harness. It should replay held-out scenarios on every change and score each bucket (domain, condition, camera rig, subgroup) against its own explicit threshold, so a regression in one bucket blocks the release instead of disappearing into the mean.
A pass rate that went up while one case broke
The clearest small example comes from RegressionGate, a CI harness for tool-using LLM agents published on GitHub. An early test added the instruction “Always approve refunds” to the agent’s prompt, and the overall pass rate went up, because a few flaky cases happened to flip the right way. Underneath, one case went from passing 3 of 3 to failing 3 of 3: the agent refunded an order ID that does not exist. The author’s conclusion was that one aggregate number can hide a sharp break, and that a case going from all-pass to all-fail is a much stronger signal than the pass rate.
NVIDIA’s NeMo Evaluator documentation warns that a quantised checkpoint can keep its MMLU accuracy while getting worse at maths reasoning, and that a fine-tuned model can improve at code generation while regressing on instruction following. In the PoseTrack tracking benchmark, the best-performing submission ranged from nearly 80 MOTA on some test sequences to below zero on others, which is why the benchmark authors began reporting easy, moderate and hard splits alongside the full set. Simpson’s paradox is the general form: an aggregate gain can alter or reverse what holds inside each subgroup, so the way you segment the data decides which failures you can see at all.
TechnoLynx treats per-bucket gates as release-readiness engineering. Production AI reliability is a different discipline from model accuracy, and what it delivers is task-specific eval harnesses and regression suites that catch silent quality loss after a model update. The same discipline applies to any comparison of two similar models, whether checkpoint against checkpoint or a quantised build against its original, which is why it matters to AI infrastructure and production SaaS teams.
How do we split a held-out set into buckets so one regression isn’t averaged away?
Split the held-out set along the attributes you expect to change behaviour, report every metric both pooled and per bucket, and treat the per-bucket error breakdown as the deliverable, not the single headline score. Disaggregated evaluation means stratifying the evaluation data by subgroup and computing metrics and confidence intervals separately for each one.
TechnoLynx’s Computer Vision R&D engagements can start at this step: agreeing the held-out domain and the bucket scheme with the team before any candidate model is compared, so that the error breakdown answers the questions the deployment decision depends on.
A camera-trap detector that has to work at night shows what this looks like in practice. For cross-domain generalisation, the defensible design holds out the target domain (night-time infrared frames, say) as a test set that was never used for training or tuning, because a domain the model has already seen during tuning stops being a test of transfer. You then run the same detection and tracking metrics (mAP, precision and recall, MOTA or IDF1) separately on the day and night domains, and the gap between them is the generalisation gap you report. Within the night domain, slicing by species, time of night and distance to camera turns “accuracy dropped” into something an engineering team can act on. Species confusions, missed detections in darkness and degradation at range each suggest a different cause: a sensor-domain problem points towards data or preprocessing, while a model-capacity problem points towards architecture.
Choosing the attributes is where the judgement lies. The SAFE framework for AI weather models stratifies forecasts by territory, global subregion, income and landcover, and found that every model it benchmarked showed performance disparities across every one of those attributes. In one study, teams building natural-language-to-SQL assistants stratified test cases into easy, medium and hard bands, to check that simple cases stay reliable while hard ones surface specification gaps.
Two caveats belong in any per-bucket report. First, a confounder can sit behind a slice difference. If most night shots are also close to the camera, a “night” deficit and a “near-camera” deficit cannot be read independently. Research on disaggregated evaluation shows that performance differences between subgroups can follow directly from a confounder being distributed differently across them, and that weighted evaluation procedures can control for this. Second, rare species at night make tiny buckets, whose estimates are inaccurate because their variance scales as 1/n. That is a sample-size problem, and it gets its own treatment below.
Worst-case sets you replay on every change, instead of someone watching footage
You need a curated worst-case set that the harness replays automatically on every candidate, scored in a way that stays comparable across candidates.
TechnoLynx builds this kind of worst-case harness for a team’s specific workload. TechnoLynx starts by defining what counts as worst-case input, then designs an automated sampler and a severity score comparable across candidates, runs it at production-representative scale on the target hardware, and wires it into the release pipeline as a gate.
Fast-motion video: super-resolution and frame interpolation
Motion magnitude is the attribute to slice on. A 2024 frame-interpolation benchmark of 666 synthetic nonuplets at up to 4K resolution analyses interpolation error by motion magnitude because larger inter-frame motion is harder to interpolate. The FILM authors observed that state-of-the-art interpolation methods score well on standard benchmarks yet produce blurry frames in large disoccluded regions, and report that an earlier model trained on extreme motion was of limited effectiveness on small motion. A model trained for large motion did worse on small motion. Your replay set therefore needs both sports pans and slow talking-head material, not only the hardest clips.
Sampling can be automated. ITU-T P.910 (10/2023) states that test scenes should span the full range of spatial information (SI) and temporal information (TI) of interest, and SI/TI are widely used to check that video material covers an appropriate range of spatiotemporal complexity. Computing SI and TI for every clip in the archive and drawing from the extreme tails gives you a worst-case sampler that needs no one to watch hours of footage. For scoring, be careful with off-the-shelf metrics. The VFIPS work notes that existing interpolation methods mostly rely on off-the-shelf metrics, and its dedicated perceptual metric agreed more closely with human rankings on a large-motion 4K test set.
Pricing engines before the audit slot
The same logic applies to numerical systems: build a suite of oracle-lag paths and extreme price moves, and run it against the chosen discretisation scheme before you pay for a third-party audit. Treat the scenario list as your own engineering artefact, built for your scheme. Passing it is evidence about how the engineering behaves on those inputs. It does not mean the auditor will accept your test design, which only the audit firm can decide.
Classifier APIs under evasion attempts
For a hosted classifier, add an evasion regression gate that compares precision and recall under known bypass techniques against the same metrics on clean traffic, and block promotion when the adversarial bucket regresses even if the clean numbers improve. Run it on synthetic adversarial samples for repeatability, and add shadow traffic where you can.
To investigate a breach, look at the per-problem detail on that benchmark. NeMo Evaluator’s compare command lists per-problem flips: which items regressed, what each model answered and whether the regression is statistically significant. Scenario coverage matters too: in research on automated-driving testing, scenarios generated at different risk levels revealed different collision types, so covering one risk level alone misses failure modes.
A worst-case set is only as good as its coverage of real failure conditions. Content types it does not represent are the problem the production-sampling section below deals with.
Gating a regression suite by stadium, camera rig and condition
The stratified suite should treat each stadium × camera rig × condition cell as a stratum, with its own gating metrics, its own pass/fail threshold and a sample size calculated to detect the smallest regression you care about. A promotion passes only when every stratum clears its bar, not just the pooled average.
TechnoLynx can build that suite with your team. TechnoLynx agrees the stratification scheme and per-stratum thresholds, chooses metrics and a power-analysis method that suit your historical data, back-tests the gating against regressions you already know about, and integrates it into the CI/CD promotion gate with monitoring for under-sampled strata.
Different tracking tasks need different metrics. HOTA’s authors note that its breakdown into detection and association sub-metrics lets users select or tune algorithms for their use case. In their examples, sports and crowd analysis care most about identity-preserving trajectories (association recall and precision), driving assistance needs every pedestrian detected without phantom objects (detection recall and precision), and surveillance prioritises detection recall. For a sports-analytics product that means gating tracking on IDF1 or HOTA association scores per rig, alongside MOTA, and gating event detection on precision and recall for each event type, since a pooled event F1 lets a collapse in one rare event type hide behind common ones. Per-event metrics need a stable event taxonomy and consistent labelling across rigs.
What sample size per bucket is enough to trust a pass/fail decision?
The sample size per bucket comes from a power calculation: choose the minimum regression you need to detect, the acceptable false-pass and false-fail rates, and the metric’s variance in that bucket, then solve for n. Standard power tools compute any one of sample size, power or minimum detectable effect from the other two. Resolution-diagnostics research on paired LLM evaluation separates two uses of the same equation. A pre-specified target effect gives a prospective sample-size requirement, while the observed gap tells you whether your current set could have resolved it at all. When the authors tested their prescription, empirical power at the computed sample size came out between 0.796 and 0.827 against a 0.80 target.
Use a paired design. Baseline and candidate run on the same items, so variance comes from the observed churn rather than independent-binomial assumptions. Churn sets the bill. Two 4-bit quantisations of the same model scored 0.58 points apart on 1,000 GSM8K items, yet 9.1% of items flipped from correct to wrong and 8.5% from wrong to correct, and certifying equivalence at a ±2-point margin at that disagreement rate needs 2,730 items.
Three things inflate the requirement in a per-bucket harness:
- Many buckets mean many tests. With 40 pairwise comparisons, Bonferroni correction multiplies the required sample size by about 2.11.
- Clustered items behave like fewer items. Treating the 14 MMLU-Pro subject categories as clusters raised the number of unresolved adjacent-rank pairs from 4 of 9 to 6 of 9. Frames from one match, or clips from one camera, cluster the same way.
- Run-to-run noise. RegressionGate’s author measured two runs of identical code at 80% and 72%, concluded that a 5-point threshold would be a coin flip, and settled on 10 points, roughly what 25 cases can resolve given noise of about ±7 points.
Small cells are the honest weak point. A standard per-group estimate has variance that scales as 1/n, so it is accurate only when the group is large. In one 5,000-patient evaluation set, just 6 patients were female, over 80 and Hispanic, and 8 of the 32 intersectional subgroups had fewer than 10 observations. You can borrow strength across cells with structured regression, which gives calibrated intervals even for groups under 25, or set a conservative minimum and flag the cell for monitoring until data accrues. Too many fine-grained strata leave every cell underpowered, so it is better to group rigs that share hardware than to keep a cell you can never resolve.
Which bucket results should block a release and which should only flag it?
Block on buckets tied to real operational consequences, flag the rest, and make the rule explicit in a policy file rather than a reviewer’s judgement on the day. NVIDIA’s NeMo Evaluator gate takes a policy that declares which benchmarks are required, assigns each to a tier and sets thresholds, then evaluates every baseline-candidate pair on paired per-item data with a 95% confidence interval on the delta before aggregating a single verdict. It also separates two questions: a regression can be statistically significant yet within tolerance (acceptable), or exceed tolerance on too small a sample to be sure (insufficient evidence).
| Bucket class | Example rule | Outcome |
|---|---|---|
| Critical | NeMo Evaluator’s example policy: any regression above 1 percentage point on the critical benchmarks blocks the release | Block |
| Supporting | Same policy: a relaxed threshold of 1.5 points | |
| Advisory | Tracked for information, never blocks | Report only |
| Low-baseline bucket | Relative-drop guardrail, because at 80% accuracy a 1-point drop is a 1.25% relative change but at 10% it is a 10% change | Block on relative drop |
| Any case or bucket going from all-pass to all-fail across repeats | RegressionGate’s hard-regression rule fails the run whatever the overall number says | Block |
| Breach with too few items | Gate reports insufficient evidence; without per-item results it can report nothing else | Flag, enlarge bucket, rerun |
The specific thresholds above come from NeMo Evaluator’s example policy. Set your own per bucket, together with the people who own the consequences, and recalibrate them periodically, because thresholds that are too tight block good updates and thresholds that are too loose let real regressions through.
Catching content and conditions the buckets did not anticipate
A fixed bucket scheme only catches regressions in conditions someone thought to name. To catch the rest, define the taxonomy first, size samples per cluster, and then add continuous sampling from production that flags content the taxonomy does not cover.
LynxBenchAI publishes results showing how each device runs the public PyTorch stack your models are built on, so when the change under test includes new hardware, the first step is to check its published results for the devices you are considering.
TechnoLynx can set this up for a quality-critical transcoding pipeline. That means defining the content taxonomy and per-cluster metrics, sizing samples per cluster, checking the plan against full-catalogue spot checks, and deploying continuous production sampling with novelty detection. Where the change under test includes a move to different hardware, LynxBenchAI published results show how each device runs the public PyTorch stack your models are built on, so they give a broad indication for custom PyTorch models too. They do not measure your own workload; the first step is to check the published results for the devices you are considering.
For a video catalogue, the taxonomy might cover animation, archival, HDR sports and low-light news, each with its own tolerance rather than one catalogue-wide figure. SI/TI coverage adds a complexity check, in line with P.910’s requirement that test scenes span the full SI and TI range of interest.
Stanford’s Domino uses a cross-modal embedding and an error-aware mixture model to find regions of embedding space with a high concentration of errors, which surfaces underperforming slices nobody defined in advance. Ongoing post-deployment data audits with coverage analysis give early warning of drift and representation shift. When a discovered slice turns out to matter, promote it into a named bucket with its own threshold, so the next release is gated on it. One practical caution applies: someone has to own the escalation when a slice decays. Retraining triggers and threshold re-tuning belong to change-triggered revalidation, monitoring and rollback, not to the release gate.
Wiring the gate into CI, and what replayed scenarios can and cannot prove
Run the harness as a required CI check on every prompt, template, model-version or backend change, and record enough metadata to trace any regression to the change that caused it. RegressionGate’s workflow runs on every pull request to main. It validates the dataset, runs the evals with three repeats against a saved baseline, posts a report on the pull request, uploads traces and fails the check if the eval failed. Traceability comes from versioning: every run records the dataset hash, prompt version, agent model, judge model, rubric version, repeats and scoring version, and refuses to compare runs whose metadata does not match. The general practice of regression testing for production AI covers the pipeline side in more depth. What matters here is that the gate scores per bucket.
Replaying logged scenarios, such as ROS bags through an updated autonomy stack, is the right first-line check for every update because it is fast and repeatable. Research on automated-driving testing tracked scenario-level safety scores improving across Apollo versions, with a significant gain from Apollo 7.0 to 8.0. Replay has limits, though. A 2026 study of real-world perturbation testing found that offline measurements were at best moderately associated with vehicle-in-the-loop outcomes (Spearman correlation of 0.56 between offline MSE and failure rate for the camera model) and did not reliably identify the most harmful perturbations. Its conclusion was that dataset- and simulation-based testing is necessary but not sufficient.
Escalate to on-vehicle runs when the simulator’s fidelity for a scenario class is unproven, when a replay verdict is borderline or failed, or when the permit process asks for physical evidence. Whether simulated replay alone counts as evidence for a permit renewal depends on that regulator’s documented requirements. The engineering can produce clean, traceable evidence, but the regulator decides whether it is enough. Passing on historical bags also says nothing about scenario variations the logs never captured.
Who can build a per-bucket evaluation harness for our model upgrades?
TechnoLynx designs and builds per-bucket evaluation harnesses for teams swapping, upgrading or porting models and inference backends: the bucket scheme, per-bucket sample sizing, block-versus-flag thresholds, worst-case replay sets and the CI gate that runs them. When choosing a partner, ask how they size small buckets and how they would back-test the gate against regressions you already know about. In a fraud-detector audit, TechnoLynx introduced a more dependable method for measuring the accuracy of a model that had been missing rare outlier cases. That work was offline backtesting rather than a release gate, but the measurement discipline is the same. In a share-of-shelf analytics project, TechnoLynx delivered per-shelf measurement because rack-level aggregation was insufficient for actionable compliance data, and the per-bucket reasoning carries over. Vision work runs through Computer Vision R&D.
This work has clear limits. Building in-house works when you have statisticians and pipeline engineers with time to spare. A specialist pays off when the upgrade has a deadline and nobody owns evaluation design. A harness produces engineering evidence. It is not safety certification or regulatory sign-off, and it is not worth building for a proof of concept that has no production release to gate. Teams in regulated domains will still need the relevant notified body, auditor or safety assessor alongside any engineering partner.
The harness is a measuring instrument and can regress too
A per-bucket gate is only as trustworthy as its own configuration. During the 2026 compressed-model audit, a configuration defect in a pinned evaluation-harness version moved reported GSM8K accuracy from 0.232 to 0.566 without a single token of model output changing. The same study found that harness configuration choices moved per-item correctness by amounts comparable to quantisation itself. So the harness needs a regression check of its own: pin its version, rerun the unchanged baseline whenever the harness changes, and treat a baseline that moves as a harness bug until proven otherwise. The question the team still has to settle is which buckets carry real consequences in their deployment. No statistics can answer that for them, and every threshold in the gate depends on it.
FAQ
When would you prioritise precision over recall?
When false alarms are costly and misses are tolerable. The HOTA authors give driving assistance as a case needing both detection recall (to catch every pedestrian) and precision (to avoid unnecessary evasive action), while surveillance favours recall because extra detections are easy for a human to dismiss.
