A model that scores well overall can still fail badly on particular subsets or operating conditions that nobody identified during training or testing. To see where it fails, evaluate by scenario bucket: define the operating conditions that matter (lighting, focus, sensor, region, transaction type), test several degrees of each shift rather than one, score on an annotated held-out set collected separately from training, and report metrics per slice and per degree alongside a measure of how far new data sits from the training distribution.
Why a strong aggregate score still hides failures
Researchers in medical imaging have a name for this: hidden stratification, where a model performs well overall but consistently fails on a meaningful subset that nobody identified during training or testing. Oakden-Rayner and colleagues found it can produce relative performance differences of over 20% on clinically important subsets, and that it tends to sit in subsets with low prevalence, low label quality, subtle distinguishing features or spurious correlates. Those are exactly the subsets an average underweights.
The Metrics Reloaded recommendations (Nature Methods, 2024) make the same point from the metric side: multi-class metrics can hide poor performance on individual classes, so per-class validation should be added. The same paper notes that models with nearly identical test-set accuracy can behave very differently on data from other distributions, and that disparities in minority groups get missed unless the aggregation is chosen with them in mind.
Public benchmarks show the same pattern. The ISIC 2018 skin lesion challenge reported that top segmentation algorithms still failed on over 10% of images on average, and that algorithms with equal test performance generalised differently. The WILDS benchmark, ten datasets with real-world shifts such as across hospitals, camera traps, and time and location in satellite imaging, found standard training gave substantially lower out-of-distribution than in-distribution performance, a gap that remained with existing shift-robust methods.
In a manufacturing defect study, a classifier performed almost perfectly on a validation set randomly split from the training pool, then failed on a holdout set of the same parts photographed on another day. Same parts, same preparation, different day: the random split did not reveal that shift.
Test several shift degrees, because robustness at one level predicts little
Most out-of-distribution benchmarks use diverse datasets yet keep the degree of shift between training and test domains largely fixed. A 2023 study of graded distribution shifts argues this two-point view oversimplifies generalisation, since several distinct degradation patterns can produce the same two numbers, and a model that wins at one severity can have much worse worst-case performance.
The measurements back that up. On a noisy MNIST variant with ten noise levels, the models that performed best at level D4 lost more than 10% average accuracy at D5, which is only slightly noisier. In the worst case, the relative drop reached 50.3% from D4 to D5 and 73.1% from D4 to D6. The authors conclude that even within one type of shift, a small increase in degree can severely harm performance, and that algorithms show very different generalisation patterns across shift degrees. Training on strongly shifted data did not reliably buy robustness to milder shifts either.
Large pre-trained models were not exempt: CLIP models adapted by linear probing were disproportionately sensitive to even mild shifts that were rare in pre-training. A foundation backbone is not a substitute for testing the conditions you will actually meet.
The practical consequence is that each degradation axis needs a graded set of bins, not a single “hard” set. Averaged robustness scores hide the curve: RobustBench, for instance, reports corruption robustness as accuracy averaged over corruption types and severities. Its corruptions try to mimic degradations that occur naturally. For image recognition services, IEEE 3129 provides test indicators for common corruption and adversarial attacks within an assessment framework for robustness under various settings.
How do we quantify segmentation accuracy as blur and lighting get worse?
Quantify segmentation robustness by computing mask-overlap metrics (Dice and IoU) on a held-out, expert-annotated set binned by focus blur and lighting level, and plot accuracy against each degradation level rather than reporting one average. For smartphone lesion images, this is the domain shift evaluation protocol that has to exist before any two pipelines can be compared fairly.
TechnoLynx builds this kind of protocol as engineering work: defining the blur and lighting axes and accuracy metrics that match the deployment photo distribution, deciding whether to anchor to a published dataset or a bespoke annotated set, implementing the evaluation harness and segmentation model on the target hardware, and measuring degradation curves that later serve as the production monitoring baseline. TechnoLynx’s part is the measurement, not the clinical judgement of what accuracy is acceptable for a diagnosis.
Why dermoscopic benchmarks are the wrong anchor
Most previous skin-lesion work focused on dermoscopic images captured in well-controlled clinical settings with specialised equipment, and those images include sub-surface features normal smartphone cameras cannot capture. Smartphone photos add their own distortions: illumination variation, motion blur and defocus. A segmenter that scores well on ISIC dermoscopy has therefore been tested on a different domain from the one patients submit.
Blur measurably degrades skin-image models. In a 2024 teledermatology study, diagnostic model sensitivity dropped by 0.15 on slightly blurred and 0.22 on strongly blurred skin images. That study measured diagnostic accuracy as sensitivity and precision of disease-category prediction, not mask accuracy.
Bins, metrics and aggregation
Sample the blur and lighting bins from real patient captures where you can, and use synthetic degradation only to fill bins the real data leaves thin. Compute Dice per image and then aggregate over images, because pixels within one image are highly correlated and pooling them across a dataset distorts the result. Pair Dice with a boundary metric such as Hausdorff distance, since overlap metrics are shape-unaware and struggle with small structures. Where one patient contributes several photos, aggregate per patient as well, because ignoring that hierarchy is a known metric-application flaw.
The held-out set and labelling protocol
Human ground truth for lesion masks is itself inconsistent. Across the ISIC Archive, median inter-annotator agreement on lesion masks is around 0.72 kappa, with a long tail of strongly disagreeing annotations. If two dermatologists disagree that much on a mask, a pipeline difference smaller than that disagreement is not a finding.
The IMA++ dataset (2025) shows what a workable protocol looks like: multiple annotations per image for 2,394 images, consensus masks computed with STAPLE and majority voting, per-image agreement measured as pairwise Dice and split into low, medium and high agreement strata that are balanced across partitions, and automated mask quality checks for empty, full-image and border-touching masks. Collect the held-out photos in separate sessions from the training photos, for the same reason the manufacturing holdout above failed.
Is there a consumer-camera benchmark to anchor to?
Partly. Google’s SCIN dataset (2024) contains 10,408 crowdsourced consumer images of skin conditions with dermatologist condition labels and estimated skin-tone labels, and its authors propose it for evaluating tools on real-world patient-sourced images. Its labels, as published, are dermatologist condition labels and estimated skin-tone labels, and it skews towards common inflammatory and infectious conditions, with neoplasms under 5%. It can anchor capture-condition variety and skin-tone coverage; it cannot replace a mask-annotated held-out set.
| Situation | What to anchor on | What to build |
|---|---|---|
| A published consumer-camera dataset closely matches your photo distribution and task | The public set, for comparability across pipelines | An in-domain held-out set for final validation |
| Public data matches capture conditions but lacks masks (the SCIN case) | The public set for condition and skin-tone coverage checks only | A dermatologist-annotated mask set, binned by blur and lighting, as the primary protocol |
| Only dermoscopic benchmarks exist | Nothing for robustness; treat them as a different domain | The annotated smartphone set first, before comparing any pipelines |
Can we measure domain shift automatically instead of guessing by eye?
Yes: domain shift between new deployment data and the training set can be detected with statistical two-sample tests on inputs or on the model’s learned representations. Eyeballing imagery from a new field or region tells you it looks different; it does not tell you whether the model cares.
TechnoLynx builds these cross-domain harnesses: working out which factors matter for the specific model (for crop imagery, soil colour, row spacing, sun angle and vegetation indices), choosing distance and out-of-distribution techniques that fit the architecture, instrumenting the pipeline to score shift per factor and overall, calibrating thresholds against labelled cross-region accuracy drops, and wiring the score into production so every new field or region emits a shift flag alongside its predictions.
Covariate shift, data drift and concept drift
Covariate shift is a change in the distribution of inputs between training and test while the relationship between inputs and labels stays the same. Concept drift is the other case, where that relationship itself changes, so the same input should now get a different answer. “Data drift” usually means input drift observed over time in production, and “model drift” the resulting decline in model quality. The distinction matters operationally: input-distribution tests can detect covariate shift without labels, but concept drift only shows up once you have fresh labels to score against.
Measuring it: two-sample tests on embeddings
In a 2019 empirical study of shift detection methods, Rabanser, Günnemann and Lipton found that two-sample testing on representations reduced by a pre-trained classifier performed best across the shifts they tested, and that domain-discriminating classifiers helped characterise a shift and judge whether it was harmful. In practice that means embedding a sample of training images and a sample of new-region images with the model’s own backbone, then testing whether the two samples come from the same distribution. Running the same test on single factors (soil tone histograms, row-spacing estimates, solar elevation from metadata) gives the per-factor breakdown.
Region-level analysis adds the “where”. A 2023 study decomposed the test input space by the ratio of test to training density and found performance degraded significantly as that ratio rose, with models accurate in regions dense with training data and erroneous in sparse ones. Binning new data by how far it sits from the training support is therefore a direct way to turn a shift score into scenario buckets.
Agricultural imagery is a hard case for distribution shift. A 2026 Embrapa review of 42 studies concluded that the bottleneck for crop-imaging models moving from plots to farms is robustness under distribution shift, and organised shift into four axes: scene, sensor, protocol and time. In cross-field weed detection, most zero-shot transfers collapsed to single-digit mAP@50, with exceptions only where source and target shared weed species or scale. With 25 labelled target images, fine-tuning on soybean reached 80.0 mAP@50, close to the 76.8 in-domain score, while off-the-shelf domain adaptation was inconsistent. A small labelled target sample is often worth more than a clever unsupervised fix.
What to report alongside each prediction
Report a per-input out-of-distribution score next to the prediction, because softmax classifiers are known to produce overconfident outputs even for inputs far from the training distribution. A per-input distance from the training data in the backbone’s embedding space can flag low-confidence regions, and a region-level two-sample test result is a reasonable batch-level flag. Treat both as risk indicators, and trust their thresholds only once checked against real labelled accuracy drops in comparable regions.
Can proxy ground truth replace field labels?
Use it as a secondary sanity check, not as validation. A Brazilian soybean yield-mapping study trained against municipality-scale yield statistics and reported an RMSE of 344 kg/ha (12% of the long-term mean) on withheld test data, with leave-one-year-out RMSE ranging from 259 to 816 kg/ha. Agreement with regional statistics of that kind shows the model is not wildly off in aggregate, but aggregated figures can hide per-field errors that cancel in both directions.
Building that harness so it keeps running after launch is release-readiness work, and we treat drift detection as part of production reliability for AI infrastructure and production SaaS. Our guide to building harnesses that actually detect drift and regressions covers the harness design itself.
Which fraud segments does the new model do worse on, even if the aggregate score looks fine?
The candidate-versus-incumbent fraud model question is answered by a segment-stratified, held-out comparison: score both models on the same evaluation set sliced by fraud type, transaction segment and adversarial cases, and compare per-segment metrics before any cutover. If the aggregate score is the only evidence, it is not enough to switch.
TechnoLynx builds the comparison harness that backs a cutover decision with per-segment evidence for candidate and incumbent, starting by agreeing the fraud segments, transaction types and adversarial cases with the team that owns the decision.
Stripe’s account of tuning its Radar fraud model shows why the overall number misleads: stopping training early improved overall ranking but reduced recall in its narrow false-positive range.
Pre-specify segments, then test the difference, not each segment
Fix segment definitions before looking at results. Clinical-trial methodology is blunt about this: subgroup analyses should as far as possible be restricted to those proposed before data collection, and any chosen later should be labelled as such. Segments chosen after the results are seen should be labelled as such and treated as exploratory, not as proof of a regression.
Then test whether the candidate’s advantage differs by segment, rather than running a separate significance test inside every segment. In simulations with no real subgroup effect, formal interaction tests flagged one 5% of the time, as expected, while subgroup-specific tests found effects in 7% to 66% of cases. A segment counts as a regression risk only when the candidate is meaningfully worse than the incumbent there, not merely different by noise.
Small segments and late labels
Small segments have little statistical power. The same report found that detecting an interaction as large as the overall effect needs about four times the sample size, rising to 100 times or more for interactions under 20% of the overall effect. For a rare but costly fraud type, the options are pooling more history, accepting a wider uncertainty band, or gating the segment with a shadow deployment rather than pretending the held-out set settled it. Fraud labels also arrive late, as chargebacks and fraud reports come in after the transaction, so the evaluation window must end early enough for its labels to have matured.
Public data can stress-test the harness before it touches production. The Bank Account Fraud suite provides privacy-preserving, realistic generated tabular datasets with temporal dynamics, significant class imbalance and specific types of data bias. Adversarial slices still cover only fraud patterns seen so far, and this whole comparison judges scoring quality, not latency, infrastructure cost or integration effects.
A per-bucket evaluation checklist before trusting a new model
Run through these before a cutover or a new-region launch:
- Buckets named in advance. Operating conditions and segments are written down before results are seen, with the reason each one matters.
- Several degrees per axis. Each degradation or shift axis has graded bins, not one “hard” set.
- Separately collected held-out data. Holdout comes from different sessions, sites or time windows than training, not a random split.
- Checked ground truth. Inter-annotator agreement is measured, and differences smaller than annotator disagreement are not reported as wins.
- Right aggregation. Metrics are computed per image or per case first, then per bucket, and paired with a complementary metric.
- Difference tests, not per-bucket tests. Candidate versus incumbent is tested for a differential effect, with small buckets flagged as underpowered.
- Shift score in production. Each prediction or batch carries an out-of-distribution or drift score whose thresholds were calibrated against real labelled drops.
When the per-bucket numbers and the headline disagree
The hard part of this method is not computing the slices but deciding what to do when the headline improves and one bucket gets worse. That call depends on what the bucket costs when it fails: a 2% regression in low-light smartphone photos of pigmented lesions is a different decision from a 2% regression on a low-value transaction type, even if the statistics look alike. Write the cost of failure per bucket into the acceptance criteria before the first comparison runs, and the slicing becomes a gate rather than a report someone reads after the release.
FAQ
What happens if a model performs well on training data but poorly on test data?
In one manufacturing study, a classifier near-perfect on a random validation split failed on identical parts photographed on another day, which its authors attributed to over-fitting to spurious features.
What are masks in image segmentation?
A mask is a per-pixel label marking which pixels belong to the object, such as a skin lesion, and which belong to the background. Segmentation accuracy is measured by comparing predicted masks with reference masks using overlap metrics like Dice and IoU.
