In a 2026 arXiv study of imbalanced classification, the first-stage variant of the study’s own method scored 87.5 ROC-AUC and 4.3 PR-AUC on the Credit Card fraud data (0.17% positives), both in percent. ROC-AUC measures how well a model ranks positives above negatives. PR-AUC focuses on the positive class, and its baseline moves with how rare positives are. For procurement, ask for PR-AUC with the prevalence stated beside it, ROC-AUC, a threshold metric at the operating point you plan to run, and a calibration measure. Neither AUC on its own is enough to sign against.
Why Does a Model Show a High ROC-AUC but a Low PR-AUC?
ROC-AUC is the probability that the model ranks a randomly chosen positive above a randomly chosen negative. The curve plots true positive rate against false positive rate, and both are ratios within a single class, so the curve does not depend on the class distribution. Fawcett (2006) showed this directly: with ten times as many negatives, the ROC graphs of two classifiers stayed identical, while their precision-recall graphs changed substantially.
Yet ROC curves can present an overly optimistic view when positives are rare. The false positive rate divides false positives by all negatives. When negatives vastly outnumber positives, even a large change in the count of false positives barely moves that rate, and ROC-AUC stays high. Precision divides true positives by everything flagged positive. With few true positives to work with, those same extra false positives drag precision down hard, and PR-AUC falls with it.
The published tables show how wide the gap gets:
| Dataset | Positive prevalence | ROC-AUC | PR-AUC | Source |
|---|---|---|---|---|
| Credit Card fraud (first-stage variant of the study’s own method) | 0.17% | 87.5 | 4.3 | VAE-Inf study, arXiv 2026 |
| Census (six models) | 6.2% | 56.2 to 93.9 | 7.4 to 59.0 | VAE-Inf study, arXiv 2026 |
| Census, ultra-imbalanced (six models) | 0.21% | 57.4 to 90.6 | 7.95 to 52.4 | VAE-Inf study, arXiv 2026 |
| ISIC 2020 skin lesions | 1.76% (584 of 33,126 images) | 0.87 | 0.16 | Transfer-learning ensemble study, arXiv 2021 |
| Citation-impact prediction, skewed sets | rare positive class | 0.63 to 0.75 | 0.28 to 0.46 | Citation-impact study, arXiv 2024 |
The same citation-impact study gives the balanced comparison: on balanced datasets PR-AUC ranged from 0.67 to 0.76, so moving to skewed data cut PR-AUC sharply while ROC-AUC barely changed. A 2024 study of hidden Markov model ensembles found the same pattern, with growing imbalance causing significant decay in average precision alongside a misleading increase in ROC-AUC.
Worked Example: The Same ROC Point at Two Prevalences
Saito and Rehmsmeier (PLOS ONE, 2015) give a clean illustration. They take one point on a classifier’s ROC curve. On a balanced dataset it corresponds to 500 true positives and 160 false positives; in the imbalanced case, the same point corresponds to 500 true positives and 1,600 false positives.
The arithmetic follows from the definition of precision (true positives divided by all flagged positives):
- Balanced: 500 / (500 + 160) = 500 / 660, about 0.76.
- Imbalanced: 500 / (500 + 1,600) = 500 / 2,100, about 0.24.
The ROC point has not moved, yet roughly three of every four alerts are now wrong. Nothing about the ROC curve was hidden: precision can be derived from TPR, FPR and the class ratio, as Richardson et al. (Patterns, 2024) point out. The ROC curve does not depend on the class distribution, so the buyer has to supply the prevalence. That is the practical case for asking a vendor to state prevalence beside every metric.
Reading a PR-AUC Score Against Its Prevalence Baseline
There is no universal good PR-AUC, because its floor moves with prevalence. A random classifier’s PR-AUC equals the positive fraction, P / (P + N): 0.5 on balanced data, about 0.09 at a 1:10 ratio. ROC-AUC, by contrast, has a fixed random baseline of 0.5.
Boyd et al. (ICML 2012) describe a second effect. Part of precision-recall space is unachievable for a given class ratio, which defines a minimum PR curve, and any model collects the area under that curve “for free”. Changing skew from 0.01 to 0.5 (for example by subsampling negatives) raises the minimum AUCPR by about 0.3, with no change to the model. The authors give an example where an AUCPR of 0.4 is worse than random on one task and reasonably high on another.
So read any PR-AUC against its baseline. The proposed ensemble’s average ISIC 2020 PR-AUC of 0.16, the best among the methods compared, sits against a random baseline of roughly 0.018 (584 malignant images out of 33,126). That is a long way above chance. Be careful with dividing PR-AUC by prevalence to get a “lift” number: Richardson et al. tested subtracting the baseline, dividing by it and min-max scaling, and found none of them removes the effect of class imbalance, because the effect depends on the classifier itself.
Two practical rules follow. Compare PR-AUC values only on test sets with the same prevalence, and never accept a vendor’s PR-AUC without the positive rate of the set it was measured on. If the vendor subsampled negatives, its PR-AUC rises with no change to the model.
Is an ROC-AUC of 0.75 Good Enough to Accept a Model?
On its own, no, and no agreed cutoff makes that decision. De Hond et al. (Lancet Digital Health, 2022) note that AUC values between 0.7 and 0.8 have been labelled variously as poor, moderate, fair or good. They recommend reporting AUC without labels and looking beyond it to calibration and net benefit. They also note that the lower the AUC, the more easily miscalibration after transporting a model to a new setting can make it harmful, depending on the decision threshold and event rate.
An ROC-AUC of 0.75 says nothing about which threshold to run. Fawcett notes that a high-AUC classifier can perform worse than a lower-AUC one in a specific region of ROC space. In the MCC-F1 paper (arXiv, 2020), a classifier with AUROC 0.73 had an AUPR of 0.20 on an imbalanced dataset, with precision generally below 0.5, while a rival with a lower AUROC of 0.69 scored AUPR 0.30.
Whether 0.75 is acceptable depends on the costs and risks of each error type and on whether the scores are calibrated. Agree those before anyone looks at the vendor’s number.
The 2024 Case Against Treating PR-AUC as Simply Better
The common advice to swap ROC-AUC for PR-AUC under imbalance has published critics, and a buyer should know them. McDermott et al. (arXiv, 2024) set out to refute the claim that AUPRC is the better metric for model comparison under class imbalance, showing theoretically that it is not generally superior. They also showed it can unduly favour improvements in subpopulations with more frequent positives, which can widen disparities. Richardson et al. (Patterns, 2024) argue that the ROC curve is robust to class imbalance while the PR curve is highly sensitive to it, and that imbalance cannot easily be separated from performance measured by PR-AUC. Their proposed alternative for focusing on the positive class is a partial ROC-AUC computed up to a chosen maximum false positive rate.
PR analysis also has its own blind spot. When positives outnumber negatives, precision barely moves even if false positives change substantially, and ROC analysis separates classifiers more clearly. Davis and Goadrich (ICML 2006) showed that a curve dominates in ROC space if and only if it dominates in PR space, but that optimising ROC-AUC does not guarantee optimising PR-AUC.
We do not read this as a reversal of the procurement advice. It supports reporting both numbers. ROC-AUC gives a view of ranking that does not depend on class distribution. PR-AUC shows precision, the fraction of correct positive predictions, at your class ratio. When the two disagree, that disagreement is the finding.
How ROC-AUC and PR-AUC Relate to F1 Score and Accuracy
Both AUCs are threshold-free ranking metrics: they summarise how well a model separates the classes across every threshold. ROC-AUC weighs both classes equally, and PR-AUC focuses on the positive class. Accuracy, balanced accuracy and F1 each require a fixed decision threshold, and a generic threshold of 0.5 would not be suitable for imbalanced problems. Average precision, as scikit-learn’s average_precision_score computes it, summarises the PR curve as a recall-weighted mean of the precision at each threshold.
Google’s ML Crash Course gives the standard example: with positives at 1%, a model that always predicts negative scores 99% accuracy and is useless. F1 balances precision and recall and is preferable to accuracy on imbalanced data, but it is still a summary at one threshold. Boyd et al. warn that it can hide information needed to interpret performance, and Van Calster et al. (arXiv) class F1 as an improper measure that also ignores misclassification costs.
In a vendor comparison, then, the AUCs tell you how well each model separates the classes. F1, precision, recall or error rates need a threshold, which in practice is chosen from the error costs involved. Ask for both, and make sure the threshold metrics are reported at your operating point, not one the vendor picked. Our wider piece on choosing which model evaluation metrics to report covers the metrics beyond these two curves, including explainability evidence.
PR-AUC vs ROC-AUC in a Vendor Evaluation Report: The Checklist
The checklist below is what we would expect a classification model’s evaluation report to contain before it goes in front of a procurement panel. Each row is phrased so it can be pasted into a request for proposal.
| Item | What to ask for | Why |
|---|---|---|
| Test set prevalence | Positive rate of every evaluation set, and whether any rebalancing or subsampling was applied | PR-AUC baseline equals prevalence; subsampling inflates it |
| ROC-AUC with interval | ROC-AUC with a confidence interval or variance across runs | Prevalence-independent ranking; comparisons need a variance measure |
| PR-AUC or average precision | PR-AUC (state whether computed as average precision) on a set at production prevalence | Shows alert quality at your base rate |
| Operating-point metrics | Precision, recall, F1 or error rates at a stated threshold you agreed | AUCs say nothing about the threshold you will run |
| Partial ROC-AUC, if early retrieval matters | AUC up to a stated maximum false positive rate | Separates classifiers whose full ROC curves cross |
| Calibration | Calibration plot or Brier score | Decisions use probabilities, not ranks |
| Utility at your costs | Net benefit or cost-weighted error at your threshold | Ties performance to misclassification costs |
| Held-out, representative data | Results on a held-out set drawn from your data, not only a public benchmark | Public benchmark results rarely match production distributions |
| Subgroup results | The same metrics per relevant subgroup, with each subgroup’s prevalence | PR-AUC can favour high-prevalence subgroups |
If a vendor can only offer ROC-AUC on its own data, treat that as a gap to close in the proof of concept rather than a figure to compare.
Fixing the Metric Thresholds Before the Proof of Concept Starts
The metric choice protects a buyer only if it is written down before results arrive. Vendor-evaluation guidance from Machine Learning Authority recommends specific task definitions, such as a churn model with a stated AUC-ROC threshold on named data, and scoring criteria for a proof of concept set before it begins, to avoid post-hoc rationalisation. For imbalanced tasks, we would extend that example to state a PR-AUC floor at the production prevalence and a precision or recall target at the agreed threshold alongside the ROC-AUC figure.
The NIST AI Risk Management Framework (AI RMF 1.0, 2023) names the underlying problem: the developer’s metrics may not match the deployer’s, and the developer may not be transparent about the methods it used. It also notes that measurements in a controlled environment may differ from risks that emerge in operational settings. Machine Learning Authority also notes that standard SLAs cover uptime and latency but typically not model accuracy drift unless negotiated, so MLOps drift monitoring has to be scoped as a separate contractual item.
TechnoLynx treats this kind of approval-grade evidence (metrics a buyer, auditor or model-risk committee can read and sign against) as an engineering output, not a marketing or policy document. It is produced as engineering artefacts, not as legal advice or certification. For the vision-specific version of the same discipline, the automotive perception validation package shows what reviewers sign against.
One question stays open after all of this, and it is about your threshold, not the vendor’s metrics. A report can include every item above and still leave you to decide which precision you can afford at which recall, given what a missed positive and a false alarm each cost you. Settle that number first, and the choice between ROC-AUC and PR-AUC becomes a question of which number makes your threshold visible.
FAQ
When Should a Buyer Ask for PR-AUC?
PR-AUC summarises the precision-recall trade-off across all thresholds and is used mainly when the positive class is rare and the quality of positive predictions matters. Fraud detection and disease screening are examples of such rare-positive problems.
Is a Higher PR-AUC Better?
On the same test set at the same prevalence, yes: higher PR-AUC means better precision across recall levels. Across datasets with different prevalence it is not directly comparable, because the baseline and the minimum achievable area both move with the class ratio.