Report PR-AUC when your deployment class balance is skewed toward negatives and the cost of a false positive is what you actually pay for; report ROC-AUC when classes are roughly balanced or when you care about general separability across all thresholds. The two metrics answer different questions, and picking the wrong one during a vendor eval is a common way to buy a model that looks good on a slide and underperforms on your data.

Why does ROC-AUC look so good on imbalanced data?

ROC-AUC plots true positive rate against false positive rate as you sweep the decision threshold. The false-positive-rate axis has the total number of true negatives in its denominator, and on a task where negatives dominate — fraud, defect detection, rare-event flagging — that denominator is enormous.

The consequence is a known trap: ROC-AUC can look reassuringly high on a heavily class-imbalanced task precisely because the true-negative-dominated false-positive-rate axis is insensitive to a large absolute number of false positives. You can generate thousands of false alarms and barely move the FPR needle. PR-AUC (precision-recall) does not let you hide that, because precision directly exposes how many of the model’s positive predictions are actually correct. On imbalanced data, PR-AUC is the more honest single number.

A concrete illustration: at a 1% positive prevalence, a model can post a 0.95 ROC-AUC while its precision at a usable recall level sits below 0.30. Nothing is contradictory there — the two numbers measure different geometry. But if you only saw the 0.95, you would badly overestimate how often a flag is worth acting on.

Which metric maps to a procurement decision?

The right metric to report in a procurement eval is the one aligned to the buyer’s cost of errors on their own base rate. ROC-AUC answers “how well does the model separate classes in general.” PR-AUC answers “when the model flags a positive, how often is it right at my prevalence.” A defensible eval reports the one that maps to the deployment’s actual class balance — not the one that flatters the model.

This is the part that gets skipped. Vendors quote whichever number is higher, and buyers rarely re-run the eval on their own prevalence. Precision is prevalence-dependent, so a PR-AUC computed on a balanced or vendor-selected test set tells you almost nothing about behavior at your 0.5% base rate. If you cannot re-weight the eval to your prevalence, treat the vendor number as directional at best. This concern sits inside the broader discipline of running a task-specific LLM evaluation that survives procurement scrutiny, where every reported figure needs to trace back to a decision the buyer actually faces.

How do you choose between them? A decision rubric

Use the rubric below, checking conditions top to bottom. The first row that matches your situation is the metric to lead with in the eval report.

Your situation	Report this	Why it holds
Positive class is rare (<~10%) and false positives carry a real per-item cost (analyst time, customer friction, a shipped defect)	PR-AUC, computed at your prevalence	Precision reflects how many flags are worth acting on; ROC-AUC will overstate quality here
Classes are roughly balanced and you care about overall ranking/separability	ROC-AUC	FPR axis is meaningful when negatives don’t dominate; ROC is threshold-agnostic and stable
You have a fixed operating threshold from a business SLA (e.g. “review at most 200 items/day”)	Precision and recall AT that threshold, not the AUC	AUC summarizes all thresholds; a procurement decision usually lives at one
Prevalence in production differs from your test set and you cannot re-sample	ROC-AUC as the portable number, with a prevalence caveat noted	ROC-AUC is invariant to class balance; PR-AUC is not, so a mismatched PR-AUC misleads
You genuinely need one number for a scorecard and the task is imbalanced	PR-AUC	It is the more honest single summary when negatives dominate

In the procurement evals we’ve run for teams shipping detection and classification services, the failure pattern is almost always a strong ROC-AUC hiding a precision problem that only surfaces once the model meets the real base rate. Our habit is to force the eval onto the buyer’s prevalence early and report precision at the intended operating point alongside the AUC, so nobody signs off on a number that won’t reproduce in production. If you’re standing up the surrounding evaluation and monitoring pipeline — the production monitoring harness that keeps an eval reproducible after the model ships — that instrumentation work is the kind of thing our AI infrastructure and SaaS practice tends to own.

What should the eval report actually contain?

Report both AUCs, but lead with the one matched to prevalence and state the prevalence you computed it on. Add precision and recall at the concrete operating threshold the deployment will use, because that is where the money is spent. Include a one-line note on how test-set prevalence compares to expected production prevalence — that single sentence is what stops a reviewer six months later from asking why the model “regressed” when in fact the eval was never run at the right base rate.

None of this is about producing a reference benchmark or ranking models on a leaderboard. It is narrower and more useful: interpreting a metric so a specific buyer can make a defensible choice for a specific deployment.

Frequently Asked Questions

Is PR-AUC always better than ROC-AUC?

No. PR-AUC is the more honest summary on heavily imbalanced tasks because precision exposes false-positive volume that ROC’s FPR axis absorbs. But on roughly balanced data, or when you need a class-balance-invariant number to compare across differing test sets, ROC-AUC is the more appropriate and portable choice.

Why does PR-AUC change when prevalence changes?

Precision has predicted positives in its denominator, which includes false positives, and the volume of false positives scales with how many negatives exist. So the same model produces a different PR curve at a 1% base rate than at 20%. This is exactly why a PR-AUC computed on a vendor’s test set can mislead if your production prevalence differs.

Can I just report accuracy instead?

Not for an imbalanced procurement decision. A model that predicts “negative” for everything at 1% prevalence scores 99% accuracy while being useless. Accuracy hides the error you care about; PR-AUC and precision-at-threshold expose it.

What if the vendor only gives me ROC-AUC on their own data?

Treat it as directional and ask for the test-set prevalence and predictions so you can recompute precision at your base rate. If they can’t share that, note the gap explicitly in your eval and don’t let a single unverifiable number drive the buy decision.