Model evaluation metrics in machine learning should be chosen from four things: the type of prediction task, the cost of each kind of error, the class balance, and the conditions the model will meet in deployment. That choice belongs at the start of a project, before anyone considers models or architectures, and without a very good reason it should not change once the results come in. Even a well-chosen metric is only trustworthy if it was measured on held-out data that resembles deployment and reported with an interval showing how much of a difference is noise.
Many evaluations report a standard set of metrics without first checking whether those metrics suit the data regime in question, and the same predictions can look strong under one metric and inadequate under another. Write the chosen metric down and version it as a release gate in your MLOps pipeline, checked before anything ships.
Choosing model evaluation metrics by task, error cost and class balance
Every metric carries an implicit penalty structure, so choosing a metric means choosing which errors matter. A 2025 arXiv study of cost-sensitive evaluation spells this out: accuracy assumes false positives and false negatives cost the same, ROC-AUC corresponds to a spread of cost distributions, and switching metric often changes which classifier comes out on top. The table below is the starting decision. The subsections after it cover the cases that need more than a table row.
| Condition in your data or use | What the default hides | Metric to lead with |
|---|---|---|
| Severe class imbalance | Accuracy is dominated by the majority class and can conceal poor minority-class detection | PR AUC, MCC or balanced accuracy, with the majority-class rate reported as a reference |
| False negatives and false positives cost different amounts | Accuracy implies equal costs | Weighted accuracy or normalised expected cost built from the cost ratio |
| Multiclass with rare but important classes | Micro-averaging across all labels reproduces accuracy | Macro F1 when performance across classes should be even; micro F1 when only instance-level totals matter |
| Regression with outliers or heavy tails | R² alone can conceal systematic error | MAE when extremes should not dominate, RMSE when large errors are operationally dangerous; avoid MAPE near zero targets |
| Downstream decisions use predicted probabilities | Checking only whether labels are correct ignores overconfidence | Log loss alongside calibration checks |
| The user receives a categorical decision | AUC, EER and FPR@FNR set the threshold on the evaluation data itself | The chosen metric, measured at the threshold fixed during development |
Why accuracy misleads under class imbalance, and when PR beats ROC
Saito and Rehmsmeier (PLOS ONE, 2015) built a simple example in which one classifier scores an accuracy of 0.6 on both a balanced and an imbalanced dataset, while its precision falls from 0.6 to 0.33 on the imbalanced one. Accuracy stayed at 0.6, and most other basic measures were unchanged. Only precision, MCC and the F-scores changed. A precision of 0.33 means a third of positive predictions are correct, and that is the figure that matters when acting on positive predictions.
ROC curves have a related blind spot under severe class imbalance. The false positive rate is divided by the total number of actual negatives, so when negatives vastly outnumber positives, a large absolute count of false positives can still give a low FPR. ROC and AUC work well for comparing models on roughly balanced data. On imbalanced data the precision-recall curve often gives the better comparison.
The PR curve has one property worth stating carefully. The ROC baseline is always the diagonal, but the PR baseline is the positive rate, P / (P + N): 0.5 for balanced classes and about 0.09 when positives and negatives run 1:10. A random classifier’s PR AUC therefore equals the positive rate, not 0.5. A 2012 paper on the unachievable region of PR space showed what follows from this: moving a dataset’s skew from 0.01 to 0.5, for example by subsampling negatives, raises the minimum attainable AUCPR by roughly 0.3 without any change to the model. Raw PR AUC scores from datasets with different positive rates therefore start from different minimums. The same paper proposes a normalised score, AUCNPR, that subtracts the minimum for the given skew.
Worked example: unequal error costs reverse the ranking
The 2025 cost-sensitive evaluation study gives a clear case, reproduced here with its assumptions: a false negative costs nine times as much as a false positive, and positives make up 20% of the data.
- Model M1 predicts negative for everything. Its accuracy is 80% and its weighted accuracy about 30%.
- Model M2 has TP = 15, FN = 5, TN = 50 and FP = 30. Its accuracy is 65% and its weighted accuracy about 71%.
The accuracy figures are easy to check: out of 100 examples, M1 gets the 80 negatives right, and M2 gets 15 + 50 = 65 right. Accuracy picks M1, while the cost-weighted metric picks M2. The authors are careful about the cause. The problem is not that predicting the majority class is wrong in itself; it is the strong cost asymmetry, and when costs are balanced, weighted accuracy equals accuracy. Correcting for imbalance carries its own risk. Rebalancing metrics and resampling only stay consistent with true cost near a specific relationship between cost ratio and class balance, so applying imbalance corrections by reflex can make model selection worse.
Set the decision threshold for deployment, not on the test set
Elkan (IJCAI, 2001) states the decision rule behind all of this: predict the class with the lowest expected cost, computed from the classifier’s probability estimates. He recommends training on the data as given and then making cost-sensitive decisions explicitly from those probabilities. Google’s Machine Learning Crash Course gives the practical version: where false alarms are expensive, choose a threshold with a lower FPR even if TPR drops, and where misses are expensive, choose the threshold that maximises TPR.
Two details decide whether this works. First, fix the threshold during development and treat it as part of the system, because metrics that set the threshold on evaluation data report performance no user will ever see. Second, use the class balance of the population the model will run on, which may differ from the development set; the 2025 study shows how to adjust for that base rate.
How do you validate a model so its metrics reflect unseen data?
Validate on data that was never used for any development decision and that looks as much like deployment as possible. Testing on the data a model was fitted on is a methodological mistake, and tuning hyperparameters against the test set leaks information into the model, so the metric no longer measures generalisation. A 2024 arXiv guide to good evaluation practice lists the usual ways this goes wrong: reporting results on the data used to choose hyperparameters or the random seed, splitting after augmentation or upsampling so near-duplicates end up on both sides, and ignoring dependence between samples.
The split has to follow the structure of the data. When samples come in groups (several from each patient, for example), every sample in a validation fold must come from groups that are absent from the training fold. Time series needs forward-looking splits, because shuffled folds put samples that are close in time on both sides and inflate the score. Stratified folds fix an engineering problem with rare classes, but the scikit-learn documentation notes that stratification also hides some real variability, so the spread across folds understates the uncertainty.
Model selection can overfit as well. Cawley and Talbot (JMLR, 2010) showed that overfitting the selection criterion degrades performance by amounts comparable to the differences between learning algorithms, so these effects cannot be ignored in evaluation. Finally, report results on meaningful slices as well as in aggregate, because a strong overall metric can hide a subset where the model fails.
When is a small metric difference real rather than noise?
A difference is significant when a bootstrap interval for it sits well above or below zero. A 2017 study found error bars of around ±10% on accuracy at neuroimaging-scale samples of 100, whichever cross-validation strategy was used. The same study found that the standard error across folds underestimates those bounds, because the folds are not independent. In our judgement, a two-point gain on a few hundred test examples is often indistinguishable from sampling noise.
Bouthillier et al. (2021) showed that variance from data sampling, parameter initialisation and hyperparameter choice can change benchmark conclusions substantially. The 2024 evaluation guide recommends confidence intervals over bare p-values and describes building them by bootstrapping. In order:
- Bootstrap the test set to get an interval for the metric, and bootstrap the paired difference between two systems; if that interval sits clearly above or below zero, the difference is significant.
- Resample by group (speaker, patient, site) when samples are correlated, or the interval will be too narrow.
- Train with several random seeds, bootstrap the test set for each, and pool the results so the interval covers seed variation too.
- Where compute allows, bootstrap the training data as well, so the interval reflects training data, seeds and test data together.
- A permutation test checks whether the estimator found a real dependency between features and targets; scikit-learn suggests more than 100 permutations and 3 to 10 folds.
If you have to use a classical test, choose it carefully. Dietterich (Neural Computation, 1998) found that a test for the difference of two proportions and a paired t-test over repeated random splits have a high type I error rate in certain situations and should not be used. McNemar’s test suits algorithms you can run only once, and the 5×2 cross-validation test suits those you can run ten times.
Why credible validation looks different for a risk model and an edge-AI benchmark
The right validation approach depends on the model’s objectives, methodology and data. For financial risk models, outcomes analysis includes established statistical backtests. An edge-AI accelerator verdict needs a disclosed, reproducible benchmark that separates the contribution of the silicon from that of the software.
LynxBenchAI publishes reference results for how each device runs the public PyTorch stack, giving an edge-AI comparison a disclosed hardware baseline that is broadly indicative for custom PyTorch models but does not measure your own workload; the first step is to check its published results for the devices under consideration. TechnoLynx builds the benchmark itself, starting by defining what the comparison must isolate on the target hardware (the silicon versus the software stack), then building a reproducible harness and publishing the method and results with any vendor involvement stated.
Backtests a risk committee will accept
For Value-at-Risk style models, the standard check counts violations against the number the confidence level implies. A study of VaR backtesting describes the Kupiec proportion-of-failures test as the most widely known: it asks whether the number of violations is consistent with the confidence level, under the null hypothesis that the model is correct. The Christoffersen interval forecast test checks whether violations cluster in time; it is a likelihood-ratio test following a chi-square distribution with one degree of freedom. Combining the two gives Christoffersen’s conditional coverage test. Because conditional coverage only looks at one lag, the same study adds the Engle and Manganelli dynamic quantile test, which regresses current violations on past ones.
For the report, the citable basis is the original Kupiec and Christoffersen papers, which that study references. For the supervisory framing, banking model risk guidance treats outcomes analysis (including back-testing) as part of validation and expects its rigour to match the model’s materiality. State the significance level you use in the report and justify it.
A Basel traffic-light zone classification is a banking-supervision tool. If a committee finds it a useful reading aid, include it as supplementary context labelled illustrative, never as a replacement for the likelihood-ratio tests.
A benchmark verdict readers and vendors cannot dismiss
Reddi et al. (2020) describe MLPerf Inference as a set of rules and best practices for keeping results comparable across systems with very different architectures. They also note that the sheer number of hardware and software combinations makes architecture-neutral, reproducible assessment hard. Mattson et al. (2020) add the traps on the training side: optimisations that raise throughput can lengthen time to solution, and time to solution itself varies from run to run. A defensible verdict therefore needs pinned software versions, the exact workload and precision, repeated runs, and enough published detail for someone else to reproduce it.
Vendor involvement belongs in the same disclosure. Describing dev kits, early access or embargo briefings may be commercially sensitive, but leaving them out is exactly what lets a sceptical reader or a losing vendor dismiss the result. State what was received and when, and what the vendor did and did not see before publication.
Which machine learning performance metrics to report for training versus inference
Training and inference need separate metric families, shown in separate dashboard panels, so a stack change is credited to the workload it actually affected. Training success is measured by training loss over time and sustained FLOP throughput; inference success by p50/p99 latency, tokens per second per GPU and cost per token.
LynxBenchAI measures how devices run the public PyTorch stack, giving hardware planning a reference point that does not measure your own training and inference jobs; the first step is to compare its published results for the devices you are planning around. TechnoLynx sets up separate training and inference panels so a stack change is credited to the workload it affected, starting by identifying which jobs on the shared pools are training and which are inference.
On the training side, RunPod’s GPU guide describes success as training loss over time and FLOP utilisation across a sustained run. Throughput on its own is not enough. The MLPerf Training authors note that optimisations that raise training throughput can increase time to solution, and the MLPerf Power study found diminishing returns in training time as the number of accelerators grows, with accelerator-hours and energy to train both rising. We would report throughput per unit of cost, such as tokens per second per dollar, next to time to target quality, because the first on its own rewards changes that slow convergence. Google’s production ML guidance also recommends testing training steps per second for each new model version against the previous version and against a fixed threshold.
On the inference side, the same RunPod guide lists p50/p99 latency, tokens per second per GPU and cost per token. Percentiles matter because, as NVIDIA’s AIPerf write-up puts it, a server with a healthy mean and an outlier p99 looks fine in aggregate and fails in production. NVIDIA’s AIPerf documentation also defines goodput, the number of requests per second that meet every SLO threshold, and notes that a system can have high throughput and low goodput at the same time. For LLM serving, AIPerf’s core latency metrics are time to first token, inter-token latency and end-to-end request latency. NVIDIA’s NIM documentation warns that tools define these differently, so compare results only when the definitions match. Triton’s Perf Analyzer splits server latency into queue, compute and overhead time.
Three caveats keep the split honest. On truly shared node pools, contention between training and inference jobs can still carry one workload’s change into the other’s panel. Throughput per dollar is only as accurate as the cost allocation per node and per job, which is hard to isolate on shared infrastructure. And tail percentiles need enough requests to be stable, so a low-traffic endpoint needs a wider measurement window before its p99 means anything.
Retiring an outdated prompting guide without breaking teams that shipped on it
Version the guide instead of deleting it. Freeze the old edition as legacy, publish the replacement as the current edition, and remove the legacy content only after an announced notice period has passed and migration notes are in place. Customers who miss the notice may still break at removal, and versioned docs add to the maintenance load if kept indefinitely, which is why the removal date matters.
What should replace the old chain-of-thought and few-shot examples?
Choose each replacement by what the task needs. Use a structured-output (JSON schema) example where the output has a fixed schema, a tool-calling example where the task calls an external action or API, and keep chain-of-thought guidance only for open-ended reasoning with no fixed output shape. Not every old example has a clean structured equivalent, so check each replacement for schema conformance against every model API it targets, and mark clearly which pattern is now preferred.
Mapping your existing ML tests to the ML Test Score and scoring the gaps
Use the rubric’s own categories and scoring definitions as the report template. Map each existing test to the criterion it actually covers, and record any criterion with no covering test as a gap, not a score. Breck et al. (Google, 2017) set out 28 specific tests and monitoring needs drawn from production ML systems, intended as a road map towards production readiness and lower technical debt.
TechnoLynx runs this as the AI Readiness Scorecard, which scores an AI programme against a named rubric such as the ML Test Score, with an evidence trail and a remediation backlog.
The rubric groups its tests into four areas: features and data, model development, ML infrastructure, and monitoring. Which area a given test belongs to is a judgement, but in our view a typical mapping looks like this:
| Rubric area | Existing tests that typically map here | What counts as a gap |
|---|---|---|
| Features and data | Data validation and schema checks, feature distribution checks | No check that training and serving inputs match |
| Model development | Held-out and sliced evaluation, baseline comparisons, adversarial robustness tests | No comparison against a simpler baseline, no slice metrics |
| ML infrastructure | Unit tests on pipeline code, reproducible training runs | Training cannot be reproduced from versioned code, data and seed |
| Monitoring | Drift checks, serving latency percentiles, prediction-quality tracking | Drift is tracked but no threshold triggers a review |
Score a criterion only where a test actually exercises it, and record its owner alongside it. Different reviewers will map the same test differently, so write down the reasoning for each mapping; that note is what lets the report stand up to a second reader. Risks that a single snapshot hides belong in the report too. If several model variants are due in quick succession and re-tuning is manual, record that as a risk under monitoring and infrastructure, rather than scoring those areas as covered because of how things stand today.
A precedent from another organisation would not guarantee acceptance by your auditor. Documentation practices from established validation guidance apply here as well. KPMG (2024) lists the scope of validation (inputs, theory and design, output, independent review, implementation, monitoring and reporting) and calls for documented procedures, data ranges, results and remedial actions, reported promptly to the right level of authority.
Who can help us set metrics, validation and audit-ready release gates?
TechnoLynx builds the engineering behind trustworthy evaluation for teams putting models into production: reproducible evaluation and benchmark harnesses, separate training and inference metrics wired into dashboards, and readiness reviews that turn existing tests into a scored release artefact with the gaps listed. That work includes eval harnesses and regression suites with per-bucket accuracy gates and drift detection.
This work is aimed at production AI systems. For the formal model risk validation a bank or insurer needs, the route remains its independent validation function or an external model validation firm, with engineering artefacts like these as inputs.
FAQ
What is the difference between model evaluation and validation?
Saito and Rehmsmeier separate evaluation during training, which selects the most effective model using validation subsets, from evaluation of the final model on test data that should reflect the real class distribution. In regulated settings, validation also means assessing a model’s reliability and limitations, with rigour scaled to its materiality.
