A perception model lands on a reviewer’s desk with a single headline number attached: mAP 0.91. The number looks strong. It also answers almost none of the questions the reviewer actually has. Does the model miss pedestrians at the operating threshold you shipped? Does it stay calibrated when the camera gain shifts at dusk? Is that 0.91 an average that quietly buries a class you cannot afford to miss? A metric is not a verdict on model quality. It is an evidence surface — a controlled measurement that proves one specific thing and stays silent on everything else. That distinction is where most validation packs go wrong. The naive move is to pick the aggregate score that looks strongest and treat it as the answer. The result is predictable: the reviewer reads the number, asks “what does this hide in production,” and the whole clarification loop restarts. A metric set chosen to answer the reviewer’s real questions closes those questions the first time. This is the difference between reporting a score and building an argument. What does a machine learning model performance metric actually prove? Every metric collapses a model’s behaviour into a scalar, and the collapse is lossy by design. The engineering discipline is knowing exactly what got thrown away. Precision and recall are the base pair, and they answer opposite questions. Recall asks: of all the pedestrians that were really there, how many did the model find? Precision asks: of all the boxes the model called pedestrian, how many were real? A model can score beautifully on one while failing catastrophically on the other, which is why reporting either one alone is an incomplete claim. The failure that survives an aggregate score is almost always a distributional one. mAP — mean average precision — is the standard object-detection headline, but the word mean is doing heavy lifting. It averages average-precision across classes and across confidence thresholds. A model with mAP 0.91 might carry 0.97 on cars (abundant, easy, forgiving) and 0.62 on cyclists (rare, small, safety-critical). The aggregate looks strong precisely because the easy classes dominate the average. In our experience reviewing perception packs, a high mAP paired with no per-class breakdown is the single most common trigger for a re-review round. Here is the reframe that matters: metrics are chosen against questions, not maximised against a benchmark. The reviewer approving an ADAS release is not asking “is this model good.” They are asking a specific, bounded question — “does it miss the vulnerable road users it must catch at the threshold we deploy?” — and the correct metric is whichever one answers that. We cover how those thresholds interact with the detector architecture in what a DETR-versus-YOLO validation pack must actually show; the metric-selection layer sits underneath that decision. Which metric answers which reviewer question? The mapping below is the operational core of this discipline. Each row pairs a question a reviewer actually asks with the metric that answers it and — just as important — what that metric stays silent about. This is a planning heuristic drawn from perception-validation engagements, not a benchmarked ranking; the point is the mapping, not the exact numbers. Reviewer’s question Metric that answers it What it does NOT prove Does the model miss things it must catch? Recall (per safety-critical class) Whether it over-triggers on clutter Does it raise false alarms at the shipped threshold? Precision at operating threshold Whether it misses rare true positives Is the overall detect/reject balance sound? F1 (at the deployed threshold) Behaviour away from that one threshold How does it trade precision for recall across thresholds? Precision–recall curve / AP Performance on any single operating point Is aggregate detection quality acceptable across all classes? mAP (with per-class breakdown) Any single class hidden by the average Can I trust the confidence scores as probabilities? Calibration (ECE / reliability diagram) Whether the ranking of detections is correct Does real-world performance match the test set? Drift metrics on live data Anything about the frozen validation snapshot Two rules make this table usable. First, F1 and precision are only meaningful at a stated threshold — a metric reported without its operating point is unfalsifiable, because the reviewer cannot reproduce it. Second, mAP without a per-class table is a claim the reviewer cannot audit; always ship the decomposition alongside the headline. Why calibration is the metric everyone forgets Accuracy metrics tell you whether the model got the answer right. Calibration tells you whether you can trust the model’s confidence in that answer — and in a system that fuses detections downstream or gates a control decision on a confidence threshold, that trust is load-bearing. A model that outputs 0.9 confidence should be correct about 90% of the time when it says 0.9. Modern deep networks are famously overconfident: they will report 0.99 on detections that are right barely 80% of the time. Expected Calibration Error (ECE) and reliability diagrams surface this directly. The mechanism is worth stating plainly because it is where a high-accuracy model still causes a production failure: if the downstream fusion stack treats a poorly-calibrated 0.95 as a near-certainty, it will act on detections that the model has no business being that confident about. Temperature scaling and similar post-hoc corrections exist precisely because raw softmax outputs from a network trained with cross-entropy do not behave like probabilities out of the box. We go deeper into where calibration enters the training and deployment pipeline in how a self-driving-car perception model learns and where calibration fits. The reviewer’s question here is subtle: “can I build on this confidence score?” No accuracy metric answers it. Only a calibration metric does, and leaving it out of a pack that gates decisions on confidence is a gap the reviewer will — and should — catch. How do metrics behave under drift? A validation metric is computed on a frozen snapshot. Production is not frozen. The question that separates a static evaluation from a monitored one is whether the numbers you approved still hold when the camera ages, the fleet moves to a new geography, or the seasons change the scene statistics. This is where metric selection extends past the release gate into monitoring. Drift shows up unevenly across metrics, and knowing the order of appearance is a practical early-warning system: Calibration degrades first, and quietly. Before recall visibly drops, the confidence distribution shifts — the model stays roughly as accurate but becomes miscalibrated on the new distribution. ECE tracked over time on live traffic is one of the earliest signals that the world has moved away from the validation set. Per-class recall drops next, on the tail classes. The rare, safety-critical classes are the first to suffer because they had the least training support to begin with. An aggregate that still looks healthy can sit on top of a collapsing cyclist recall. Aggregate mAP moves last. By the time the headline number visibly falls, the failure has usually been present in the distribution for a while. Watching only the aggregate means watching the slowest indicator. This is a directional pattern from monitoring perception systems, not a universal law — the exact ordering depends on the drift mode. But the principle holds: the metric that degrades first is rarely the metric on the front page of the pack. The perception failure modes that survive benchmarks are largely the ones that hide in the gap between a static aggregate and a per-class, drift-aware view. Tying each metric back to the test that produced it A metric a reviewer cannot trace is a metric a reviewer cannot trust. The evidence surface is only as strong as the link between the number and the exact test that produced it: which dataset split, which model version, which threshold, which preprocessing. When that chain is missing, the reviewer’s rational response is to ask for it — another clarification round, another re-run. The discipline that closes this gap is treating every reported number as an artifact with provenance rather than a standalone figure. That means the metric, the evaluation config, and the model version travel together. We describe the versioning half of this in machine-learning model versioning for automotive perception in practice, and the broader reliability discipline these metrics sit within is a cross-vertical concern — the same evidence-surface logic underpins perception validation regardless of the computer vision system it serves. FAQ What does working with machine learning model performance metrics involve in practice? A performance metric collapses a model’s behaviour into a single number by comparing predictions against ground truth on a fixed dataset. In practice, each metric proves one bounded thing — how many real objects were found, how many predictions were correct, how well confidence is calibrated — and stays silent on everything else. The practical skill is knowing what a given number deliberately discards. What does each core metric — precision, recall, F1, mAP, calibration — actually prove about a perception model? Recall proves how many real objects the model found; precision proves how many of its detections were correct; F1 balances the two at a stated threshold. mAP proves aggregate detection quality averaged across classes and thresholds, and calibration proves whether the model’s confidence scores can be trusted as probabilities. None of them individually proves a model is production-ready — each answers a distinct question. Which metric should I report for a given reviewer approval question rather than reporting the strongest-looking number? Map the metric to the question, not to what looks best. For “does it miss safety-critical objects,” report per-class recall; for “does it over-trigger at the shipped threshold,” report precision at that operating point; for “can I trust the confidence scores,” report calibration. Reporting the strongest aggregate instead invites the reviewer to ask what it hides, which restarts the clarification loop. Why can a high aggregate score still hide a production failure mode, and how do I surface that in the pack? Aggregates like mAP average across classes and thresholds, so a strong number can sit on top of a weak safety-critical class — high car accuracy masking poor cyclist recall. Surface it by always shipping the per-class decomposition alongside the headline and stating the operating threshold, so a reviewer can audit the number rather than take the average on faith. How do metrics behave under drift, and which ones signal that the model’s real-world performance is degrading? Under drift, calibration typically degrades first and quietly, followed by per-class recall on the rare tail classes, with aggregate mAP moving last. Tracking ECE and per-class recall on live traffic therefore gives earlier warning than watching the headline aggregate. The exact ordering depends on the drift mode, so treat it as a directional monitoring pattern rather than a fixed law. How do I tie each metric back to the test that produced it so a reviewer can trust the evidence surface? Treat every reported number as an artifact with provenance: the metric, the evaluation config, the dataset split, the threshold, and the model version travel together. A number the reviewer can reproduce is a number they can trust; one they cannot trace forces a re-run. Binding metrics to their producing test is what turns a score into evidence. The next time a perception pack opens with a single headline number, the useful question is not whether the number is high. It is which reviewer question that number was chosen to answer — and which questions it was quietly chosen to avoid.
