No single confidence threshold tells you a model is accurate enough to trust. Trust has to be earned through a documented protocol: an independent evaluation against labelled or adjudicated reference data, sized in advance; confidence scores checked against what actually happens; and a pass/fail threshold written down before the first test sample is scored. In medical-device verification that last point is not optional: auditors compare protocol approval dates with test execution dates to catch acceptance criteria written after the results were known.
This work is part of production AI reliability, an engineering discipline distinct from model accuracy. Its deliverables include task-specific evaluation harnesses, regression suites and validation packages for vision and perception workloads; a dashboard is not the deliverable.
Verifying a claimed detection accuracy before sign-off
A vendor’s claimed detection accuracy should be verified by an independent evaluation against a reference set the vendor did not build, sized before testing starts. On its own, a vendor-reported figure is a claim, not evidence. Device-software practice treats algorithm accuracy the same way: as something verified by comparison to reference datasets.
LynxBenchAI measures throughput, latency, memory and cost per unit of work across candidate accelerators when a validated model is about to justify a GPU scale-up, and the first step is to name the workload and the accelerators you are weighing.
It measures hardware, not your model’s accuracy.
The protocol, in the order a reviewer applies it
- Fix the acceptance criteria first. As device-verification practice requires, each criterion needs a quantitative threshold, a justification and explicit pass/fail logic, and it is approved before the first sample is evaluated.
- Check the reference labels. Measure inter-annotator agreement on an overlapping subset and send disagreements to adjudication.
- Characterise the measurement system. Where the system measures a physical quantity, a Gauge R&R study estimates repeatability and reproducibility.
- Measure calibration and discrimination separately. Use a reliability diagram for calibration and an ROC curve for discrimination.
- Size the sample and report an interval. Use a reliability-and-confidence sample size for pass/fail data, with exact or Bayesian intervals when the sample is small.
- Stratify before blending. Report performance per condition before any single headline figure.
- Record the decision. Document the intended use, study design, limitations, and the conditions under which the conclusion stops holding. In medical-device verification practice, a failed test triggers root cause analysis, never re-testing until something passes.
Which branch of evidence you need depends on what the claim is about. Some claims are measured against a physical or manual process: logo pixel size and on-screen duration against manual counts, or defects against a human inspector. Those call for repeatability work plus a challenge set of known defects, scored with precision-recall curves. Other claims are probabilistic: sensitivity to adjudicated adverse events, recall on historical manipulation episodes, or whether a drift detector fires. Those call for a replay of archived, adjudicated cases with confidence intervals and calibration curves. A replay runs on a data-specific evaluation harness, the task-specific kind production AI reliability work builds. And the replay is only as good as the coverage of the adjudicated set behind it: an episode type that is missing from the archive is never tested.
Is there a single confidence threshold that proves a model is accurate enough?
No. Even in classical measurement-system analysis, guidance on Gauge R&R warns against applying a universal percentage threshold as an automatic accept/reject rule. Acceptability there depends on intended use, the tolerance or process relationship, the study design and the consequences of a wrong decision. A standard on acceptance testing of classifying machine-vision systems says their variety makes a general assessment scheme scarcely possible. Instead it suggests procedures for assessing classification performance during acceptance.
A detection model’s confidence threshold is an operating point on a precision-recall trade-off. It is not a certificate of accuracy. The threshold belongs to the task, set from the cost of a miss against the cost of a false alarm. For per-field-type baselines and the two-tier acceptance testing that follows, see the companion piece on confidence-threshold tuning for industrial vision inspection. Thresholds tied to operational consequence are a decision of their own, and whoever carries that consequence should agree them before testing begins.
Calibration, verification and Gauge R&R answer different questions
In the international metrology vocabulary (VIM), calibration relates the values given by measurement standards, with their uncertainties, to an instrument’s indications, and then uses that relation to obtain measurement results. Calibration is a prerequisite for verification, which confirms that specified requirements are met. Verification is a pass/fail against tolerance; it neither quantifies the error nor creates traceability. Validation asks a third question: whether the system is fit for its intended use at all.
Calibration and Gauge R&R answer different questions. Calibration establishes the relationship to reference standards under defined conditions. Gauge R&R estimates repeated measurement precision across the study design. A calibrated instrument can still repeat poorly in use, and a highly repeatable instrument can still be biased. One guide puts the split as calibration proving a gauge accurate and R&R proving it repeatable and reproducible.
The trap is treating Gauge R&R as an accuracy test. A system can have excellent repeatability and reproducibility and still read consistently high, change bias across its range, drift over time or lack the resolution to separate meaningful differences. It names choosing parts to manufacture a favourable percentage as another.
In machine learning, “calibration” carries a second meaning: a probability forecast is calibrated if events happen in 100·p percent of the cases where the model said p. Both meanings matter for a camera-based detector. The metrological one covers the measurement chain (camera, lighting, fixture); the statistical one covers the model’s scores.
Should you require a Gauge R&R study before sign-off?
Yes, wherever the system reports a physical quantity (size in pixels or millimetres, duration, position) and decisions sit close to a tolerance. Define the intended decision before choosing the comparison basis. A gauge for tight specification decisions may need strong discrimination relative to tolerance even when process variation is broad. The NIST/SEMATECH handbook’s design guidance asks for more than two artefacts, a random sample of more than two operators when the study represents a larger pool, and more than three gauges when it represents a larger fleet. For a vision station, we would treat each camera-plus-lighting-plus-model configuration as a gauge, and the people loading parts or adjusting fixtures as appraisers. Hide earlier readings where you can, because appraisers who see the previous result drift towards it and make repeatability look better than it is. When an item cannot be remeasured equivalently, a nested design replaces the crossed one.
Can the reference labels themselves be trusted?
The labels deserve their own check before anything is scored against them. Inter-annotator agreement measures how consistently independent annotators label the same data, and it sets an upper bound on what a model trained on that data can learn. Annotation provider Claru puts the ceiling bluntly: a model evaluated on labels from a process with a kappa of 0.80 cannot reliably exceed 0.80 accuracy on that label distribution. Its standard practice is to have 10-20% of items labelled by several annotators, with high-disagreement items sent to a senior annotator for adjudication. Its guidance treats a Cohen’s kappa of 0.70 as the usual minimum for production classification labels, and a bounding-box IoU of 0.75 or more between annotators as good.
When two people label the same shelf photo differently, the disagreement is information. Low agreement on one label class points to ambiguous guidelines that need revision. Whatever disagreement survives adjudication belongs in the report next to the accuracy figure, not left out.
How is a confidence score different from real-world accuracy, and how do you calibrate one against the other?
A confidence score is the model’s own estimate. Accuracy is what happens when that estimate meets labelled reality, and the two agree only if the model is calibrated. For a detector, calibration means that of 100 predictions made at 80% confidence, about 80 should be correct. Guo et al. (2017) found that modern neural networks, unlike those of a decade earlier, are poorly calibrated. Küppers et al. found common object detection and instance segmentation models miscalibrated too. Semantic segmentation models in that study were already well calibrated and needed no post-hoc correction, so the honest rule is to measure, not assume.
Calibration and discrimination have to be judged separately. An ROC curve does not change under any strictly increasing transformation of the scores, so it measures discrimination only and is blind to calibration. The “triptych” of Dimitriadis, Gneiting and co-authors pairs a CORP reliability diagram for calibration with a concave ROC curve for discrimination and a Murphy curve for overall predictive value. The authors propose the miscalibration component of the Brier score as the calibration metric, and they add consistency bands showing whether a departure from the diagonal is larger than estimation noise alone would produce. For detectors, the detection expected calibration error (D-ECE) bins predictions by confidence and measures the gap between average confidence and observed precision in each bin.
Miscalibration can be reduced after inference, either with binning methods such as histogram binning and isotonic regression or with scaling methods such as Platt and temperature scaling. Guo et al. found temperature scaling, a single-parameter method, surprisingly effective on most datasets. Scaling methods are monotonic, so they leave ranking metrics such as AUPRC untouched. Calibration can also depend on where an object sits: Küppers et al. measured a clear rise in calibration error towards the image boundaries.
Worked example: fixing the evaluation before scoring
The Küppers et al. study is a small example of this discipline. The authors split the 5,000-image MS COCO validation set at random, 50/50, fitting calibration on one half and evaluating it on the other. For calibration they used IoU match thresholds of 0.50 and 0.75 and kept only boxes scoring above 0.3. An acceptance test should follow the same pattern: calibration data kept apart from evaluation data, and match rules fixed before results exist.
How much sampling do you need to show a station meets its stated detection rate?
Enough real positives to demonstrate the claimed rate at a stated confidence. The count that matters is defects or events presented to the system, not images processed. Under most test conditions, the number of correct detections in n trials follows a binomial model. For pass/fail data with zero failures allowed, the required sample size is n = ln(1 − C) / ln(R), where R is the reliability to demonstrate and C the confidence. At 95% reliability and 95% confidence, that comes to 59 samples with zero failures. Higher-risk requirements warrant higher reliability and confidence, and the justification should point to the risk analysis.
Worked example: zero-failure sample sizes
Assumptions: independent pass/fail trials and zero failures allowed.
| Detection rate to demonstrate (R) | Confidence (C) | n = ln(1 − C) / ln(R), rounded up |
|---|---|---|
| 0.90 | 0.95 | ln 0.05 / ln 0.90 = −2.996 / −0.105 → 29 |
| 0.95 | 0.90 | ln 0.10 / ln 0.95 = −2.303 / −0.051 → 45 |
| 0.95 | 0.95 | ln 0.05 / ln 0.95 = −2.996 / −0.051 → 59 |
| 0.99 | 0.95 | ln 0.05 / ln 0.99 = −2.996 / −0.010 → 299 |
The arithmetic comes from the formula above, and the 0.95/0.95 row matches the published 59-sample plan. One miss fails a zero-failure plan. It does not mean you top up the sample: if failures are allowed, the confidence calculation changes and n grows, as Joshi (2023) works through for f failures in n samples.
When the evaluation is small, the choice of interval matters. NIST Technical Note 2119 finds that large-sample methods diverge noticeably at n ≤ 40, prefers Wilson or Agresti-Coull over the standard Wald interval there, and recommends exact or Bayesian methods below roughly 20 to 40 samples. Clopper-Pearson exact intervals never under-cover, but they can over-cover strongly at small sample sizes. If the goal is a margin of error rather than a demonstration, the same note gives n ≥ z²·p(1 − p)/δ². For a 95% interval (z = 1.96), an expected rate p* of 0.95 and a margin δ of 0.02, that works out to 3.842 × 0.0475 / 0.0004 ≈ 457 positives.
Recording the actual performance, such as time to detection, instead of a bare pass/fail opens up parametric estimation. That can be more precise from smaller samples, provided the model fits the data. Even so, a tight interval narrows the claim without ruling out a rare failure mode the sample never contained.
Should you stratify the evaluation set before reporting one accuracy figure?
Yes. Stratify by the conditions that change difficulty (lighting, shelf category, defect class, device type) before blending, because one aggregate figure can hide a weak stratum. AlphaFold evidence shows how large the gap between strata can be. In one preprint, high-confidence failure rates ran from 80-97% for proteins with complete secondary-structure refolding down to 0-2% for local backbone rearrangements. Each stratum you intend to claim needs its own sample size. The design questions, from choosing strata to handling sites and hardware, are covered in why one blended metric is not enough.
What acceptance threshold belongs in the purchase contract?
The contract should name a measurement, not a headline number: the metric, the reference set and how it is drawn, the sample size, the interval method, and the bound that must clear the threshold. Device-verification practice expects each criterion to be quantitative, justified and stated with its pass/fail logic, for example “all samples must pass” versus “x of y must pass”. The GUM (JCGM 100:2008) sets general rules for evaluating and expressing measurement uncertainty. When uncertainty is not negligible against the tolerance, a documented decision rule says how a borderline result is handled. Guardbanding is the common one: it pulls the acceptance zone inward so that a pass means the item is confidently in tolerance.
The protocol should cover who is notified of deviations and who may approve continued testing. It should also require root cause analysis before any re-test. An auditable per-item report that the buyer can sample independently makes all of this enforceable. Whoever runs the acceptance test should have no stake in the result; the conditions for real separation are set out in independent model validation: separation, evidence and scope. Lawyers draft the clauses; the engineering job is to make the measurement they point to unambiguous.
Who can independently review our validation metrics before we present them to leadership?
TechnoLynx builds the evaluation harnesses and validation packages behind a model’s accuracy claim. The work covers a task-specific evaluation harness that replays your archived cases and a validation package a leadership team or customer can audit. TechnoLynx delivers it as part of Production AI Reliability. When choosing an outside partner, look for one with no stake in the model’s success who will show you the protocol before any results.
This is engineering evidence, not regulatory sign-off. Notified Bodies still assess verification protocols and reports themselves during conformity assessment. Where traceable measurements are required, the instrument needs calibration by a traceable, accredited source. A statistician can design the sample if the harness already exists, and an internal second-line validation team fits well where one is in place.
What pLDDT score is accurate enough to trust an AlphaFold model for construct design?
No single pLDDT cutoff marks a model as safe for construct design. The workable approach is a project-specific threshold, set by comparing predictions with whatever solved structures exist and checked region by region. It appears per residue in the B-factor column of the model file and as a global per-model score in the log.
How well the self-estimate tracks reality depends on the structure type. A 2024 Bioinformatics benchmark found pLDDT an accurate descriptor of tertiary model quality against observed lDDT-Cα (Pearson r = 0.97). The same benchmark found weaker agreement for quaternary structures (r = 0.67), with significant overprediction for some lower-quality models. A 2026 bioRxiv preprint goes further for fold-switching proteins, reporting that pLDDT and pTM fail systematically there. Across 27,098 predictions for 18 experimentally validated fold-switching proteins, it found that 33.6% were high-confidence predictions committed to the dominant fold that failed to capture the known alternative conformation. A strong population correlation still leaves individual models wrong, which is why a correlation is not a cutoff.
The decision depends on what reference data you have. If solved structures of the target or close homologs exist, a head-to-head lDDT or RMSD comparison is the primary gate, weighted towards the regions the construct depends on. If none exist, pLDDT is an internal heuristic only. In that case, inspect the construct-critical zones residue by residue and apply a conservative, project-defined threshold, with orthogonal evidence such as biophysical assays rather than one structural number.
A good mean pLDDT can hide confidence failures in specific regions. The preprint found that variance in pLDDT across clusters separated reliable from unreliable predictions even when mean pLDDT values were indistinguishable. Two further limits apply. Thresholds learned on one fold class may not transfer to a novel domain, and the crystal structures used as references carry their own experimental uncertainty, which bounds how precise the comparison can be.
How do you validate AlphaFold predictions when only a handful of solved structures exist?
A handful of solved structures gives little statistical confidence on its own. Treat the result as provisional, report the uncertainty explicitly, and add independent evidence before calling the model validated. The arithmetic makes the point. Suppose each structure counts as a pass or a fail against an agreement criterion set in advance. Then five passes out of five give a confidence of 1 − 0.9⁵ ≈ 0.41 that the true pass rate is at least 90%, using the zero-failure confidence relation in Joshi (2023). At that scale, NIST’s guidance points to exact or Bayesian intervals rather than large-sample approximations. The same note encourages priors built from historical or other external information where it exists, keeping noninformative priors as a last resort.
LynxBenchAI handles the hardware side of this work: when a structure-prediction pipeline is about to justify a GPU scale-up, it measures throughput, latency, memory and cost per unit of work across candidate accelerators. It measures hardware, not the accuracy of the predictions, so the validation steps here still apply.
Two kinds of evidence widen the base. The first is cross-checking against public reference structures in the Protein Data Bank for the same fold class. We would prefer structures released after the model’s training data was fixed, since agreement on structures the model has effectively seen says little. Benchmark sets may still not cover the structural class you care about. The second is an orthogonal assay, another experimental or computational method, so the result does not rest on one comparison method alone. Its cost and time should be proportionate to the decision riding on it.
How much confidence this case can reach has not been measured here. It depends on which external datasets and assays are actually used. And a statistically sound comparison measures agreement with reference structures, nothing more. Whether the structural conclusion is biologically right remains a separate question, and it belongs to the structural biologists making the construct decision.
FAQ
What is the difference between accuracy and precision in measurement?
Accuracy is closeness to the true value; precision is how closely repeated measurements agree with each other. A system can be very repeatable and still consistently biased, which is why repeatability studies do not establish accuracy.
Is false positive rate the same as precision?
No. False positive rate is the share of true negatives the system wrongly flags, while precision is the share of flagged items that are truly positive. The two can move independently when the positive class is rare.
