A defensible IoU threshold is one you can trace back to its origin. It starts with a downstream tolerance (how far off a box can be before the planner, gripper or reject gate fails), passes through a stated matching rule, and ends at a metric limit framed by a named standard or benchmark convention. Taking 0.5 because it is the framework default gives a reviewer nothing to check. Do not expect a safety standard to hand you the number either: UL 4600 is goal-based and technology-agnostic, and it requires a safety case arguing that relevant objects will be detected, without prescribing how object detection is achieved.

An acceptance threshold is an argument with a number at the end of it, and the argument is what survives review.

Evaluation IoU threshold vs NMS IoU threshold: which one goes in the spec

Only the evaluation IoU threshold belongs in an acceptance criterion. The evaluation threshold decides whether a prediction counts as a true positive, a false positive or a miss. In the usual matching rule, a detection qualifies as a positive match only if it has the same class as the ground truth and an IoU above a threshold, conventionally 0.5 unless stated otherwise.

The NMS IoU threshold does a different job. In the Ultralytics YOLO configuration, iou (default 0.7) is the threshold for non-maximum suppression, and lower values remove more overlapping boxes to cut duplicates. A spec that says “IoU 0.7” without saying which one leaves the most important term undefined.

NMS IoU and the confidence threshold still matter to acceptance, as part of the configuration under test. Ultralytics notes that precision-recall curves, mAP and confusion matrices all use the conf value (default 0.001), and that a higher setting such as 0.25 can lower mAP. Freeze both in the acceptance protocol and record them alongside the result, so that the reported number refers to the pipeline that will ship.

Is an IoU threshold of 0.5 strict enough for vehicles and pedestrians?

For anything that uses box position downstream, 0.5 is loose, and the detection literature says so: the Cascade R-CNN authors describe 0.5 as “a very loose requirement for positives”. Benchmarks also set the threshold per object class instead of using one value for everything. These published conventions compare as follows.

Convention	Matching rule	Values
PASCAL VOC	AP at a single IoU	0.5
COCO primary metric	AP averaged over 10 IoU thresholds	0.50 to 0.95 in steps of 0.05; AP75 reported as the “strict” metric
KITTI object detection	Overlap required per class, ranked on moderate difficulty	0.7 cars; 0.5 pedestrians and cyclists
3D AP in the Waymo LET-3D-AP paper	3D IoU per class	0.5 vehicles; 0.3 pedestrians and cyclists
Testing the Safety of Self-driving Vehicles by Simulating Perception and Prediction	Box matching per class; AP reported at stricter values	Matching at 0.5 cars, 0.3 pedestrians and bicyclists; AP at 0.7 and 0.5
Pedestrian miss rate (Cascade R-CNN evaluation)	Miss rate at IoU	0.5, with 0.75 for high-quality detection
nuScenes	2D centre distance on the ground plane, not IoU	2 m

Small objects are hard for IoU: nuScenes moved to centre distance because objects with small footprints give zero IoU after only a small translation error. Raising the threshold also has a cost: detection performance often drops at larger thresholds, partly because positive training samples become scarce.

Whether 0.5 is enough depends on what consumes the box. A tracker that only needs to associate objects tolerates much more error than a stage that plans a manoeuvre or a pick from the box edges. Below, we turn that tolerance into a number.

[email protected], [email protected]:0.95, or both in a validation report?

Report both, and tie the acceptance criterion to the one whose matching rule reflects the downstream tolerance. COCO’s averaged AP is its primary challenge metric, and averaging over IoUs rewards detectors with better localisation. AP at 0.5 has the opposite weakness: the Cascade R-CNN authors call the VOC metric “fairly saturated”, and their gains were 0.8 points at AP50 against 6.1 at AP80 and 8.7 at AP90. Two detectors that tie at [email protected] can therefore differ a lot in how tightly they localise.

Write the IoU range out in full in the report. Some papers label the averaged metric “[email protected]” while meaning the average over 0.5 to 0.95, a label a reader can take for a single threshold. Recall summaries need the same care: COCO’s AR is the maximum recall given a fixed number of detections per image, averaged over categories and IoUs, with at most 100 detections per image.

Is 0.91 [email protected] good enough for foreign-object detection on a packaging line?

Whether 0.91 is good enough depends on what a miss and a false reject cost at your line speed. The FDA’s compliance policy guide on hard or sharp foreign objects (CPG Sec. 555.425) sets its criteria by object size: objects of 7 mm to 25 mm in ready-to-eat products, objects over 25 mm, and objects under 7 mm where a special-risk group is among the consumers.

The AI Readiness Scorecard from TechnoLynx is the assurance offer on this path.

The FDA size bands are useful because they mark which object sizes the guidance treats as hazardous. The board behind the guidance found that objects under 7 mm rarely cause trauma or serious injury except in special risk groups. So the test set should be stratified by size around those bands, and the acceptance clause should state recall for the hazardous band separately from cosmetic contaminants. A score that looks good on cosmetic defects says little about the safety-critical ones.

External figures are a sanity check, not a target. The nearest published foreign-object result comes from power transmission lines: the SESYOLO paper (Scientific Reports) reports [email protected] of 93.9% for bird’s nest detection and a 9% [email protected] gain over YOLOv8. That is a different domain with different object types and test sets, so it says nothing about what your line needs.

Should the sign-off metric be recall at a fixed false-positive rate?

For a reject gate, yes. Average precision is the area under the precision-recall curve across all confidence levels. Whether to favour precision or recall depends on the cost of false positives against false negatives in the application, and the false-positive rate depends heavily on the decision threshold chosen. Recall on the hazardous size band, measured at the false-reject rate the line can absorb, maps directly onto the two costs the plant manager cares about. Treat this as judgement, and present it to reviewers as reasoning, not established practice.

Text models need the same discipline: a finance evaluation harness

Text models need the same kind of task-specific evaluation, and ROUGE is the wrong tool for it in finance. ROUGE measures n-gram overlap with a reference, so a summary that changes one figure or attributes a quote to the wrong executive can still score almost the same as a correct one. For broker notes and earnings-call transcripts, we would use a held-out set checked automatically for figures and quote attribution against the source, then sample outputs for blinded analyst review.

Which standard should an IoU acceptance threshold cite for OEM and regulatory review?

Cite the standard that defines how the threshold is derived, because none of them supplies the threshold itself. TechnoLynx treats the chain from requirement to metric as engineering evidence that a non-engineer can read and sign against, which is different from certification or regulatory sign-off.

Acceptance criteria tied to a named standard or rubric

For road vehicles, the anchor is ISO 21448 (SOTIF), whose goal is the absence of unreasonable risk. The need to define risk acceptance criteria is normative there (Clause 6.1). Commission Implementing Regulation (EU) 2022/1426 also requires the ADS manufacturer to define the acceptance criteria from which validation targets are derived. Once quantitative acceptance criteria exist, validation targets can be derived from them at the level of harms, hazardous behaviours or triggering conditions. Formulations named for those criteria include GAMAB, positive risk balance, ALARP and minimal endogenous mortality.

ISO 26262 adds less here than people expect: it does not state the risk acceptance criteria behind ASIL. For the ML component itself, the ISO/PAS 8800 report used below requires refined AI safety requirements to trace to safety requirements allocated to the AI system, or to root causes of functional insufficiencies, with a justification that they are reasonable. KITTI and COCO then supply the metric definitions, and the safety case supplies the value.

Worked example: from a 1.2 m localisation requirement to an mAP limit

The BSI ISO/PAS 8800 report gives a published mapping that runs in order:

1. A vehicle-level safety analysis under ISO 26262 sets a permissible number of missed stop signs per 1,000 miles.
2. That becomes a recall requirement: correct detection in 95% of frames where a stop sign is within 80 m (SR1).
3. A per-frame false-detection requirement covers frames with no stop sign (SR2).
4. A localisation requirement follows: detection within 1.2 m of the true position (SR3).
5. Because the ML component measures localisation as IoU, not metres, the system requirements are met by choosing a limit on mAP.
6. The ML requirement is set at mAP no less than 0.90 (MLR1).
7. Validating MLR1 means showing, with distances calculated from video, that an IoU violating the mAP limit also violates the 1.2 m requirement.
8. MLR1 must hold across the ODD’s ranges of variability (MLR2); the selected model met mAP 0.9 up to a 0.5 haze perturbation.

Without it, 0.90 is just a number someone chose.

Worked example: what an IoU floor tolerates in box offset

The same mapping works in the other direction for a gripper or a reject actuator. Assume a predicted box the same size as the ground truth, shifted along one axis by a fraction f of the box width. Intersection is (1 − f) of the box area and union is (1 + f), so IoU = (1 − f) / (1 + f). Solving gives a tolerated shift of one third of the box width at IoU 0.5, about 18% at 0.7 and about 14% at 0.75. If the gripper’s mechanical tolerance is a smaller fraction of the object width than the IoU floor allows, the floor is too loose. Real errors also include scale and two-axis offsets, so treat this as a bound to check against measured error distributions, not a derivation on its own.

Sensor specifications can anchor the tolerance as well. The SORBET robustness study derived perturbations from LiDAR specifications showing 2 cm to 10 cm distance inaccuracy, found that even 1 cm to 2 cm can change detection results, and observed obstacles whose IoU changed by more than 0.5. Waymo’s LET-3D-AP authors state that the longitudinal tolerance should depend on what downstream modules such as tracking or behaviour prediction require. BEVFormer, with 30% lower absolute 3D AP than PointPillars, outperformed it on LET-3D-AP(L) once tolerance passed 10%. Evidence that a chosen threshold correlates with real grasp success on a warehouse validation set is a separate question, covered on its own page.

Per-class minimum recall at a fixed IoU, not a single mAP

Yes, put the gate on per-class recall at a stated IoU and report mAP as the summary. Recall has to be evaluated per class in multi-class settings, and under severe class imbalance the choice of macro or micro averaging significantly changes the reported figure. Per-class recall levels already appear in published evaluation: that perception-simulation study computed final displacement error at 90% recall for vehicles, 80% for pedestrians and 70% for bicyclists. Safety-oriented work goes further, arguing that precision and recall alone are insufficient for driving and that closer objects need stricter accuracy because there is less time to react.

This also answers the common question of how to show that [email protected] of 0.71 satisfies 95% recall on critical defects: it cannot be shown from the mAP figure. mAP integrates over operating points and classes, so the 95% clause has to be measured directly at the deployed confidence threshold for each critical class, such as a weld defect type.

One acceptance test method agreed with the OEM

Design validation and production QA should be judged by one written protocol, because the evaluation toolkit alone can move the result. The Roboflow100-VL authors found that Ultralytics’ trapezoidal integration of the precision-recall curve inflates mAP by as much as 2.7% compared with pycocotools. The matching rule can reorder models: on nuScenes, IoU matching gave small objects zero AP while centre-distance matching produced a clear ranking. Type-approval testing already works this way, with tests run in controlled conditions to ensure repeatability against a minimum performance.

The protocol both parties sign should fix:

• the evaluation IoU per class, or the distance-based matching rule if one is used
• the toolkit and version, including the PR-curve integration method
• confidence threshold, NMS IoU and maximum detections per image
• difficulty strata and what each excludes (KITTI, for example, defines easy, moderate and hard by box height, occlusion and truncation)
• the frozen test set, its size bands or distance bands, and who holds it
• per-class recall clauses and the operating point they are measured at

When bench results disagree with the OEM’s own rig

Settle the disagreement in the protocol before arguing about the model. Run both pipelines on the shared frozen set with both toolkits, then check integration method, matching rule and configuration in turn. Ultralytics’ trapezoidal integration alone can inflate mAP by as much as 2.7% over pycocotools. Where a gap remains after that, look at the labels: KITTI notes that about 2% of hard-case boxes were not recognised by humans, which caps recall at 98%, and a survey of more than 70 participants found that people judge localisation errors with identical IoU differently. That is a good reason to resolve borderline cases by agreed adjudication, not by re-tuning the threshold.

Caption-quality tolerance and Legal review

Captioning error-rate thresholds follow the same pattern: map complaint or regulatory risk per territory to a stated error metric and test method, then have Legal ratify the mapping. We would not assume any regulator has published a WER figure to cite until someone has checked each territory.

Acceptance thresholds set this way are release-gate validation work, part of production reliability engineering and distinct from model accuracy.

Who can help us set a detection acceptance threshold the customer will sign?

TechnoLynx builds the evidence that links a customer’s recall, localisation or defect-cost requirement to a specific IoU and mAP acceptance criterion through its Computer Vision R&D practice, as engineering work on your models and your test data. That covers mAP/IoU accept/reject threshold methodology with a customer sign-off package and a repeatable bench-to-vehicle test protocol. Where the first question is whether the current acceptance evidence is fit for review at all, the AI Readiness Scorecard is the assurance-stage entry point.

The engagement has clear limits. It does not certify a system or grant regulatory sign-off. Type approval sits with an Approval Authority or Technical Service.

The number is the last thing to agree

The argument over whether 0.5 or 0.7 is right is really an argument about what the box is for. Agree the downstream tolerance and the matching rule first, write down the step that shows a box failing the IoU limit also fails the physical requirement, and the threshold mostly follows. What stays uncertain is the evidence for that step on your own data, which is why it belongs in the validation package and not in a slide.

FAQ

How do you calculate IoU?

IoU is the area where the predicted and ground-truth boxes overlap divided by the area of their union. A prediction counts as positive when that value exceeds the chosen threshold.

Is recall the same as false positive rate?

No. Recall is true positives divided by all actual positives, while the false positive rate is false positives divided by all actual negatives.

What is a good mAP value?

No value is good in general. A mAP figure only means something next to its IoU range, toolkit and test set, and against the downstream requirement it is meant to satisfy.