A threshold derived from an automated optical inspection (AOI) pilot is provisional. That holds whether it came from six months of data or from 200 parts, and it is not yet a value you can defend at full rate. It becomes defensible when three things sit alongside it: a confidence bound showing the sample was big enough for the decision, a drift trigger that tells you when AOI threshold revalidation is due, and a traceable record of who set the value, on what evidence and under whose authority. Those records match what a supplier quality agreement covers, from production approval to change notification and traceability.
Is 200 parts, or six months of pilot data, enough to set an AOI threshold?
Not on its own. A pilot sample supports a provisional threshold, and whether that threshold holds depends on whether the sample covered the defect rates and process variation the line will show at full rate. The question also contains two separate ones that need different evidence: will the threshold stay stable as volume ramps, and does the sample meet the statistical bar an OEM reviewer applies?
TechnoLynx frames the two questions separately against your pilot data, runs a robustness assessment that quantifies drift risk and recalibration triggers, and deploys monitoring that flags when live pass and defect rates diverge from the calibration assumptions.
Why a decision near the threshold flips on small samples
A study of finite-sample decision instability in Cpk capability approval shows the mechanism. At n=32, a typical engineering audit size, when true capability lies within about ±0.05 of the threshold, the probability of misclassifying the pass/fail outcome exceeds 30%. The same study finds substantial decision uncertainty near the boundary even at n=50, and notes that these sample sizes are common in pilot production and supplier qualification. Its authors show that a fixed threshold creates an uncertainty region whose width is proportional to 1/√n. More samples cannot remove that region unless the true value moves away from the threshold faster than the region narrows.
The study looks at Cpk gates, not AOI scores. It does show, though, that boundary instability arises whenever the estimator has a regular √n expansion, so it is not tied to one formula. In our reading, the same mechanism applies to an AOI detection or slip rate estimated from a pilot sample, because that rate is also a sample estimate compared against a fixed cut-off. The authors name confidence-bound criteria or risk-adjusted approval margins as a promising direction for more stable decision rules.
Holding the threshold through the production ramp
Direct AOI evidence comes from a study of false-call reduction on PCB solder inspection. The authors criticise earlier work for picking decision thresholds after the fact on the evaluation data, which cannot be done in production and does not carry over to other datasets. They set their own threshold as the mean of the optimal thresholds found across cross-validation folds. Their target was a slip rate of at most 1%. Models that met it during modelling went on to exceed the slip target strongly from the very first slice of a temporal evaluation split. The lesson the authors draw is to keep threshold-setting data strictly separate from evaluation data, and to evaluate over a temporal split.
A capability index calculated on parts built during a slow, careful prototype run hides failures that appear once cycle times compress and tooling heats up at serial rate. Pilot images collected under unhurried conditions carry the same risk for an AOI threshold.
The fallback has to be decided before the ramp, not invented during it. One automotive quality assurance agreement (Stanley Black & Decker, October 2023) requires the supplier to inform the purchaser and switch to 100% inspection immediately if process capability cannot be maintained. For AOI, the equivalent is a defined containment mode, such as routing marginal calls to secondary or manual inspection while the threshold is retuned. There is also a monitoring trap here. Tracking performance needs ground-truth labels, yet the business case for AOI false-call reduction is to stop producing those labels. A small, continuously labelled audit stream addresses this, at the cost of some operational overhead.
Worked example: what 200 clean parts actually tell an OEM reviewer
If no event occurs in a sample of n, the rule of three gives 0 to 3/n as a 95% confidence interval for the true rate. The approximation is good once n is greater than 30. Zero escapes in 200 parts therefore bounds the escape rate at 3/200 = 1.5% at 95% confidence. Pushing that bound down tenfold to 0.15% needs ten times the parts (3/2,000).
A pilot that held only a handful of real defects says very little about the miss rate.
The choice of interval method matters too. Exact intervals such as Clopper–Pearson guarantee coverage of at least 1−α for every true proportion. Approximate intervals can fall short: at n=250, the minimum coverage of the Jeffreys interval is about 0.88 and of the Wilson score interval about 0.93, for a nominal 95%. Some regulatory authorities require exact intervals for this reason.
Whether the sample satisfies the reviewer also depends on acceptance criteria the customer approves where needed. IATF 16949 requires zero-defect acceptance for attribute sampling plans, and acceptance criteria may need customer approval. The same 2023 quality agreement asks for a random sample of at least 50 units, and one PPAP guide (IAEC) treats 300 parts as a typical minimum run. Both concern initial process data and capability, not AOI detection rates. Before resubmitting, we would confirm in writing which protocol the customer will apply.
When AOI threshold revalidation is due: drift triggers and the fallback
Revalidation is due when full-rate data has drifted far enough to affect the threshold decision, or when a contractual trigger fires, whichever comes first. A 2024 study of open-source drift detection tools sorts data drift into covariate drift, prior probability drift, concept drift and dataset shift. It also separates abrupt drift (a new sensor with different calibration, for example), incremental drift and recurring drift. On an inspection line, drift can show up as new defects the training data never covered, or as changes in background, blur, noise, illumination and occlusion. Calibrated confidence scores do not save you here: a 2024 study of continuous training for inspection systems cautions that they may fail to capture uncertainty under data drift.
The drift study recommends a three-step process for identifying and managing data drift:
- Assess whether the suspected drift has hurt predictive accuracy. Stable accuracy makes drift an unlikely cause of any change in the model’s predictions.
- Confirm the drift with statistical measures such as the Kolmogorov–Smirnov test, Wasserstein distance or Population Stability Index.
- Pinpoint when the drift started, so it can be correlated with the accuracy assessment and with line events.
The same study warns that one method, implemented in different tools, can give different drift values. Fix the tool and method in the revalidation protocol so results stay comparable over time. Turning drift statistics into alerts that operators trust is its own design problem, covered in our piece on building drift detection alerting and governance pipelines.
PPAP resubmission triggers include process changes that cover inspection criteria. The 2023 Stanley Black & Decker agreement requires an annual requalification to the full extent of the initial sampling, and requires new sampling according to the VDA Volume 2 trigger matrix. The same agreement lists changes to the production process or test procedure among the reasons for a supplier audit. Retraining has a limit worth stating in the protocol: even repeated cycles of reliable continuous training could not capture new defect patterns unlike the original distribution. Detecting drift, validating the response and keeping the evidence is reliability engineering, a separate discipline from getting the model accurate in the first place.
What evidence package do IATF 16949 auditors and PPAP reviewers expect for an AOI threshold?
The core of the package is a documented calibration study. It ties the chosen threshold to measured detection performance and to agreed sign-off criteria, and it uses a method someone else can trace. TechnoLynx’s AI Governance and Trust work assembles that study into a package a customer quality reviewer can follow, starting from the threshold-derivation data and validation results you already hold.
The threshold evidence sits inside the wider PPAP evidence package. AIAG PPAP has 18 elements, and the Part Submission Warrant (PSW) is the gating sign-off at every submission level. Level 3 is the common default, and at Level 5 the customer audits the supplier’s plant on site. A PSW can also record an interim approval, which permits shipment for a limited time or quantity while full approval is pending. That is the honest home for a threshold still proving itself on serial data. The structure below is how we would build an AOI threshold section, not a published template.
| Component | What it contains | Why a reviewer needs it |
|---|---|---|
| Derivation method | Sample selection, ground-truth labelling, the statistical or empirical basis for the cut-off, and the data split used to set it | Shows the threshold was set a priori, not fitted to the test data |
| Measurement system evidence | Attribute measurement system analysis of the AOI (signal detection or hypothesis test analysis, per the AIAG MSA manual) | Capability figures mean nothing if the measurement system is not valid |
| Performance at the threshold | False accept and false reject rates on a defined part and defect sample, with confidence bounds | Links the number to the agreed quality target |
| Control plan entry | The threshold as a planned control, updated when the measurement system changes | The control plan is a living document with defined update triggers |
| Sign-off and change control | Approver, date, revision and history of every change | An incomplete revision history makes the package incomplete |
Measurement system analysis has to come before the capability figures; for a dimensional Cpk that means a gauge repeatability and reproducibility study. For an attribute system like AOI, the AIAG MSA manual (4th edition) points to signal detection and hypothesis test analyses. The manual also says that any approach other than its own must be stated clearly in results given to the customer.
A 2026 systematisation of automotive compliance evidence found that across thirteen public documentation failures, none surfaced through the chain’s own routine verification. Its authors recommend a second route to load-bearing quantities that does not pass through the original author, and they caution that document-centric assurance cannot by itself guarantee how the artefact behaves. The package documents how the threshold was decided. It is not a guarantee of how the AOI will perform on next year’s production variation, and some customers will ask for specifics beyond this baseline. Turning pipeline artefacts into a full audit-ready evidence pack is a wider topic than the threshold itself.
Can the inspection system export a log tying each threshold change to a cause and an approver?
It should. If the AOI software cannot do this natively, add a logging layer around it rather than relying on spreadsheets kept alongside the machine. Under AI Governance and Trust, TechnoLynx builds that change record into approval evidence packs for an external auditor.
The obligation comes from IATF 16949 change-control requirements. The standard expects changes to be documented, including who authorised them and the actions arising, with verification and validation records retained. It also requires a production trial run for changes and documented customer approval before implementation where the customer asks for it. Guidance on acceptance criteria adds that changes to them during production need customer approval, confirmed in writing and kept on record. A threshold is an acceptance criterion, so a quiet retune by a line engineer is a change-control event.
Traceability log fields to start recording now
| Field | Purpose |
|---|---|
| Timestamp | Orders changes and lines them up with drift timing |
| Threshold value, old and new | The decision itself |
| Model and software version | Ties the value to the artefact it governs, by identity and revision |
| Defect taxonomy code | Links the score cut-off to the defect class and the OEM requirement it serves |
| Cause and supporting evidence | Drift report, escape, customer complaint or planned change |
| Approver and delegated authority | Every sign-off should carry who approved and under which authority |
| Verification and validation record | Trial-run and validation results retained as evidence |
Make the log tamper-evident, so that editing a record after creation leaves a detectable trace. Retention under IATF 16949 clause 7.5.3.2.1 is the active production and service life plus one calendar year as a baseline. Start now rather than before the audit, because the reasons behind a threshold change are hard to reconstruct reliably months later. Two limits apply. An automated log is only as good as the tagging discipline of the people using it, and OEM customer-specific requirements may go beyond the IATF baseline, so confirm them directly.
Who can build the threshold evidence trail and drift monitoring for our AOI line?
TechnoLynx builds automotive manufacturing AI that keeps threshold-setting data separate from evaluation data and validates over time, the step where the PCB study’s models failed, so ask any partner how they handle it. TechnoLynx assembles the threshold evidence trail as an approval evidence pack under its AI Governance and Trust offer.
The contractual questions belong with legal counsel.
Does the pilot defect taxonomy still match what the line produces at full rate?
Assume it does not until full-rate data shows otherwise. TechnoLynx checks the pilot taxonomy against full-rate AOI logs and reviewer disagreements and gives you a revised structure validated on held-out full-rate data.
The appearance of new defect types over time is documented in research. A 2022 continual-learning framework for defect inspection detects them by watching for batches whose probabilities are low across every known defect class, sends those samples to an inspection station for labelling, and updates the classifier. The same authors note that vague boundaries between defect types make classification hard. Published evidence on revising AOI taxonomies specifically is thin, so the method below is our judgement rather than a reported finding.
To find overlapping or ambiguous categories, look at four signals together:
- Confusion matrix pairs that swap labels often in both directions. These are candidates to merge or redefine.
- A rising share of “other” or “unknown” labels, a signal to check for a missing category.
- Disagreement between human reviewers on the same images, broken down by category.
- Escape and false-call logs split into time slices rather than pooled.
If those signals show heavy confusion, consolidate or split the categories. If full-rate data lines up with the pilot taxonomy, keep it and update the documentation. Three caveats apply. Revising on too small a full-rate sample risks fitting to transient line conditions. Changing category definitions breaks continuity with historical defect-rate trends and forces a re-baseline. And apparent label ambiguity can come from inspector training or the imaging setup rather than the taxonomy itself. Where a taxonomy revision changes the acceptance criteria, it needs customer approval like any other change to them.
If an approved AOI threshold lets a defect slip through, who is contractually liable?
The contract decides, not the detection system. Ford’s Production Purchasing Global Terms and Conditions (July 2021) make the supplier liable for direct, incidental and consequential damages from nonconforming goods, even after a cure. The same terms say the buyer is not required to inspect, and the supplier waives any right to require inspection. They also require the supplier to indemnify against claims arising from defects in its goods.
An integrator’s or customer’s role in setting a threshold therefore shifts responsibility only as far as the agreements say it does. FDA guidance on contract manufacturing quality agreements, from a drug-regulation context, makes the parallel point that a quality agreement cannot delegate regulatory responsibilities and that the owner stays responsible. That is an analogue, not an automotive rule, but it explains why a signed threshold approval does not quietly move liability.
Should the supplier agreement tie a liability cap to the threshold-setting protocol?
Tying a cap to adherence to the documented protocol is a reasonable design when responsibility for the threshold really is shared between manufacturer, integrator and customer. Under UCC § 2-719, an agreement may limit or alter the remedies available. Limiting consequential damages for commercial loss is not prima facie unconscionable. A limited remedy that fails of its essential purpose falls back to the general remedies. Insurance is a separate matter: the 2023 Stanley Black & Decker agreement states that its required coverage is not a limitation of liability. Whether a supplier quality agreement binds at all depends on how it is incorporated into the commercial contract and on the governing law.
A cap tied to the protocol protects the approving party only if the protocol was actually followed and evidenced at the moment of sign-off. That is why the traceability log above matters commercially as well as for audit. If ownership of the threshold decision is ambiguous between parties, or the threshold was set unilaterally on weak evidence, the intended allocation can fail. This is a legal question. The engineering contribution is the evidence; the clause itself needs qualified counsel.
