Before you talk to any AOI vendor, you need three things written down. The first is a measured estimate of how many defects get past inspection today. The second is your old inspection records, cleaned up under written rules for what is kept and what is excluded. The third is a defect taxonomy that quality has signed off, with an evidence threshold for each class. Visual inspection works on probabilities, and catching fewer than 100% of defects is normal, especially for small or low-contrast ones. That is why you cannot read the baseline straight off the reject log. Generic standards and vendor sample images are a useful starting frame. They do not tell you what your line produces, how often it produces it, or how well your inspectors catch it.
Three artefacts to have before vendor talks
| Artefact | What it contains | What makes it defensible |
|---|---|---|
| Escape-rate baseline | Estimated true defect rate and inspection detection rate per class, with confidence intervals | Two-inspector estimation or probability-of-detection studies that estimate what today’s inspection misses |
| Reconciled historical dataset | Mapping from old reject codes to the new classes, a rule for each kind of gap, a named data owner | Written data requirements, with records excluded only on objective, documented criteria |
| Signed-off defect taxonomy | A closed list of classes with severity tier, size range, location and reference photos, plus a tolerance for each class | Built from real production rejects, with inspector agreement measured by kappa |
What do we compare the vision system against when we don’t know how many defects slip through today?
Compare it against an estimated true escape rate, not the current reject rate. The reject rate counts only what today’s inspection catches. Manual inspection misses a real share of defects. In a Sandia study, 82 inspectors checked 140 parts for eight defect types. They correctly rejected 85% of the defective items and wrongly rejected 35% of the good ones. The pharmacopoeia chapter USP 1790 makes the same point another way: a particle size detected 70% of the time can still be missed up to 30% of the time. Pace matters too. A 2012 Sandia literature review by See cites a 1973 study by Schoonard, Gould and Miller where doubling the inspection rate raised misses from 23% to 30%.
To measure today’s escape rate, run a sampling study large enough to estimate a rare rate, over a set window. Small samples rarely show defects as rare as 1 in 1,000. A study of confidence intervals for rare events shows that for an event with a 1 in 1,000 chance, a sample of 100 or fewer has more than a 90% chance of containing none at all.
Standard acceptance sampling has the same blind spot. A sampling plan is described by its operating characteristic curve: the chance of accepting a lot at each true defect level, which spells out the risk to the producer and to the customer. Those calculations usually assume the inspection itself is perfect, which is exactly the assumption your baseline is there to test.
A second inspection pass gets you past this. If you run two independent passes (two inspectors, or one person and the existing machine), you can estimate both the defect rate and each inspector’s detection rate. You don’t even need to record which defects both of them found. The method comes from a paper on estimating defect rates under imperfect inspection. It assumes the number of defects per item follows a Poisson distribution and that each inspector finds each defect with a fixed probability. Under those assumptions, the defect rate and both detection rates can be estimated from per-item counts alone. With only one imperfect inspector and an unknown detection rate, you cannot get a consistent estimate of the defect rate at all.
Two assumptions carry the result. The process must be stable over the sampling period, and the two inspectors must be independent. Independence breaks down when some defects are simply easier to see than others. If you can match which defects both inspectors found, the classic capture-recapture estimator is more accurate.
Whichever method you use, keep its limits in view. A short window misses variation between batches and across seasons. Extra manual sampling costs line time. And the baseline is only a fair comparison if it uses the same defect definitions the vision system will be scored against.
Reconciling years of inconsistent reject records into a usable baseline
Historical reject data only becomes a usable baseline once you have a written reconciliation protocol. It needs a mapping from the old categories to the new taxonomy, a rule for each kind of gap, and a named owner who signs off the result before any comparison between model and baseline is run. The CRISP-ML(Q) process model recommends written data requirements that state expected values, formats and the maximum number of missing values. Records that fail them are set aside for manual review, not quietly added in. Any discarded sample should be removed on objective, documented quality criteria.
Categories changed over the years: re-label a sample under the new taxonomy
Sometimes the categories themselves changed: an old catch-all “solder fault” code might now split into bridging, insufficient solder and voiding. In that case, re-label a representative sample of historical rejects under the new taxonomy before the pilot starts, and record how the old codes map onto the new ones. Keep that mapping table as audit evidence. The re-labelled sample also gives an early test of whether inspectors apply the new classes consistently.
Missing or inconsistent fields in archived inspection data: drop the comparison or impute?
Fill in missing values (impute) only where a documented, defensible method exists. Otherwise, exclude the affected records and say plainly that the comparison covers less data. Single imputation, where each gap gets one best guess such as a mean, distorts the data distribution and usually biases the results, so it is best avoided. The generally preferred approach is multiple imputation: each missing value is replaced by several simulated values drawn from a model of the observed data.
The difficulty is knowing why the data is missing. A review of real-world health data finds the mechanism is often hard to determine, and the standard categories (missing completely at random, missing at random, missing not at random) can all co-exist. In health studies that pool several databases, missing data is highly likely because the databases record different variables. Take a shift that stopped logging defect location during busy periods. Those gaps are not random. Filling them in as if they were would understate exactly the conditions the vision system will face. Whichever route you take, disclose the bias it introduces.
Across sites and lines: harmonise without erasing real differences
You need to normalise codes across sites for a like-for-like baseline. Push it too far, though, and you hide real differences between lines. Keep the site as a recorded variable and compare within each site before pooling.
Who owns the clean-up, and will it survive an audit?
Name a data owner for the reconciliation before verification begins. We would put that role in quality rather than with the vendor, because the vendor benefits from a flattering baseline. Document the method so it can be reproduced. Even so, an auditor may not accept it, and some gaps will force a comparison to be labelled indicative only.
| What is wrong with the records | Rule | What to disclose |
|---|---|---|
| Categories changed over time | Re-label a representative sample under the new taxonomy and publish the mapping | Codes that do not map cleanly, and how uncertain the mapping is |
| Fields missing in otherwise comparable records | Multiple imputation with a documented model, or exclusion | The imputation model and the share of records affected |
| Records fail the written plausibility requirements | Set aside for manual review on documented criteria | The criteria and the number of records set aside |
| Gaps that cannot be reconciled | Exclude, and mark the comparison as indicative | The reduced coverage |
| Sites or lines log differently | Normalise the codes and keep the site as a variable | Any differences between sites that the harmonisation could mask |
Which defect types are behind the legacy AOI tool’s false calls?
Before deciding anything about a new model, break the legacy tool’s false-call rate down by defect class and component package using the existing inspection logs. The breakdown shows whether the problem is concentrated enough for a model change to help. Where the logs record the call type and board location, this is a query rather than a research project. That holds only if defect types are tagged consistently. If they are not, run the reconciliation protocol first, or the breakdown will reflect logging habits rather than the line.
Small passives such as 0402 packages are where we would look first. A study of an improved YOLOv11 detector for electronic products reports that one-stage models, including YOLOv11, miss defects against complex backgrounds such as overlapping components and metal reflections. They also raise false alarms where defect and background look alike, and small targets are a known weak point. The same study shows why evidence per class matters. Resistors and pads look alike and were easily confused once components were packed densely, while capacitors reached a precision of 0.991 and a recall of 0.998.
Whether a YOLO-based model would cut false calls on your dominant classes depends on evidence for comparable classes under comparable imaging. Check how far evidence like that transfers. Its PCB dataset, for example, is a public synthetic set of 1,386 images covering six defect types. If no comparable evidence exists, or the false calls follow a process issue such as solder paste volume, look at lighting, calibration or the process before the model.
How many examples of a rare defect class do we need before quality will sign it off?
There is no universal number. The count follows from how precisely quality needs to know that class’s detection rate. That precision is agreed explicitly and written down, then checked class by class against what is actually on file. Two published reference points help frame the agreement.
MIL-HDBK-1823A, the US Department of Defense guidance on assessing inspection reliability, suggests at least 60 defect sites for a system that only gives a hit or miss, and at least 40 for one that gives a measured signal. It calls these minimums. It also asks for at least three times as many defect-free sites, so that the false-call rate can be estimated. The handbook is guidance only and cannot be cited as a requirement.
For rates rather than counts, the same rare-event study argues that the margin of error must scale with the rate itself. It recommends a relative margin of error (the margin divided by the rate) between 0.1 and 0.5. Tighter targets demand impractically large samples. Looser ones give intervals too wide to use. For example, at a 1% defect rate, a 95% Wilson interval met both a 94–96% coverage tolerance and a relative margin of error of 0.5 or less at every sample size from 1,600 up. The authors note that in high-volume production it matters a great deal whether a defect rate is one in a thousand or one in ten thousand. That is why pinning down the order of magnitude is critical.
Turning repeated inspection into a per-class evidence threshold
USP 1790 shows how to turn repeated inspection into an evidence threshold for each class. Each reference defect sample is inspected 30 to 50 times, ideally by several inspectors. Its probability of detection (POD) is the number of times it was rejected divided by the number of times it was inspected. Samples with a POD of 0.7 or more go in the “reject zone”. Samples below 0.3 count as acceptable. Samples in the grey zone between can be included as acceptable units if you want. Each defect class’s acceptance criteria are then based on the POD measured when the test set is qualified. There is a separate limit on false rejection, with a recommended target below 5% of good units. Repeated inspections tell you about the inspection, not the parts. They are not independent samples and cannot replace more distinct defect examples.
Polarity reversal and other classes that barely exist on file
For a class like polarity reversal, the first job is a plain count of real examples on file, compared with the agreed minimum. Defect samples are scarce for structural reasons: tight yield control makes them rare, labelling them is costly, and companies rarely publish them. Generative models can synthesise defect samples to ease the imbalance. Their quality depends on the generator’s training data, though, and they may not reproduce the real range of defects. One study of synthetic training data found that moderate amounts gave the largest improvements and improved rare-class representation. Adding too much gave diminishing returns or slightly worse results.
Synthetic examples can fill a training gap, but they should not count towards the evaluation minimum. The held-out test set should hold real defects. If a class stays rare after a targeted collection effort, record that as a finding. Either the process rarely produces it, or it needs a different inspection route, and the taxonomy should say which. See also our guide to designing a regulator-defensible known-defect challenge set.
Rare classes are also where a small pilot sees the fewest examples. In retail and operational vision, our position is that off-the-shelf models fail at deployment scale, not because individual tasks are intractable. Scale breaks assumptions smaller deployments never expose, including severe imbalance across thousands of SKU classes and visually similar products.
Building the taxonomy from what your line actually rejects
No single published defect taxonomy fits every product. The defensible route is to start from your own rejects and use the nearest sector standard as scaffolding.
If the line is new, or you have no access to a running line, build the class list from a short structured survey of what nearby manufacturers and your own customers actually reject on, then trace every class back to a named requirement or reject log. TechnoLynx can run the structured capture of inspection criteria and design the traceability from each class to its source.
Research on material defect detection itself flags the lack of standardised datasets and their patchy coverage of defect types. USP 1790 describes the procedure. You identify defect types from naturally occurring rejects on the production floor, then sort them into critical, major and minor. Each type is described with its typical size range, its location on the part and, where feasible, a photograph, so the defect can be picked from production or re-created consistently.
For electronics assembly, IPC-A-610 is the most widely adopted acceptance standard. Its 2024 Revision J sorts conditions into acceptable, process indicator or defect, across three product classes with progressively stricter requirements. If a condition is not explicitly covered, the standard assumes it is acceptable unless it affects function or long-term reliability. Your own taxonomy should either adopt that default or override it in writing. Outside electronics, survey literature splits material defects into two groups. Surface defects include cracks, scratches, dents and colour variation. Structural defects include deformation and faulty joints. That split is a sensible first cut, but it says nothing about your tolerances.
Each class then needs its own tolerance. One QC checklist outline lists minor defect tolerance levels, major defect identification criteria, critical defect zero-tolerance policies and product-specific examples. For pharmaceutical visual inspection, USP 1790 lists typical acceptable quality limit (AQL) ranges of 0.010 to 0.10% for critical defects, 0.10 to 0.65% for major and 1.0 to 4.0% for minor.
Who should define and sign off the defect taxonomy and thresholds before we talk to any vendor?
Quality should own and sign the taxonomy, with one named person accountable. It should be built with the inspectors and production engineers who apply it, and signed before any vendor sees it. The held-out test set should be labelled against it. CRISP-ML(Q) recommends that the test set be assembled and curated by experts who can judge whether it represents real cases.
Sign-off should rest on measured agreement, not a meeting. A statistic called kappa measures whether several inspectors classify the same samples the same way. The automotive quality body AIAG suggests that a kappa of at least 0.75 shows good agreement, and values such as 0.90 are preferred. Where severity is graded on a scale, Kendall’s coefficients weight a misclassification by how serious it is, whereas kappa treats every misclassification the same.
The signed taxonomy then belongs in the specification sheet. The VDI/VDE/VDMA 2632 Part 2 standard exists to structure that specification between supplier and user. It now includes the acceptance and classification-performance tests from Parts 3 and 3.1.
What defect classes and detection rates can go into a warranty or audit commitment?
To stay enforceable as the line changes, a warranty needs a closed, versioned list of defect classes. Each class has a severity tier and its own detection-rate and false-call targets, measured against a named test set. TechnoLynx’s manufacturing AI engineering team can scope which classes, cameras and severity levels the commitment covers, derive per-class targets with a cost-weighted, audit-traceable method, and test the system against the agreed taxonomy before any figure goes into the policy.
A single flat percentage across all classes is hard to defend. A miss costs different amounts in different classes, so the target should differ too. For the derivation, see setting a defensible, cost-weighted threshold for defect detection.
For claims purposes, define a miss against a specific validation dataset and severity class, not as any disagreement with an inspector. MIL-HDBK-1823A says the point of a reliability demonstration is to find the largest defect the system can miss, not the smallest it can find. It also says a false-positive estimate belongs alongside, because a detection figure means little if the false-call rate is unacceptable. The handbook ties the number of false-positive opportunities to the precision the contract calls for: estimating a false-positive rate below 1% needs at least 100 independent opportunities, and below 0.1% at least 1,000. Accuracy is not a relevant figure for industrial defect detection. Strong class imbalance makes it uninformative, so the figures to write down are the false positive and false negative rates. VDI/VDE/VDMA 2632 Part 3, published in October 2017, defines measures of a machine vision system’s classification performance for use at acceptance, aimed at both users and suppliers.
On the contract side, the US federal acquisition rules (FAR subpart 46.7) treat being able to enforce a warranty as essential. They require some assurance that an adequate system for reporting defects exists. They say a warranty should generally give a stated period, amount of use or triggering event for demanding correction, and they require the warranty terms to agree with the specifications and inspection clauses. Those rules govern US federal government contracts. The same discipline is a sensible test for any inspection warranty.
Thresholds demonstrated on today’s line do not carry over to a significantly changed process without a new demonstration. The commitment should therefore oblige the client to keep supplying evidence: periodic sample audits, and re-validation after any camera, lighting or line change. Claims should be judged against the taxonomy version and thresholds in force when the alleged miss happened. MIL-HDBK-1823A applies the same rule to inspection systems: once a demonstration is complete, its process parameters should not change without a new demonstration showing the effect. USP 1790 adds that acceptance sampling after automated inspection catches defects that were never part of the system’s development and training. Finally, write a procedure for defects that do not fit a listed class. Without one, an over-rigid taxonomy turns every new kind of defect into a dispute.
When to re-run feasibility: pilot data, new clients and shared GPU capacity
Treat feasibility as a finding with gates and written re-trigger conditions, not a one-off report. Run a small labelled pilot for each client or use case. Set a go/no-go gate before committing shared production capacity. Re-run the assessment when real data moves outside what the pilot covered. TechnoLynx can scope the pilot for each client, decide whether the taxonomy should be shared or per client, and build and validate the pilot pipeline on the target hardware against labelled held-out data, with go/no-go and re-trigger thresholds defined in advance. CRISP-ML(Q) treats a feasibility check before project set-up as best practice that reduces early failures caused by false expectations. Its first phase delivers a documented scope, success criteria and a data quality report to approve feasibility.
Keeping the finding valid as pilot data volume and variety grow
CRISP-ML(Q) names the main reason models degrade: production data stops resembling training data. It recommends testing performance under production conditions that are introduced step by step, recalibrating to the deployed hardware at each step. Re-run feasibility when pilot data brings part families, cameras, lighting or defect classes the original sample did not contain. Re-run it too when the per-class evidence counts change materially. We would not borrow a fixed threshold for these triggers from the literature. Each programme therefore has to set them as concrete numbers in writing, such as a count of unseen classes or the share of images that fail the data requirements, rather than leaving them to judgement on the day. Data that fails the documented requirements is itself a warning sign: CRISP-ML(Q) notes that checking new data against requirements helps catch a shift in the data. Versioning the taxonomy and thresholds, so that each change does not reopen the whole feasibility question, is the subject of keeping defect taxonomies and inspection pipelines robust through change.
One shared taxonomy or per-client calibration on a shared endpoint?
Clients whose parts, cameras and lighting fall within what has been validated, and whose pilot meets the threshold, can go onto shared capacity; otherwise require a per-client validation gate or taxonomy calibration. Make the “not feasible” call before the client commits budget, as soon as early pilot results show the taxonomy or detector cannot generalise within the agreed error. CRISP-ML(Q) states plainly that when success criteria are not met, the team may go back to an earlier phase or stop the project.
What changes when the vendor finally walks in
With the three artefacts in hand, a vendor demo changes character. The vendor is no longer showing detection rates on its own sample images. It is scored on your held-out set, against your classes and your measured baseline. That is a very different conversation. Rare classes still carry real uncertainty, because accurately estimating rare rates needs quite large samples. An honest feasibility finding says so class by class, and marks those figures as indicative rather than letting one headline number cover them.
