If you do not yet know whether your defect can be seen and separated in images at all, run a cheap feasibility check first. If you already know it can, and the open question is which vision inspection system to specify for your board family, run a full independent pilot. The two come one after the other, but they answer different questions, and treating a good feasibility result as a stand-in for a pilot (or skipping the check and buying pilot budget to learn something a few hundred images could have told you) is the conflation this page argues against. A feasibility check asks whether the defect is visible and learnable from little data. A pilot asks which system catches known defects on boards it has never seen, at a false-call rate the line can live with.
Feasibility check and full pilot compared, question by question
| Cheap feasibility check | Full independent pilot | |
|---|---|---|
| Question it answers | Can this defect be separated from acceptable variation in images? | Which candidate system finds seeded defects on unseen boards, per defect class, at a tolerable false-call rate? |
| Typical data | Tens to a few hundred labelled images per class, often archived or taken with whatever camera is to hand | Golden and seeded boards from the actual board family, counted in inspection sites |
| Imaging | Not the final rig | Each candidate system, programmed and thresholded as it would run in production |
| Output | A directional go or no-go | Escape rate and false-call rate per defect class, with confidence bounds, plus repeatability |
| What it cannot tell you | Production detection performance | Long-term performance as board designs and processes change |
Can we find out cheaply whether the defect is solvable before we buy a camera rig?
Images you already hold, such as archived photos from manual inspection, are a cheap first test of whether your defect classes separate before anyone specifies optics or lighting.
TechnoLynx runs this check as the first step of AI engineering for manufacturing and automotive work: it agrees the defect taxonomy and acceptance thresholds with your inspectors and quality engineers, chooses texture-analysis techniques suited to the archived images, builds a lightweight pipeline, and reports whether the classes separate well enough to justify buying hardware. If separability holds up, the same engagement scopes the in-house model that later sits in the pilot alongside the vendor systems.
The taxonomy comes before any image is counted. A test of “acceptable wear versus rejectable pitting” means nothing until someone has written down where wear stops and pitting starts, in terms two inspectors would apply the same way. Landing AI’s manufacturing whitepaper (2020) makes a similar point from the vendor side: manufacturers need tools that help inspectors set clear defect definitions and reduce their ambiguity. MIL-HDBK-1823A (2009) asks that the feature a probability-of-detection study measures be explicitly defined and unambiguously measurable. How to write that taxonomy so it survives contact with a vision system is the subject of our article on feasibility assessment and defect taxonomy definition for automated optical inspection.
Why classical texture analysis is a reasonable first tool
For a surface-texture question, classical features are cheap to try and easy to inspect. A study in the journal Wear classified worn cast-iron surface images into severe and mild wear groups. A 2025 study in MDPI Eng classified four wear-particle textures (rough, striated, pitted and fatigued) from CCD microscopy images, with Laplacian and Canny edge-detection front ends reaching 97.9% and 98.9% classification accuracy.
That evidence comes from cast iron and wear particles, not from solder joints or pads on a populated board, so it shows that texture classes like pitting can be separable with classical methods, not that yours will be. Individual methods have blind spots: in the MVTec AD benchmark (CVPR 2019), a GMM-based texture model worked on grayscale only and failed to detect most colour-based defects, and no evaluated method performed consistently well across all object and texture classes. Judge separability per defect class, never as one blended score.
Sometimes the honest first question sits even earlier: whether the two conditions have different optical signatures at all (specular versus diffuse response, spectral band, angle dependence). That is an optics question to settle before image analysis, and a no there makes the dataset question moot.
What a positive feasibility result does not prove
Archived images were taken under whatever camera and lighting the inspectors used, not under the rig you will eventually buy, so a positive result is directional. Tensoria (2025) advises benchmarking on a representative subset of actual production images before committing to a training pipeline, because performance varies substantially between part types and defect morphologies. The fragility runs deeper than lighting: in a 2023 arXiv study of manufacturing defect detection, a classifier that was almost perfect on its validation set failed on a holdout set prepared the same way but acquired on another day. Landing AI (2020) notes that a proof of concept is generally an offline test under highly controlled conditions, and a long way from a deployable system.
How do we run a pilot batch with seeded defects through each candidate AOI system on our own board family?
Build one shared set of golden and seeded boards from your actual board family, let each candidate system program and debug on separate boards, then run every system once on the same unseen boards and score each defect class separately. That is the structure the National Physical Laboratory used when it benchmarked AOI systems on defect-free and seeded assemblies (NPL Report CMMT(A) 173). Participants received CAD data, a golden assembly and a bare board, had up to twenty further golden assemblies for up to eight hours of debugging, and were then benchmarked on ten assemblies they had not seen: five golden and five with defects. Defects were seeded before reflow where possible, so joint characteristics were not unduly affected, and by rework after reflow where they could not be. Flex took five test boards with 246 components each, at pitch sizes of 100, 150 and 200 µm, to the R&D labs of five AOI vendors.
Fine pitch is where the candidate systems separate
Detection gets harder as pitch gets finer. In the NPL study, all systems found missing components, but performance dropped when moving from 1.27 mm-pitch SOICs to 0.5 and 0.65 mm-pitch QFPs: no system rated high for insufficient solder on the QFPs, and only two rated high for bridges. In the Flex study of 03015 microchip components, 3D algorithms were unusable because of component reflection, so every machine fell back to 2D algorithms, and the machines differed in detecting misalignment, tombstoning and shorts. In both studies the systems separated on specific solder and placement defect classes, which is why the scoring has to be per class.
Keeping the conditions identical (as far as anyone can)
Here, “identical conditions” means the same unseen boards, the same debugging allowance, and production software and thresholds. MIL-HDBK-1823A says the system software and decision levels used in a demonstration should be those planned for production, and that the procedure should not be tuned to achieve a high score that misrepresents true capability. Full equality is hard to guarantee: in the Flex study, one vendor’s support engineer changed its machine from 19 µm per pixel to 10.5 µm per pixel to improve resolution, and each system ran its own algorithm with its own thresholds. Record those differences, because they belong in the comparison.
Seeded defects carry a bias of their own. MIL-HDBK-1823A warns that targets not generated the way they occur in the field can make a demonstration unnecessarily difficult or uselessly easy, and suggests comparing defect signatures with rejected hardware to show similarity. It also calls actual service-flawed hardware the ideal test where a representative selection exists, which is the case for using your own scrapped boards.
NPL found that programming skill requirements vary between systems and that high per-design programming overhead may restrict use in lower-volume production. Including an in-house model in the same run puts it under the same rules as the vendors, which keeps the build-versus-buy comparison honest.
What false-negative and false-positive rates should we require per defect class, and how do we measure them?
There is no universal acceptance figure; derive the requirement per defect class from what an escape costs you and what a false call costs you, then size the pilot so the measured rate can actually support that requirement. A 2023 arXiv study on manufacturing defect detection puts the trade-off plainly: a missed defect is usually the more serious error, but the false-positive rate generally decides whether automating inspection is economically viable, because it sets how much manual work is avoided.
Published targets give a sense of scale, not a standard. Flex set a false-call target below 5,000 PPM, and no machine met it with 2D algorithms. In the NPL benchmark, two systems achieved false-detect rates below 1% per joint, and none achieved zero. A Siemens team building an ML gate to screen AOI false calls treated a slip rate at or below 1% as a realistic shop-floor requirement.
Report escapes and false calls together
Flex reported defect escape percentage alongside false-call PPM, and found that escapes fell as false calls rose. Its 2D results, machine by machine, were:
| System (Flex, 2017) | Defect escaped % | False-call PPM |
|---|---|---|
| AOI 1 | 7.09% | 17,981 |
| AOI 2 | 35.22% | 17,886 |
| AOI 4 | 46.33% | 41,463 |
| AOI 5 | 2.09% | 30,081 |
| AOI 6 | 1.26% | 36,285 |
The two lowest escape rates came with false-call rates well above the 5,000 PPM target. Flex also ran an attribute Gage R&R on 78 components, with three operators each testing the board three times, and AOI 5 scored 100% both within appraisers and against the standard. Repeatability belongs in the comparison because a system that gives different verdicts on the same joint cannot be trusted at any threshold.
Worked example: how many sites a credible pilot needs
Assumptions: the systems give a hit/miss verdict per inspection site, each solder joint or placement is an independent site, and the requirement is a false-call rate below 1%.
- Flawed sites: MIL-HDBK-1823A suggests at least 60 targeted sites for a hit/miss system, as a minimum; at least 40 apply only where the system reports a quantitative response. Treat 60 as the floor per defect class you want a rate for.
- Unflawed sites: the handbook asks for at least three times as many unflawed as flawed sites, so 180 against 60. A site need not be a separate board, provided sites are independent.
- False-call opportunities: a false-positive rate below 1% needs at least 100 opportunities, and below 0.1% needs at least 1,000.
- Zero events still leave uncertainty. Hanley and Lippman-Hand describe the rule of three for zero numerators, and give 19% as the correct upper confidence limit after zero events in 14 trials. The rule puts the approximate 95% upper bound at 3/n after zero events in n trials. So a system that misses none of 60 seeded joints is only shown to miss fewer than about 3/60 = 5%, and zero false calls in 100 clean opportunities only shows a rate below about 3/100 = 3%. To claim below 1% from zero false calls you need roughly 3/0.01 = 300 clean opportunities.
- Do not inflate counts with re-runs: repeated inspections of the same specimens tell you about the inspection’s consistency, not about more specimens, so they do not increase the effective sample size.
- When some defects are found, use an exact binomial interval rather than a normal approximation; the NIST/SEMATECH e-Handbook’s example of 4 defectives in 20 units gives a 90% confidence interval of 0.071 to 0.400.
On a populated board, unflawed opportunities pile up quickly because every good joint counts; the bottleneck is seeding enough flawed sites per class. Small sets also distort rankings: NPL’s own confidence ratings could rest on a single defect occurrence.
A pilot that passes can still fail on later production data. In a Siemens case study on AOI false-call reduction, models that met their targets on a random test split exceeded the slip-rate target on the first chronological evaluation slice, and the authors concluded the models would fail from the first moment in production. Their recommendation is to check for temporal drift and at least use the chronologically last data as the test set.
Who can run several vendors’ AOI systems against our own scrapped boards under identical conditions?
TechnoLynx can design and run that comparison as the independent party: it defines the defect taxonomy and per-class acceptance thresholds with your team, specifies the golden and seeded board set and the site counts the statistics need, sets the common debugging and threshold rules each vendor works under, and scores every system on per-class escape rate, false-call rate and repeatability. Where an in-house model is a candidate, TechnoLynx builds it as Computer Vision R&D work and enters it under the same rules. The deliverable is the feasibility findings and pilot results, written up to inform your full-line AOI specification.
That engagement does not suit everyone. TechnoLynx does not sell AOI machines, so the candidate systems still come from their vendors, run on their hardware, at your site or in their labs. If you need an accredited test report for a customer or regulator, a test house that issues one is the right route. If only one vendor is in contention, that vendor’s own evaluation on your boards, run under the no-tuning and unseen-board rules above, may be enough.
How many labelled images do we need before anyone can say it is feasible?
Far fewer than a production build: the studies below worked from 5 to 20 examples per class up to a few hundred instances, and the count depends on how many classes you have and how variable they look, not on a universal number.
TechnoLynx builds from that small set as Computer Vision R&D work when feasibility holds, scoping the in-house model for the pilot.
The literature supports small numbers at this stage. A 2025 study of few-shot adaptation for PCB defect inspection evaluated foundation models with only 5, 10 or 20 labelled examples per class. The 2023 arXiv defect-detection study cites a transfer-learning approach that reached over 80% accuracy with a few dozen labelled data points, and reports training a performant detector from a few hundred instances. Build-stage counts are larger: Tensoria (2025) gives 300 to 800 labelled images per class for pass/fail classification and 1,000 to 3,000 per defect type for object detection. Those are the figures for training a deployable model, not for answering whether one is possible.
Diversity matters, not just count. The same arXiv study found that a classifier trained on similar defects across diverse flat metal parts kept a 0.92 AUC from validation to holdout, while one trained on uniform mending plates failed on a later holdout, and clustering showed some data mattered less for performance. A hundred images of the same board under the same light tell you less than fifty spread across lots, shifts and boards.
Can we start with a phone camera, or do we need professional equipment?
A phone camera or the inspectors’ archive is enough to ask whether the defect visibly differs from acceptable variation, but not to predict how a production system will perform. Tensoria’s readiness checklist, by contrast, asks for at least 200 images of good parts under production lighting before a feasibility assessment. We read them as two stages: ad hoc images answer the visibility question cheaply, and production-like imaging comes in before anyone forecasts detection rates. Either way, the images must resemble the defects, lighting and surface texture of the real use case, or the verdict will not transfer.
Building the first labelled set with your current inspectors
Your inspectors are the cheapest labelling resource you have, but their labels need checking against each other. Landing AI (2020) cites studies putting manual inspection error rates between 20% and 30%. Having two inspectors label the same images, and resolving disagreements against the written taxonomy, turns the feasibility set into the first draft of a reproducible ground-truth standard.
If real defects are too rare to collect, start from good parts. Tensoria (2025) suggests unsupervised anomaly detection when the defect rate is below 1%, using only normal-part images, with the trade-off that it does not name the defect type. The MVTec AD benchmark is built on the same idea, training only on defect-free images and testing on 73 manually generated defect types.
When is a team ready to move from a feasibility check to a full independent pilot?
Move on when the feasibility check has done its job and the remaining uncertainty is about systems, not about whether the defect is detectable. In practice that means:
- each defect class separated from acceptable variation on images that reasonably resemble production imaging
- a written taxonomy and acceptance criteria that inspectors apply consistently
- a shortlist of candidate systems, with or without an in-house model
- access to scrapped and seeded boards in the site counts the pilot statistics need
- per-class targets for escapes and false calls, derived from what each error costs
MIL-HDBK-1823A draws the same line between stages: preliminary tests on a small training set may be run, but they cannot substitute for the full demonstration.
The pilot’s verdict is also dated the day it is run. In the Siemens study, models that met their success criteria on a test split failed on a later temporal split. The decision that lasts is the evaluation protocol itself: the taxonomy, the seeded board set and the per-class scoring. Kept and re-run when the board family changes, it turns a one-off vendor choice into a check you can repeat.
FAQ
What does false positive vs false negative mean in vision inspection?
A false negative is a defective part the system passes, which can reach the customer; a false positive is a good part flagged for manual review. The false-negative rate governs risk, while the false-positive rate largely decides whether automating inspection pays.
