When the people who know your defects are about to retire, a large photo-labelling campaign is the wrong place to start. Start with two cheaper steps. First, a short feasibility check that the defect is physically visible to a camera and lighting setup you can afford. Second, structured sessions in which several inspectors label the same images independently, so their agreement can be measured before any of it becomes ground truth. Inspector judgement is measurably inconsistent. In a study by Aust and Pons, 50 aviation maintenance practitioners judged photographs of 26 engine blades twice each. On average they agreed with their own earlier decision 82.5% of the time and matched the expert ground truth 67.7% of the time, and the whole group agreed on only 15.4% of the blades.
Choose the model after that. In any computer vision engagement, the first things to check are the quality and quantity of the data, whether you can get at the process that produces more of it, and what accuracy the business actually expects. Accuracy close to 100% is almost never a realistic target, so the workflow needs fault tolerance and room for human review.
Can off-the-shelf vision inspection systems see your defect?
Sometimes. The way to find out is a scoped check of the defect’s physical signature against the resolution and lighting you can realistically get, done before you spend anything on a pilot or hardware.
TechnoLynx runs a feasibility-first defect taxonomy study under its Computer Vision R&D offer, ending in a go/no-go verdict before you commit to a full vision pilot.
The camera alone does not decide the answer. Cognex, a machine vision vendor, describes poor lighting as the most common cause of poor machine vision performance and says no camera or software can make up for inadequate lighting. Cognex also says good lighting maximises contrast on the features of interest, stays consistent as parts vary, and has no single best setup for every application. The authors of an aluminium-casting defect dataset paper make a related point about 2D imaging. Earlier studies’ optics, it says, are highly sensitive to lighting angle, and so to the optical engineer’s expertise, and flat images struggle to tell a shadow from a stain.
The practical consequence is that “can a camera tell acceptable edge dross from a burn mark?” is mostly a question about lighting and optics, and only partly about the model. The same goes for “can a rig replace our black-light check on stitching and foam density?”. If the two classes differ in surface relief, raking light may separate them. If they differ in fluorescence, you need an equivalent of the UV source. If they differ in neither under any lighting you can achieve, no network will separate them reliably.
Will our existing cameras resolve the defect? A worked example of the pixel arithmetic
The test is a few lines of arithmetic, and you should do it before buying a new camera. Cognex advises sizing resolution from the smallest feature and the field of view rather than from megapixels. It recommends three to five pixels across the smallest defect you need to catch, and ten or more for dimensional measurement. A technical guide from the lens maker Commonlands (updated May 2026) gives three to five pixels for high-contrast presence or absence under controlled lighting, and eight to fifteen pixels for low-contrast or textured defects.
| Step | Formula | Worked values |
|---|---|---|
| Spatial resolution needed | smallest feature ÷ pixels per feature | 0.25 mm ÷ 4 px = 0.0625 mm/px (Cognex example) |
| Pixels needed across the field of view | FOV ÷ spatial resolution | 40 mm ÷ 0.0625 mm/px = 640 px |
| What an existing camera delivers | FOV ÷ pixels across FOV | 100 mm ÷ 2592 px ≈ 0.039 mm/px (Commonlands example) |
| Coverage of a 0.2 mm defect on that camera | defect size ÷ mm per pixel | 0.2 ÷ 0.039 ≈ 5 px: workable for a high-contrast defect, short of the 8-15 px a low-contrast one needs |
| Exposure limit on a moving line | (0.5 × mm/px) ÷ line speed | about 120 µs at 200 mm/s and 0.049 mm/px (Commonlands example) |
Thermal cameras go through the same arithmetic, usually with far fewer pixels to work with. Before asking whether thermal alone is enough, ask whether each defect class has a thermal signature at all. An overheating connector does. Dust and a frayed sheath are mostly visual, which is where adding an RGB camera earns its place. Phone photos are a fair first filter for whether the signature exists. They do not replace a proper resolution and lighting study before the final buying decision.
What evidence lets you call vision inspection feasible before a pilot?
Feasibility is a claim about separation: defect signals must sit reliably apart from the variation in good parts. The evidence has to show that separation across the real spread of your process, not on a few hand-picked images. Andrew Ng, interviewed by IEEE Spectrum, describes Landing AI’s process as starting with a conversation and a look at a few images to check the problem is feasible with computer vision. He also says that 50 well-engineered examples can be enough to build something valuable, such as a defect-inspection system.
To make a small-sample claim defensible, NASA’s Limited Sample POD method is a useful template. It sets a probability-of-detection limit from flawed specimens and a probability-of-false-call limit from unflawed ones, and judges whether a decision threshold can sit between the two. NASA suggests at least 40 unflawed measurements that exercise every source of signal variability, with more of them given to the sources that vary most, and treats 10 flawed specimens as the point below which extra statistical review is needed. The method assumes signal responses are Normally distributed. If you have any reason or data to doubt that, the method is not valid. It also asks for blind testing where practical, so the inspector cannot improve a signal at a flaw location they already know about.
| Evidence you hold | Verdict |
|---|---|
| Sample images show the defect signature clearly apart from acceptable variation, pixel coverage has margin, and samples span heats, dies, lots and lighting | Feasibility is plausible: run a scoped pilot |
| Signature overlaps acceptable variation under the optics you have | Not yet feasible: lighting and optics work comes before any pilot |
| Signature looks separable, but only a handful of defects or one material batch is covered | Not yet feasible: widen the sample before deciding |
| Inspectors cannot agree on the historical labels | Not yet feasible as labelled: fix the taxonomy first |
The heat and die question in the last two rows matters most for forgings and castings. A sample drawn from one heat says nothing about the next one.
Should we spend this year’s budget on a camera system or wait?
Put plainly: buy when the feasibility verdict is positive, spend a smaller amount on an optics study when the verdict is “not yet”, and stay with manual inspection, improved, when the defect cannot be seen at all. Staying with the eye has a cost too. The Sandia National Laboratories 2012 literature review on visual inspection notes that error rates of 20% to 30% are frequently quoted across inspection tasks. The same review cites a finding that defect detection can deteriorate by up to 40% within 30 minutes.
The review also describes ways to improve manual inspection. The Sandia review reports that comparison standards turn an absolute judgement into a comparative one and improve detection. It also reports that having two inspectors check every item, with a part rejected only when both reject it, gave the best detectability of the methods studied. Its principal recommendation against that decline in vigilance is to limit inspection stints to 30 minutes. The useful benchmark is your own inspectors measured against an adjudicated reference set. Three cautions apply. In Scortex’s study, a model trained on limited real imagery lost much of its accuracy once test conditions shifted. Phone imagery screens but does not decide. And “feasible” can still mean specialised lighting or a better camera than you expected.
A defect taxonomy built to survive pilot-scale data
A taxonomy that survives scale-up is written against real, noisy imagery from the line. It keeps the number of classes small, states its exclusions and severities, and comes with written triggers for when it must be reviewed. TechnoLynx frames the feasibility question with you and defines the defect classes and severities. It then sets a pilot scope and imagery source cheap enough to give a believable signal, runs a baseline detector on representative images, and checks accuracy and false-reject rate against real line data before the go/no-go decision.
Can inspectors’ intuition be written down, and who decides when two disagree?
An inspector’s intuition is hard to capture by asking for the rules. The Sandia review describes experts recognising defects at a skill-based level that is the least effortful and happens largely unconsciously, while novices work through rules and explicit reasoning. Asking a 30-year inspector to dictate rules therefore tends to produce a tidier version of the job than the one they actually do. Labelling sessions work better. Each inspector labels the same images independently, and then the group talks through the images they disagreed on. Those images are where the tacit rules surface.
Keep the taxonomy small. The Sandia review reports that inspectors appear to search for a subset of about five defect types, and cites six as a practical maximum. The review also finds that without clear defect definitions, inspectors form personal criteria that drift over time, so that parts once rejected are later accepted. Include an explicit “no defect / natural variation” class. The authors of a 2025 surgical-instrument dataset added a Miscellaneous class of non-defective instruments and background objects, which helped their model tell true defects from natural variation in material texture.
Then measure agreement against thresholds. A glossary from Claru, an annotation provider, gives a Cohen’s kappa of 0.70 as the usual minimum for production classification labels, an IoU of 0.75 as good for boxes and masks, and 0.85 as excellent. It calls overlap on 10% to 20% of items standard practice, with high-disagreement items going to a senior adjudicator. The MSDD casting dataset shows the mechanism in practice. Two trained inspectors annotated the images independently. Where their boxes overlapped at an IoU above 0.8, the boxes were averaged, and anything below went to a quality supervisor who made the final call. Low agreement is information as well. Claru reads it as evidence that the labelling guidelines are ambiguous, and notes that a 0.80 kappa caps how well any model can be shown to perform on those labels.
Bring in enough inspectors. Aust and Pons recommend at least five appraisers, preferably ten. In their data, two appraisers reached 65.4% reproducibility where all 50 together reached only 15.4%. The adjudicator should be one named person with quality authority, not whoever is free that day. Where they cannot settle a label from the image, the physical part should decide, as it did for the expert ground truth in the blade study, which was inspected under optimal lighting with magnification. Some regions are genuinely ambiguous. A 2023 paper on surface-defect labels proposes treating them as a third, uncertain category alongside normal and abnormal, instead of forcing a choice.
Do vendor sample images represent what our line will produce?
Not necessarily, and you can test it cheaply. Scortex’s study of simulated training data is a sobering reference. A model trained only on real images scored an mAP of 0.913 on its own test set, then fell to roughly 0.4-0.6 depending on whether the camera and lighting setup, part reference or defects changed. Most of the loss came from false positives. A feasibility finding built only on vendor samples can therefore be optimistic. Before go-live, get even a small batch of real line images and re-run the taxonomy against it. The VDI/VDE/VDMA 2632 Part 3 guideline (published October 2017) discusses accepting classifying vision systems against sample catalogues and against products from current production, and weighs the advantages and disadvantages of each. When the pilot turns up a defect type nobody listed, the fallback is to add the class, re-measure agreement on it and extend the reference set. That keeps the feasibility case intact instead of reopening it.
Keeping the taxonomy valid after scale-up, and who owns re-qualification
The taxonomy stays valid when changes trigger review on purpose, and stops being valid when nobody is watching. One machine vision validation service provider states that any camera, lens or lighting change, software update or line relocation that could affect output needs a documented change-impact assessment and partial or full revalidation. It names ISPE’s GAMP 5 as the usual risk-based framework for that. For process drift such as a new steel heat or die wear, NASA suggests monitoring signals from regions assumed to be unflawed over time, because changes there reveal shifts in instruments, parts or inspector performance. An FDA-hosted presentation on visual inspection of injectable products recommends requalifying at least annually, and qualifying each product family separately. Change control should belong to the quality function, with the vision team as a required reviewer, because the people who define a reject should be the ones who sign off on what the camera now calls a reject. The mechanics of keeping the taxonomy and pipeline aligned over years have their own treatment in keeping defect taxonomies and inspection pipelines robust through change.
Do we need a site-by-site qualification pass?
Yes, wherever cameras, lighting, product families or line layout differ, even when the taxonomy is shared. The same provider counts line relocation as a revalidation trigger, and describes performance qualification as running under real or simulated production conditions on a statistically significant sample. For automotive production acceptance it cites Cpk ≥ 1.33 as the common minimum. The efficient pattern is one central taxonomy and reference set, fixed at the feasibility and taxonomy scoping step, with a short per-site performance qualification against local parts and its own repeatability validation.
Who can turn our inspectors’ judgement into a dataset we can test feasibility on?
TechnoLynx runs this as a feasibility-first engagement. It works out whether cameras can see your defects, turns your inspectors’ calls into a written taxonomy with measured, adjudicated agreement, and tests a small labelled pilot set before you buy hardware or fund a full pilot. When choosing a partner, ask how they would test separability and agreement before training anything. TechnoLynx delivers this work under its Computer Vision R&D offer, as part of its wider AI engineering for manufacturing and automotive.
The engagement covers feasibility, taxonomy and a pilot model, not conveyors, rejection mechanics or line integration. A machine-vision integrator is the more direct route for that, once feasibility is settled. Regulated sign-off (for example, a pharmaceutical or aerospace qualification) stays with your quality authority. TechnoLynx’s part is validating accuracy and false-reject rate against real data. And if your inspectors already work from an agreed reference library and the defect is plainly visible, you may not need a feasibility study at all.
Bootstrapping labels with weak supervision and synthetic defects
Synthetic defects and weak supervision can speed up coverage. They cannot replace a manually labelled, adjudicated reference set, and that reference set has to come first, because it is the only thing you can check the bootstrapped labels against. Scortex’s results show why synthetic data cannot stand alone. Simulated images alone were not sufficient, but combining them with real images improved generalisation to unseen distributions, mainly by cutting false positives. In their words the benefit reached roughly “one part reference away”. Speed is the attraction: Scortex generated 9,413 simulated images in three days. The limit is realism. The surgical-instrument dataset authors found that synthetic data did not reproduce real-world material texture, lighting variation and surface wear.
Weak supervision follows the same logic. Let a classical edge or ridge detector, or an early model trained on the seed set, propose candidate labels, and have people correct them, starting with the ambiguous boundary cases. Review effort goes furthest when it is targeted. Northcutt and colleagues (NeurIPS 2021) found that images flagged by confident learning were 2.6 times as likely to be mislabelled as unflagged ones, and they recommend that kind of prioritisation. Ng describes the same move: tools that point you to the inconsistently labelled subset so you can relabel it quickly.
QA keeps the final labels trustworthy. The surgical-instrument team used initial labelling, cross-validation by a second group, and an audit of a random 10% by senior experts. Spend the most care on the test set. Northcutt and colleagues estimate an average of at least 3.3% label errors across ten widely used benchmark test sets, and advise curating test labels to the highest quality even when the budget only allows rougher training labels. The bias risk is specific. A heuristic labeller or a synthetic renderer teaches the model what the generator thinks a defect looks like, and that error may not show until deployment.
A 2023 paper on power-line inspection describes broken strands as thin objects whose features blur at distance, and real samples as rare because damage is repaired as soon as it is found. Label them with polylines or fine masks at high zoom, not coarse boxes. Capture across the real spread of angles and glare, because the easy viewpoints will dominate otherwise. Measure agreement per feature type with a stated tolerance. A one-pixel offset on a two-pixel-wide crack wrecks IoU even when both annotators found the same crack. One public pitting dataset ships labelme annotations as JSON, convertible to VOC and COCO formats.
When there is no line to photograph: open defect datasets
Open datasets are a sound starting set for teaching or early prototyping, and a weak stand-in for your own parts. TechnoLynx can define the defect classes the work needs, assemble and label a small pilot set from open or donated images, and build a baseline classifier that shows whether the taxonomy holds before anyone invests further. The Charmve Surface-Defect-Detection repository collects public sets, including magnetic tile images with pixel-level ground truth, the RSDDs rail surface set labelled by professional observers, and Baidu’s Gear Inspection Dataset with 28,575 annotations on 2,000 images. Coverage is patchy. The MSDD authors note that many studies never release their data, because of limited samples, class imbalance and annotation cost. Where a local employer will share anonymised samples, prefer them to open sets.
Building ground truth before any model exists
The same principle holds outside the factory. Build ground truth from trained reviewers applying a written confirmation protocol, with agreement checks, before any model is involved. For scan-avoidance at retail tills, that means having loss-prevention staff review sampled footage and confirm incidents against stated criteria, with a share double-reviewed. Historical stop data is useful as a check for bias, not as the only source of truth, because it records only the incidents someone caught. Use staged incidents to supplement real ones, not replace them. How many confirmed incidents you need depends on the incident rate and on how much stores differ from one another, so the count should come from a power calculation, not a round number.
What to capture before the inspectors leave
Before the last experienced inspector goes, the asset worth securing is the reference library: the boundary samples, the images the group disagreed on and how each was settled, and the physical parts behind them. That is the material that turns a personal standard into a shared one. The FDA-hosted presentation lists a defect reference library and defect samples as part of inspector training. It describes qualification kits of 300-500 units containing 30-50 defect examples. The same presentation notes that inspection is probabilistic and cannot be relied on to remove every defect.
FAQ
What type of lighting is best for visual inspection?
No single setup suits every application. Cognex describes good lighting as maximising contrast on the features of interest, minimising it elsewhere, and staying consistent as parts vary.
What is the difference between false acceptance and false rejection?
A false acceptance passes a defective part, and a false rejection scraps or reworks a good one. Aust and Pons argue the two should be tracked separately, because in aviation maintenance a missed defect is a safety problem while a false reject only costs money.
