A small, time-boxed feasibility pilot tells you whether automated visual inspection will work on your parts before capital is committed. It needs a written defect taxonomy, go/no-go thresholds agreed before any image is taken, a labelled sample set of your own conforming, defective and borderline parts, and a borrowed or candidate camera and lighting rig. A feasibility study in this sense is a bounded, time-limited investigation of whether an imaging solution can meet a defined specification, usually expressed as detection rate, accuracy, cycle time or false-positive and false-negative tolerance. Robustness to lighting and sensor variation belongs inside the same pilot as a test it must pass, not as a later phase, because a vision system inspection result that only holds under one bench lamp tells you very little about the line.
The order matters as much as the size. Check the data first: its quality, how many real defects you actually have, and access to the processes that generate more. Only then compare models.
How small can a vision inspection pilot be and still give a real answer?
A pilot exists to answer one question before capital is committed: can a camera-based system reliably solve your defined inspection problem under real production conditions? Building a working line system is not its job. Its output is a feasibility verdict against the agreed specification.
LynxBenchAI benchmarks throughput, latency, memory and energy on candidate hardware with stock PyTorch when the next question is which compute to put on the line, and the first step is to benchmark your pilot model on those candidate devices before a second-stage proof of concept on the chosen device.
TechnoLynx agrees the defect taxonomy and what “feasible” means with your quality and engineering staff, sizes the sample set and rig to the risk you want to retire, runs it against your labelled parts, and hands back a go/no-go recommendation with the data a full budget request needs.
On sample size, Machine Vision Authority’s feasibility guidance cites an AIA recommendation of at least 30 representative samples per defect class for preliminary feasibility. It also notes that deep learning evaluation commonly needs 200 to 500 labelled images per class before training. A pilot dataset guide published by The Hunger Engine puts it differently: several hundred labelled defect examples per class can be enough for a simple, well-fixtured inspection with good lighting, and more variation pushes that into the thousands. Treat both as guidance rather than law. The honest minimum depends on how visually close your classes are and how much variation the parts carry. The sample set should be fixed in advance and should include conforming parts, known-defective parts and edge cases.
Classical image processing (thresholding, blob analysis, edge detection) is worth testing first, because it is faster to iterate and easier to explain to process engineers. Deep learning comes in only if it fails the specification on the same labelled set.
Before buying any hardware at all
Whenever the question is binary feasibility rather than long-term throughput, a vendor demo or rented camera and lights is the better choice than buying equipment. Results hold only under the conditions tested, and any production deviation from them warrants re-validation.
Before committing capital to line integration or a GPU rack
A camera and lighting pilot leaves open what hardware upgrade, if any, the move to line integration needs. If the first pilot passes, the next step is to specify that hardware before line integration or full procurement.
Before a full labelling and engineering effort
Labelling quality is the cheapest thing to check and the most expensive to discover late. The Hunger Engine guide recommends having several labellers annotate 5 to 10 percent of the dataset and targeting Cohen’s kappa above 0.6 for classification and IoU above 0.7 for bounding boxes, revising the labelling guide when agreement is low. Expectations need the same scrutiny: plan the workflow for fault tolerance and a human in the loop.
Before fine-tuning a YOLO detector on the full taxonomy
Where classes overlap visually, such as tombstoning against a slightly misaligned but acceptable component, pilot on that reduced pair first and check that it can be separated at all. In published tyre X-ray work, model accuracy fell as the number of defect types in the taxonomy grew. Building that taxonomy is covered in our piece on feasibility-stage defect taxonomy and site suitability assessment.
Reading the pilot outcome
| Outcome | What it means | What happens next |
|---|---|---|
| Clean success | The algorithm met specification on the sample set with no identified environmental dependencies | Scope the second-stage proof of concept on target hardware, carrying the documented performance envelope forward |
| Conditional success | The specification was met only with a specific lighting configuration, fixture or preprocessing step | Budget the extra engineering that the condition implies before production |
| No-go | The classes could not be separated reliably across the real variation | Keep manual inspection or revisit the taxonomy and imaging approach; this is a legitimate result, not a failed pilot |
Is a handful of confirmed defects on file enough to start?
When confirmed defects are rare and manual inspection only samples part of the output, the labelled set on file is almost certainly too small to carry a feasibility study on its own. Sampled inspection also undercounts how often a defect really occurs, because unsampled units never get a label.
One tyre study shows how slowly rare defects accumulate. Researchers working with X-ray images from Pirelli’s tyre factory in Türkiye report collecting for more than a year because faulty tyres are rare in production. They ended with 3,366 faulty-tyre images against 20,000 qualified ones, all labelled by factory quality inspectors. That is roughly six qualified images for every faulty one (20,000 ÷ 3,366 ≈ 5.9).
The first step is an audit: fix what counts as a confirmed example under the taxonomy, then count. If the count falls short of the per-class minimum, enlarge the set by targeted collection or synthetic data:
- Collect targeted defect samples when defects are rare: The Hunger Engine guide recommends actively capturing and labelling every defect for a period rather than relying on random sampling.
- Add synthetic or augmented images when real defects are impractical, dangerous or too slow to produce. Validate them against whatever real examples exist.
In a 2022 study of car and person detection, not factory defects, training a COCO-pretrained YOLOv3 detector on mixed synthetic and real data cut the real-data requirement by up to 70% without losing detection performance, and real-data shares between 5% and 20% gave the largest saving. The saving varied sharply by class: the car class still needed 64% of the real data, while the person class needed only 20%. The same authors note that the domain gap stops detectors trained only on synthetic data from matching those trained on the same number of real examples. Whether those ratios carry over to defect imagery is a judgement, not a finding, so we would treat synthetic data as a supplement and keep real defects in every test set. Augmentation also has limits: small rotations and brightness or contrast changes help, but transforms that create impossible part views or unrealistic defects do not.
Treat whatever comes out of this as a provisional feasibility-stage dataset. It is deliberately held to a lower bar than production validation, and the pilot report should say so.
What miss rate and false-reject rate should the pilot hit before we approve investment?
Agree the thresholds before data collection, weighing the cost of a missed defect against a false reject, and check them against your manual inspection baseline. TechnoLynx sets those thresholds with your team, chooses an imaging approach suited to the class pair you need to separate, builds the small labelled pilot pipeline, and checks detection accuracy and false-reject rate against your manual baseline across real batch variation, ending in an honest feasibility verdict.
Published guidance gives these thresholds as examples. Machine Vision Authority cites a threshold such as a detection rate of at least 99.5% at a false-positive rate of at most 0.5%, and notes that these numbers drive every downstream hardware and software decision. The Hunger Engine guide offers, as an example decision rule for scaling, recall of at least 0.90 and precision of at least 0.85 on the test set, plus an operator A/B validation over one shift. It also recommends recall as the main metric for defect capture and precision for keeping false rejects down, with thresholds tied to business outcomes such as missed defects per 10,000 parts.
Fix the labelling rule at the acceptance limit
The tyre X-ray work shows where its model fell short. The best model detected significant defects well but struggled with small variations close to acceptable limits. Pirelli’s inspectors explained that acceptable defect sizes differ by customer, so a tyre labelled acceptable could be rejected by a race-tyre customer with a stricter standard. The authors describe the precision-recall threshold as hard to set, and note that tuning for sensitivity to tiny defects reduces misses but raises false positives. The lesson for a pilot is to agree the labelling rule for borderline parts before anyone labels, because no model can learn a boundary the labels do not hold consistently.
Spread the sample across the variation that matters
If the classes are, say, structural cracks against harmless bubbles in rubber, validation has to span compound and curing variation. A pilot trained and tested on one batch says nothing about the others, and an undersampled variation inflates the accuracy you report. Polarised-light imaging is one candidate technique for that kind of class pair. The right geometry is part-specific, though, and it gets settled in the lighting trial rather than on paper.
The Hunger Engine guide also recommends running the model in parallel with human inspectors for one shift and logging the disagreements for root-cause analysis. A formal ROI comparison between a vision pilot and continued manual spot checks is a separate decision that builds on these pilot numbers. Likewise, the procedure for re-tuning thresholds when score distributions shift between suppliers belongs to operation, not feasibility.
Is our current shop lighting good enough to trial automated inspection?
Testing lighting geometry against your sample parts for sufficient defect contrast answers that question. TechnoLynx’s Computer Vision R&D work covers that trial: selecting candidate lighting, polariser and camera configurations for your defects and part geometry, building the trial rig, and measuring detection, false-reject rate and repeatability under current and proposed lighting.
Can imaging be made repeatable enough at all?
Advanced Illumination’s practical guide sets three acceptance criteria for machine vision lighting: maximise contrast on the features of interest, minimise contrast elsewhere, and provide a measure of robustness. The guide notes that lighting meeting only the first two criteria can work, but only when part size, shape, orientation, placement and ambient light stay consistent. Surface reflectivity, geometry, texture, composition and colour together decide how much light reaches the camera. The same guide’s advice on method is blunt: there is often no substitute for testing two or three light types and techniques on the bench first, then on the actual floor.
Living with daylight and overhead lights
Ambient light can have a large effect on inspection quality and consistency, especially with white light. The usual culprits are overhead factory lights, sunlight and stray light from neighbouring inspection stations. The guide lists three active countermeasures: high-power strobing with short pulses, physical enclosures, and pass filters. Polarisers help where specular glare hides a defect. The catch is that dual polarisers can cut available light by up to 2½ f-stops, which can hurt high-speed, light-starved inspections. Where daylight varies, capture images across shifts so the model learns real conditions or you can scope the hardware fix, and document exposure, white balance and camera settings.
A short trial covers only the conditions present while it runs, so it reduces risk without proving line robustness. Longer validation of repeatability of vision-based defect detection before production is the next step. Moving the model onto an edge board or embedded target on the line is the work of the AI Porting & Deployment Pack.
Can a bench stress test catch the big gaps before the pilot is scoped?
Yes, if it is sized for that job. A small internal stress test, built from existing images plus targeted augmentation, can expose gross failures at the lighting and occlusion edges of the operating envelope before you commit pilot time with a line owner or customer. The same logic drives corruption benchmarks for autonomous driving, which probe image distortions and weather. For inspection, the equivalents are glare, shadowing, dim stations and partial occlusion by fixtures.
Public data helps here. MVTec AD 2, an industrial anomaly detection dataset of more than 8,000 high-resolution images across eight scenarios, includes test scenarios with changed lighting to assess robustness under real-world distribution shift. It also covers dark-field and back-light illumination, transparent and overlapping objects, and extremely small defects.
As for how much augmentation is enough, aim for enough samples per edge-case bucket to reveal a degradation trend, not tight confidence intervals. No fixed count applies until a risk threshold has been defined. Define the operating envelope first, so the test has a boundary to probe. Passing it also guarantees nothing about field validation later, and synthetic perturbations may not reproduce real optical effects faithfully.
What robustness checks should a camera-based inspection model have beyond drift monitoring?
Drift monitoring tells you the input distribution has moved. A robustness regression suite tells you whether the model still works when lighting, focus, noise or occlusion change, and the two answer different questions. TechnoLynx builds that suite as part of the Production AI Monitoring Harness, listing the failure modes relevant to your line, deciding which checks generic tooling covers, building perturbed test sets and custom checks, confirming they catch known regressions on past model versions, and wiring them into validation alongside drift monitoring.
The ImageNet-C benchmark, published at ICLR 2019, gives a design worth borrowing. It applies 15 corruption types from the noise, blur, weather and digital families, each at five severity levels, for 75 distinct corruptions. It measures corruption robustness as average-case performance across those corruptions, unlike adversarial robustness, which measures worst-case performance under small perturbations. Severity matters because the same corruption ranges from negligible to destructive.
The stakes are measured. Standard object detectors fell to 30 to 60 percent of their original performance on corrupted images in the Pascal-C, COCO-C and Cityscapes-C benchmarks. One finding cuts against intuition: in the ImageNet-C work, adding a non-local-means denoising step made overall corruption robustness worse, because it stripped subtle detail from images that had no noise. Targeted image restoration therefore proved harmful for overall robustness in that work.
A perturbation suite mapped to inspection causes
| Corruption family | Inspection cause it stands in for | What to check |
|---|---|---|
| Gaussian and shot noise | Low light; shot noise comes from the discrete nature of light itself | Recall on the smallest defect class as noise rises |
| Brightness and contrast | Brightness varies with daylight; contrast with lighting and object colour | False-reject rate across ambient-light levels |
| Defocus and motion blur | Out-of-focus optics; fast camera or part motion | Separation of near-limit defects from acceptable parts |
| JPEG and pixelation | Lossy compression; upsampled low-resolution images | Stability when the camera or transport pipeline changes |
Run each row at graded severities and report the curve, not a single number.
Generic tooling covers part of this. Deepchecks ships pre-implemented checks for weak performance segments, distribution drift and leakage, and data integrity issues such as conflicting labels, and it allows custom checks that run during research or automatically in CI. Class-specific perturbations and novel-defect probes for your line go in as custom checks, which Deepchecks supports. Defects nobody anticipated cannot be fully covered by any suite defined in advance, and a growing custom suite adds maintenance on top of drift monitoring.
Testing a vendor’s claims on your own parts
A vendor’s accuracy figure means something only on your parts, under your lighting, scored against your thresholds. Fix the labelled sample set in advance, including conforming, defective and edge-case parts. Have every candidate evaluated against that same labelled set, the way a classical method and a deep learning model would be compared within a single pilot. Then run the perturbation suite above on each candidate, because standard object detectors tested on corrupted benchmark images lost much of their performance.
Ask each vendor for a performance envelope. Machine Vision Authority describes a go recommendation as one that comes with the specific camera model, lens, lighting geometry and algorithm configuration that met the specification, plus the environmental conditions of the test. Any deviation in production then counts as a change that warrants re-validation, consistent with ISO 9001:2015 clause 8.5.1. A claim without an envelope cannot be compared with another vendor’s or reproduced on your line.
Who can run a vision inspection feasibility pilot on our own parts?
TechnoLynx runs bounded feasibility pilots on your own parts for manufacturing and automotive lines as part of its Computer Vision R&D work, starting by agreeing the defect taxonomy and go/no-go thresholds with your team. On building in-house versus bringing in a specialist: a team that already owns imaging, labelling and model evaluation can run the pilot itself, while a specialist adds most when the risk sits in ambiguous classes, rare defects or uncontrolled lighting. When choosing a partner, ask whether they will set thresholds before imaging and test every option on the same blind set.
TechnoLynx’s catalogue does not include installing and maintaining a turnkey inspection cell. Machine vision integrators and camera vendors serve that need directly, and their own feasibility studies are a fair route when you are already committed to their hardware. A team with strong in-house computer vision staff and clean, plentiful data may need only a review of its pilot design rather than a full engagement.
Safety-critical inspection needs a paper trail the pilot can feed
Where a missed defect can hurt someone, the pilot’s evidence has to be reviewable by people who did not run it. We are not aware of any standard that certifies an AI-based optical inspection system outright. The defensible move is therefore to make the feasibility evidence traceable and to have it reviewed independently, rather than waiting for a certificate.
The pilot already produces records that review can use. The written taxonomy and labelling rule for borderline parts, the agreed thresholds and their rationale, the inter-rater agreement results, the corruption-and-severity curves, and the performance envelope together make a failure-mode catalogue: what was tested, where the model degraded, and under which conditions the result holds. The envelope also anchors change control, since Machine Vision Authority treats any production deviation from it as grounds for re-validation. What remains uncertain is everything outside the envelope, such as defect types nobody has seen, seasonal light the trial never caught, and drift after the line changes. The safety case should name those limits explicitly rather than let a strong pilot score imply they are covered.
