To prove a machine vision system can stand in for manual inspection, run a designed Gauge R&R study on production-like parts, appraisers, shifts and fixture states, and use attribute agreement analysis where the station makes pass/fail calls. Judge the study against thresholds fixed before any data arrives. For measurements that means %GRR under 10% plus at least 5 distinct categories (ndc). For pass/fail decisions it means a kappa of at least 0.75, with 0.90 preferred. Then use a paired method comparison to show that the station detects defects at least as well as the inspection it replaces.

TechnoLynx handles that replacement as a validation engineering problem. It defines the acceptance criteria and a traceable test protocol, and runs qualification on the target hardware with versioned results an auditor can check. This is part of its AI engineering for manufacturing and automotive work.

How do you run a Gauge R&R when one of the operators is a vision system?

Treat the vision system as one appraiser and run the study you would run for people, but pick the procedure by whether anyone can still influence the result. Bosch’s measurement capability booklet (2019) separates two cases. A type-2 study covers processes with operator influence. A type-3 study covers stations where parts are clamped in a fixed position and measurement and analysis run fully automatically. On paper an automatic inline vision station looks like a type-3 case. In practice, operator-influenced analysis of raw data, placement without a reproducible clamp, or clamping force that depends on the operator all bring the operator back in. Bosch lists each of these.

Calibrate and measure bias before any R&R

The vision system has to be calibrated first. That means mapping pixel coordinates to real-world units with artefacts traceable to national standards, after lens and perspective distortion have been corrected. A Type 1 study then measures bias against a reference value. Vision Systems Design (2022) describes a single operator making at least 30 inspections of one calibration artefact for each attribute. Bosch measures a calibrated standard 50 times (at least 25), removing and re-clamping the standard between readings. Bosch treats passing the type-1 criteria as a prerequisite for the R&R procedures that follow.

Variable and attribute study designs

For measured outputs, the widely used AIAG design is about 10 parts, 3 operators and 2 or 3 trials each, with the parts spanning the real production range. The recommended version (10 parts, 3 operators, 3 replicates, 90 measurements) gives adequate power for estimating variance components. TestGage Insight notes that two trials leave only one degree of freedom for within-operator variance. Parts picked from the middle of the distribution compress the total variation, which makes the measurement system look worse than it is.

Pass/fail outputs carry less information per trial. The AIAG MSA 4th edition therefore asks for at least 20 parts in an attribute study, with 3 appraisers, 2 trials, and at least 5 known good and 5 known bad parts. Tenneco’s and Bosch’s documents ask for more. Tenneco’s PPAP guidelines call for 30 pieces across the tolerance range, 20% of them out of specification, with 3 operators and 3 trials. Bosch recommends a reference lot of 100 to 200 parts, cites 50 as the AIAG minimum, and wants at least 3 appraisers in at least 3 runs wherever handling or judgement can affect the rating. Because these figures differ, the protocol should name the standard it follows and the customer requirement it satisfies.

When the vision system sits alongside certified inspectors as one appraiser, attribute agreement analysis reports three results to read separately. Within-appraiser kappa shows whether each appraiser agrees with itself across trials. Between-appraiser kappa shows whether appraisers agree with each other. Each-appraiser-versus-standard kappa shows whether each one is correct. Number the parts so appraisers cannot recognise them, re-randomise the order on every trial, and make reference identities readable only by authorised personnel.

False rejects, false accepts and the AQL

Detection figures need a false-positive rate beside them. Report a confusion matrix against known-defect samples as well. The Indian Pharmaceutical Alliance’s best-practice document on sterile visual inspection defines the AQL as the defect rate at which 95% of lots are accepted, and says critical defects should carry an AQL of zero. Size the false-positive side on purpose. MIL-HDBK-1823A (2009) recommends at least three times as many unflawed sites as flawed ones, and at least 100 false-positive opportunities to estimate a rate below 1%.

Bosch calls for a new capability study when the test system, reference catalogue or basic test conditions change. Visual inspection is probabilistic, and detection probability varies with the product, the particle and the package. For that reason, the GSK authors writing in the PDA Letter describe automated visual inspection qualification as relative to a manual probability-of-detection baseline, not an absolute criterion.

What %GRR, ndc and kappa should a vision system meet before it replaces manual inspection?

The table sets out the acceptance criteria the sources give. Kappa limits vary between sources, so the protocol should state the exact number it uses.

Metric	Acceptable	Conditional	Not acceptable	Source
%GRR (% study variation, AIAG MSA 4th ed.)	under 10%	10% to 30%, with justification, non-critical characteristics only	30% or higher	AIAG via Harmony and TestGage Insight
%GRR (Bosch, 2019)	10% or less	over 10% up to 30%	over 30%	Bosch Booklet 10
ndc	5 or more, required in addition to %GRR	n/a	below 5	AIAG MSA
Kappa (AIAG, as reported by Minitab)	at least 0.75; 0.90 preferred	n/a	below 0.75	Minitab
Kappa (Bosch, 2019)	0.9 or higher	0.7 to below 0.9	below 0.7	Bosch Booklet 10
Kappa (Tenneco PPAP)	above 0.75	n/a	below 0.75, improvement plan needed	Tenneco
Pharma AVI performance run	no missed defects, false rejection no higher than 2.5%	n/a	any miss, or false rejection above 2.5%	Indian Pharmaceutical Alliance

The variable-gauge criteria work as a pair. A system can pass on %GRR and still fail on ndc, and it has to meet both. Criticality tightens the bar. Prestolite’s supplier handbook requires %R&R of 10% or less for special characteristics. Bosch takes the lowest kappa found in the study as the one that decides the classification. A protocol that says only “kappa above the AIAG limit” leaves the quality board guessing which limit was meant, so write the number down.

Repeatability vs reproducibility: what a failed study is pointing at

High repeatability variation points at the equipment, and the fix is to recalibrate, replace or redesign the fixture. High reproducibility points at the method, and the fix is clearer standard work, better training or a fixture that takes judgement out of the measurement. A high operator-by-part interaction means some appraisers handle certain part types differently, which calls for a redesigned procedure. Resolution is a quiet failure mode. If the gauge reads too coarsely for the tolerance, no calibration rescues the study. After any corrective action, repeat the study and confirm it meets the criteria before the system goes back into production use.

On a camera station, repeatability is the variation of one vision system measuring the same characteristic on the same part repeatedly. Reproducibility is the variation between different operators using the same vision system on the same characteristic. Analyse with ANOVA rather than average-and-range, because ANOVA detects interaction effects such as operator by part.

A vision-versus-calliper study starts from the AIAG recommended design of 10 parts, 3 operators and 3 trials. Analyse it with ANOVA so that reproducibility is separated from repeatability. Before the vision gauge replaces the calliper, a paired method comparison should show it is at least as capable as the calliper baseline. See also designing a defensible Gauge R&R study for vision inspection systems.

Is it the camera or the shift? Separating measurement error from lighting and thermal drift

Shifts belong in the study design from the start. Performance qualification for an automated inspection system means consistent results on real production units across operators, shifts and lots. A TestGage Insight author recalls a CMM result that looked capable at 12% and rose to 41% once a second-shift operator using a different probe compensation routine was added. The gauge was fine; the method was inconsistent. Bosch requires every factor that acts in series production to act during the study as well, and it lists environmental conditions among the causes of a non-capable process.

Environmental conditions affect capability. NIST defines drift as a slow change in a gauge’s response, and notes that short-term drift often comes from heat building up in the instrument. In a Minitab case study with Tenneco, failed Type 1 comparisons were traced to an out-of-range ambient temperature in a CMM lab and, in another case, to two vision systems with different lens types. iFactory describes an aluminium camera mount in an unheated area that shifted working distance enough to push borderline parts across the threshold on cold days. The fix was a low-expansion mount and a recalibration at the start of each shift, with no change to the model. On lighting, TestGage Insight recommends flicker-free LED lighting with CRI above 90 and warns that halogen lamps create thermal gradients on vision systems.

Thermal or lighting drift can push a sound vision station’s borderline results across the threshold. To tell the causes apart, log ambient temperature, lighting state and camera warm-up time against every trial. If the variance persists once conditions are held steady, look next at the camera, optics or algorithm.

Capability results only describe the period when the study ran. We would spread trials across full shifts and include runs at the start and end of each one. NIST’s approach of measuring check standards daily over a set period and plotting them over time shows the extent and nature of any drift.

Does the study have to cover every part colour and a worn fixture?

Yes, if production runs every colour and the fixture wears in service. A study on grey parts and a new fixture only tells you how the station behaves on grey parts and a new fixture. MIL-HDBK-1823A asks the experimental design to identify the process variables that cannot be tightly controlled in the real inspection environment, and to set up a matrix of conditions that represents that environment fairly. It also warns that a mismatch in surface finish between specimen and part can raise the false-positive rate on production parts.

Building colour and fixture wear into the study matrix

The full specimen set is inspected at every condition in the test matrix. Take three approved colours and two fixture states (new and worn) as an example. That gives 3 × 2 = 6 conditions. With the AIAG attribute design of 20 parts, 3 appraisers and 2 trials, each condition produces 120 judgements, so the full study is 720. The demonstration fixture should match the production setup in orientation, manipulation and line of sight.

Locating-pin wear shows up as repeatability variation, because a loose fixture is one of the equipment-side causes of a gauge disagreeing with itself. To put a number on what the wear costs, run the same design on the new and the worn fixture and compare the repeatability variance components. The difference is the wear’s contribution. A result from one wear state does not carry over to another, so if the fixture will be used to the end of its wear life, test it in that state. If the full factorial is too expensive, test at least the lowest-contrast colour on the worn fixture, and record that the other combinations were inferred, not tested.

Re-verifying the reference defect panels

Re-verifying the panels answers a different question from the R&R study. The R&R study asks whether the station’s calls are reliable. Panel re-verification asks whether the ground truth those calls are judged against is still true. Neither replaces the other. MIL-HDBK-1823A notes that many specimen sets have been ruined by mishandling. It asks, as a minimum, for specimens to be packaged individually, handled carefully, cleaned after each use, and re-validated at agency-specified intervals when used periodically. Bosch allows the boundary-samples catalogue for visual inspection to be either physical parts or photographs, and counts any significant change to that catalogue as a trigger for a new capability study.

The re-verification procedure has three parts:

• confirm each panel is still substantially the same as the standard used in the original demonstration;
• re-validate the panels at the specified intervals;
• log the outcome as a permanent record, because MIL-HDBK-1823A expects control charts or equivalent records in the process control plan that follows a demonstration.

To make sure it is not forgotten, give panel re-verification its own due date in the same calibration system that already tracks the gauges. For cameras, iFactory says there is no universal recalibration interval. On camera recalibration, iFactory’s advice is to measure the actual drift first and set the interval from that data, and the same logic fits panels: set the interval from real handling counts and measured change, not the nominal line rate.

A panel stops being a reference standard when scheduled re-measurement shows its defect features have drifted outside the tolerance band set by the original study, not when it simply looks worn. Photographing and re-measuring each panel’s defect features at every re-verification gives that decision a record, and Bosch already accepts photographic images as a form of boundary-samples catalogue for visual inspection.

One governance rule for every plant and line

Use one written policy across the group. Score each plant’s readiness against it, and make the gate stricter according to the safety classification of the inspection, not according to which plant runs it.

TechnoLynx runs the AI Readiness Scorecard to score each plant against that one policy and turn the gaps into a remediation backlog, and the first step is to agree the written policy the plants will be scored against. When the question grows into one rule across lines and plants, the route is TechnoLynx’s AI Governance and Trust work.

Readiness scoring and a stricter gate for safety-relevant inspections

A readiness score is only as good as the evidence behind it. We would score each plant on auditable artefacts (a current R&R report, a panel re-verification log, a configuration record), not on self-assessment. The sources already grade by risk. AIAG’s conditional %GRR band applies only to non-critical characteristics. Prestolite requires 10% or less for special characteristics. The sterile-inspection best-practice document assigns critical defects an AQL of zero. A safety-relevant line therefore gets the tighter thresholds and the extra evidence written into the same policy, and every plant applies them the same way.

In the Amazon SageMaker model registry, a model version registered as PendingManualApproval can be moved to Approved, and that transition starts CI/CD deployment. Requiring that approval from someone other than the person who trained or tuned the model turns the policy into a control, not a guideline.

One protocol the quality board approves once

The way to get one approval that still lets each plant show its own rig meets the bar is a single pre-registered protocol, stratified by plant, rig, lighting and part population, with one set of accept/reject criteria. We would apply the stratified approach only where plants share comparable rigs. Bosch also treats relocation of a test system as a reason to repeat the capability study, so every site still needs local verification under the shared protocol.

When can manual backup inspection be withdrawn, and should the station run in shadow mode first?

Withdraw manual backup inspection only when the study meets thresholds agreed before the data was seen and the fixture and process are stable. If the indices are borderline, the fixture is part-way through its wear life, or the line is new, run the station in parallel with inspectors first. If the current R&R results were taken on a fixture that is due for refurbishment, keep the backup in place until the study has been repeated on the refurbished fixture.

We treat the shadow-mode sign-off package as a release-readiness gate: it is the evidence that has to exist before the station runs on its own. Run the sequence below in order:

1. Write the thresholds, the decision rule and the named approver into the protocol before any trial. MIL-HDBK-1823A warns against tuning a procedure to get a high score that does not reflect true capability.
2. For measured outputs, pass the Type 1 study and then the R&R study.
3. Run a paired method-comparison study, ideally with seeded known defects, showing the new method is at least as capable as the old one.
4. Run a performance check. In the Indian Pharmaceutical Alliance’s scheme, product is inspected automatically three times and also by qualified inspectors, and the automated system may miss no defect and may not exceed 2.5% false rejection.
5. Prove consistent performance on real production units across operators, shifts and lots.
6. Have the named approver sign the decision against the evidence package.
7. Record the triggers that reopen the decision, such as camera adjustment, maintenance involving substantial disassembly, software changes, reference-catalogue changes or relocation.

In practice the length depends on defect frequency: the number of clean parts needed before a miss rate can be bounded is a confidence-interval question, covered in statistical confidence bounds for retiring manual inspection on edge vision systems. The Indian Pharmaceutical Alliance document says reinspection should use a procedure approved by the Quality department.

Making a Gauge R&R re-run reproducible on the GPU side

Making a re-run reproducible means scripting the whole inspection pipeline, pinning every version the result depends on, turning on deterministic modes, and logging enough of the configuration that an auditor can confirm the validated setup is the one running.

Pin the validated configuration and enable deterministic modes

GPU inference does not guarantee reproducibility. The Hugging Face Diffusers documentation says full reproducibility across hardware is not guaranteed because matrix multiplication is less deterministic on a GPU than on a CPU, that results can differ between CPU and GPU even with identical seeds, and that complex pipelines need the same hardware and PyTorch version to reproduce. The same documentation shows that PyTorch can be configured to use deterministic algorithms. NVIDIA’s framework-reproducibility notes record that NGC TensorFlow Docker images added GPU-deterministic ops from version 19.06 and multi-algorithm deterministic cuDNN convolutions from 19.12. They also list an open issue reporting different results despite determinism enabled, a fixed seed and the same hardware, so some nondeterminism can remain even then.

The minimum run record we would keep has two halves:

• On the software side: the application version, GPU driver, CUDA and cuDNN versions, the container image digest, a hash of the model weights, and a hash of the built inference engine (a TensorRT engine is a build artefact and should be versioned as one, not rebuilt silently).
• On the camera side: exposure, gain, focus, white balance and lighting settings.

Bosch already requires all study parameters and settings to be documented, and it treats software changes as a trigger for a new study. To check a re-run, we would replay the stored images from the validated run and compare decisions. We would require pass/fail calls to match exactly. Score differences can be allowed within a stated tolerance where bit-identical output is not achievable. Matching hashes show the running configuration is the one validated. They do not prove that configuration was right.

Re-verifying panel ground truth so it survives an audit

Record reference ratings before the study, and hide them from the people taking part. We would re-certify a panel with a second independent inspector plus a reference measurement. It mirrors the inspector-qualification practice in which units a trainee has inspected are re-inspected by a qualified inspector. Check consistency between graders with between-appraiser kappa. Reference standards used for re-verification should be substantially the same as those used in the original demonstration. Re-running the station on a panel does not re-verify the panel. Repeated inspections give information about the inspection, not about the specimen. We would re-verify every panel where the set is small, because a single mislabelled panel affects every trial it appears in. Sampling is only defensible when the acceptable risk of a mislabelled subset is written down.

Testing a vendor’s detection-rate claim independently

A credible detection claim comes from a study the vendor did not design, label or run. MIL-HDBK-1823A states that the aim of a reliability demonstration is to find the largest defect the system can miss, together with its false-positive rate. It recommends at least 60 targeted sites for a hit/miss system. It also calls for production procedures, production inspectors and an accountable test monitor. A vendor that controls the test images or the labels can still shape an “independent” result, so the test set and the ground truth should stay with the buyer.

What the quality board needs on file before manual final inspection is removed

Before manual final inspection is removed, the quality board’s file needs three things: a documented validation method, performance measured against the manual baseline, and traceable records of the test data and software versions.

TechnoLynx builds that record as validation engineering work. It sets the acceptance criteria and test protocol, runs qualification tests on the target edge hardware with versioned results, and builds the reports for leadership sign-off. During a staged rollout the manual override stays in place, instrumented so it keeps collecting evidence.

Sign-off package checklist

• Baseline of the manual method: SOPs, historical defect data, existing gauge R&R and operator qualification studies, training records, and CAPAs tied to inspection escapes.
• Installation qualification: cameras, lenses, lighting, firmware and network installed to spec, with calibration current and configuration documented.
• Operational qualification: lighting and contrast limits, sensitivity and specificity under worst-case conditions, repeatability, and a confusion matrix against known defects.
• Performance qualification on real production across operators, shifts and lots.
• Statistics matched to the output type, built around a paired method-comparison study.
• The pinned configuration record and replay check described above.
• A summary report stating whether the system met specification and, if it did not, the remedial action and an interim disposition.
• An update to the risk management file where ISO 14971 applies.
• The control plan and the triggers for repeating the study.
• The signed decision, with the approver named in the protocol.

Common validation gaps include checks that the system only powers on, change control without verification, and no statistical rationale for sample sizes.

Taking the human override off any automated decision

Define “reliable enough” as a numeric error bar before the shadow run starts. Measure against it under conditions that match production. Keep the override as an interim control until the bar has been met. Evidence gathered under test conditions that differ from deployment loses most of its value in an audit.

Get the protocol (parts, appraisers, shifts, repeats, acceptance criteria) agreed with the customer before running the study, in line with Bosch’s expectation that specification compliance is documented and, where necessary, agreed with the customer.

Who can help us validate the vision station and build the withdrawal rule?

TechnoLynx engineers the validation of a vision inspection station. That covers the acceptance criteria and traceable test protocol the R&R study runs under, the qualification runs on production hardware with a pinned and logged configuration, and the shadow-mode harness and reports the quality board signs against. Whichever partner you choose, ask whether they will fix thresholds before seeing data and hand over the configuration record with the results.

Other routes are reasonable. A quality team with Minitab gets kappa and Kendall statistics for the attribute study. An accredited metrology lab supplies calibration artefacts traceable to national standards.

FAQ

What are the three elements checked in a gage R&R?

Repeatability (variation from the gauge), reproducibility (variation from operators) and part-to-part variation (the process). The system is considered reliable when part-to-part variation dominates.

Is repeatability the same as precision?

Not exactly. Vision Systems Design defines repeatability as the variance of repeated measurements of the same characteristic by one operator on one system, and precision as how close measurements are to each other. MIL-HDBK-1823A notes that these terms are not universally agreed and are often used inconsistently.