Moving an automated optical inspection (AOI) system from a pilot cell to a full production line means answering several separate validation questions, and each one needs its own method planned in advance. After a change, the AOI has to be re-qualified as a measurement system, using attribute agreement analysis. We recommend statistically sized before/after tests of false-reject and escape rates, and a per-head check against a margin fixed in advance. Someone other than the team that tuned the system has to check those numbers. And on a GMP site, changes to the validated AOI system go through change control and an impact assessment. If all of them are answered with one number the vision team computed on its own data, none of them is settled.

TechnoLynx builds this validation evidence for vision teams on manufacturing and automotive lines: the test protocols, the clean logged data and the reports that each of those questions needs.

The kinds of proof an AOI scale-up has to produce

Each reviewer asks a different question, and each question needs its own kind of evidence. The table pairs them up.

Validation question	Who asks it	Method	What counts as evidence
Does each head or line hold its false-reject and escape rates at full speed?	Quality engineering	A per-head comparison planned in advance against the process being replaced, using known-good and known-defect test sets run at operating speed	Per-head false-reject and escape rates with confidence intervals
Do the new thresholds still separate good from bad consistently?	Quality or MSA owner	Attribute agreement analysis (attribute gauge R&R) against reference truth established independently	Kappa, percent effectiveness, and Type I and Type II error per stratum
Did the false-reject rate fall because of the change?	Operations and quality	A two-proportion test sized by power analysis, under matched conditions	The difference with its confidence interval, from a test named in advance
Did cost per unit fall, or just move somewhere else?	Finance	A shadow-mode run plus full cost accounting, including manual re-inspection	Cost per inspected unit, with volumes held comparable
Does the change trigger re-qualification or a new conformity route?	QA and machinery safety	A documented change-control impact assessment, plus a machinery modification assessment where safeguarding changes	The impact assessment and a justified retest decision

Per-head false-reject rates at full line speed

Per-head false-reject and false-accept rates for an AOI line only stand up to scrutiny when they are measured on every head, at true operating speed, under a protocol written before the run, against the performance of the inspection process the machine replaces. Pharmaceutical practice for automated inspection of injectables, a regulated reference rather than a rule for steel or PCB lines, sets that baseline explicitly. Qualifying the machine means showing that it detects known defects as well as manual inspection or better, which the Knapp and Kushner method expresses as a mean automated reject-zone efficiency at least equal to the manual one (aRZE ≥ mRZE). The USP <1790> draft text sets the same expectation for machine methods. The EU’s July 2025 consultation draft of Annex 22 extends the rule to AI models: acceptance criteria should be at least as high as the performance of the process being replaced, so that performance has to be known first.

We recommend building on that baseline with a per-head non-inferiority design. Before any data is collected, you fix the margin by which a head may fall short of the reference, and then test each head against that margin. Knapp and Abramson’s tables give the minimum number of inspections of reject-zone units needed to compare two inspection methods at a 95% confidence interval. A3P reports that using those tables cut the number of passes through the machine, and with it the time automated qualification took.

How do we know rejects are genuine particulates and not motion blur or vibration?

You separate genuine rejects from line-speed artefacts with two controlled runs at operating speed. In a run of known-good units, every reject is false by definition. In a run of characterised defect standards, every miss is an escape. Artefacts depend on how well conveyor speed and camera timing stay in step. Tucsen’s guidance on TDI line-scan imaging explains that a mismatch between conveyor velocity and camera line rate compresses or stretches the image, that motion off the scan axis blurs it, and that vibration or fluctuations in trigger frequency cause line-to-line breaks and banding. A pilot validated at reduced rate therefore says little about full-rate behaviour.

For semi-automated inspection, USP <1790> says light intensity must be controlled, and that spin speed and rotation rate should be established during validation and kept within the validated range. The chapter also names assignable causes, such as air bubbles or container variation, that result in high false-rejection rates, more common with automated systems. For those cases it allows a second inspection step for units whose status is uncertain. ECA guidance on qualification sets asks for good, particulate and non-particulate defect containers, with about 10-20% (and under 30%) defective. For the Knapp test it describes a particle set spanning 50 µm to 1000 µm in materials typical of production. A short functional set, run before and after each batch, confirms the machine is still working.

When you log the good-unit run, tag every reject with its image, head and speed. Treat rejects that cluster on one head, rise with conveyor speed or show smeared edges as suspected image-acquisition artefacts to investigate.

Do we need separate model calibration per head, or can one model generalise across heads?

Whether one model generalises across heads is an empirical question, and only a test on your own heads answers it. Run the shared pilot model on every head under the same protocol, compute per-head false-reject and escape rates, and check each head against the non-inferiority margin. If every head stays inside the margin, deploy one model. If one head falls outside, calibrate that head on its own and keep the rest on the shared model. Draft Annex 22 asks for test data that is stratified and includes all subgroups. Treating each head as its own subgroup, rather than a slice of a line-level average, stops one weak head hiding behind the others.

Two limits apply. Per-head error rates need enough defects on each head, using artificial defects where real samples are unavailable. A generalisation result also holds only for the cameras and lighting you tested. Change the optics on one head and that head is unvalidated again. Draft Annex 22 expects the metrics of a deployed model to be monitored for changes such as a shift in lighting. Standardising hardware and lighting across lines makes a per-head failure less likely, but it does not remove the need for the per-head check.

Is there a standard gauge R&R method we can reuse after a threshold retune?

Yes. Attribute agreement analysis, also called attribute MSA, handles binary pass/fail results. It suits an AOI after a retune, because MSA and gauge R&R studies are the de facto standard for checking machine vision measurements. MSA guidance calls for a new study whenever a measurement system is introduced or changed. ISO 13485 and FDA rules likewise require any change to inspection software or acceptance limits to follow change control and revalidation.

A gauge R&R study measures the quality of the measurement system, not the quality of the parts. Gauge R&R is judged as a percentage of total variation, and the AIAG treats anything under 10% as acceptable. An AOI classifies each spot as defective or non-defective, a pass/fail result, so the attribute study scored with kappa applies.

How the attribute study maps onto an AOI

SigmaXL’s description of attribute MSA gives three comparisons, each corresponding to a gauge R&R component. Within-appraiser agreement corresponds to repeatability and needs at least two trials. Between-appraiser agreement corresponds to reproducibility and needs at least two appraisers. Agreement of each appraiser, and of all appraisers together, with a known standard corresponds to accuracy. For a machine, the “appraiser” is whatever unit you doubt is consistent: each inspection head, each line, or each mill’s installation. Repeatability is then the same part presented several times to the same head, and reproducibility is the same part run across heads or sites.

SigmaXL’s rule-of-thumb kappa bands are 0.9 or higher for very good agreement, 0.7 up to 0.9 for marginally acceptable, and below 0.7 for unacceptable. For percent effectiveness, the bands are 95% or higher for very good, 85% up to 95% for marginal, and below 85% for unacceptable. The percent-effectiveness bands assume equal numbers of known-good and known-bad parts. A Type I error is a good part consistently called bad, and a Type II error is a bad part consistently called good.

What sample size and supplier mix does a re-qualification batch need?

The sizing guidance differs between sources. The CPQP MSA guideline sets a minimum of 10 production parts, three operators and three trials for an attribute study. The machine vision practice described in Vision Systems Design uses the same pattern of 10 parts, three operators and three trials for gauge R&R. Minitab’s attribute agreement guidance asks for at least 50 samples, three to five appraisers, and at least two ratings of each sample in random order. For a binary verdict, Minitab’s larger figure is the safer basis.

The mix should be roughly balanced between categories and include several parts that only just pass and several that only just fail. A paper on qualifying deep-learning inspection of medical devices recommends that 25-50% of units be defective. It also recommends mixing marginal cases with clearly nonconforming items, and using artificial defects where real ones are unavailable. MSA guidance obtains each part’s reference value through an independent body. Draft Annex 22 suggests independent verification by multiple experts, validated equipment or laboratory tests. Without that independent reference, the study only measures agreement with the old threshold.

For a retune that spans three mills, stratify the batch by mill, coil supplier and finish, and give each cell its own marginal parts. A thin cell gives a weak conclusion for exactly that mill-supplier combination, however healthy the total looks.

Proving the retuned threshold did not trade escapes for false rejects

Report Type I and Type II error for each stratum, not pooled across the batch. A retune that cuts false rejects on one supplier’s finish while letting more defects through on another shows up as a rise in Type II error in that cell. The authors of an AOI false-call reduction case study (arXiv, 2025) argue for metrics tied to the actual requirement. A slip, meaning a defective board passed on, is significantly worse than missed volume reduction, so a model has to stay below a target slip rate before its false-call savings count.

The complete study design, from appraiser selection to manual sign-off, is covered in validating vision inspection systems with gauge R&R. Where the open question is how the threshold for critical defects should be set in the first place, see calibrating and validating AOI thresholds for critical defects.

What sample size and time window prove a false-reject improvement is real?

A false-reject improvement is tested with a two-proportion test, such as the z test of equal proportions for reasonably large samples. The time window is however many days your daily throughput needs to reach the sample size that a power analysis calls for.

TechnoLynx can run this study as an engagement. TechnoLynx defines the comparison protocol and its confounder controls, and runs a power analysis against your baseline false-reject rate and daily volume to show whether ten days is enough. TechnoLynx then instruments clean before/after logging tied to the threshold change, and delivers a report covering method, data and confidence intervals that a party outside the vision team can review.

Sizing a before/after false-reject study

1. Fix the baseline false-reject rate p2 from logged data at the current threshold, on the same heads and with the same part mix.
2. Fix the smallest improvement worth detecting, δ = p1 − p2, together with alpha and the target power. NCSS’s PASS documentation defines these as the inputs to two-proportion power and sample-size calculations.
3. Choose the test you will analyse the data with. For small samples, the power of the competing tests can differ widely, so NCSS says the power analysis should be based on the test you will actually run. The NIST/SEMATECH e-Handbook uses a z test when samples are reasonably large and Fisher’s exact test when the two independent samples are small.
4. Compute n for each period from those inputs, using the power procedure for the test chosen in step 3.
5. Divide n by the daily inspected volume to get the days needed for each period, then add time for analysis and independent review.

Because n scales with 1/δ², halving the improvement you want to detect roughly quadruples the sample you need. NIST’s report NISTIR 7740 found no significant difference between survey-sampling, binomial and bootstrap confidence intervals for false accept and false reject rates, and found the binomial approach the most computationally efficient. The same report notes that for large n and small p, the binomial can be approximated by the Poisson distribution (n ≥ 20 and p ≤ 0.05, or n ≥ 100 and np ≤ 10).

If daily volume times ten covers the required sample for both periods, a ten-day study with a test named in advance is realistic. If it does not, you can extend the window, accept a larger minimum detectable effect, or use a sequential design that gives an early provisional result and a later confirmatory one.

Attributing the drop to the threshold, not to everything else that changed

The comparison needs the same heads, the same part mix and the same stations, plus a log of everything else that changed during the window: process adjustments, supplier lots, shift patterns. Time itself is a confounder. In the arXiv false-call study, models that met their slip and volume targets during modelling decayed immediately on later data, with slip rates exceeding the target in the very first evaluation slice. The authors stress keeping the data used to set a threshold strictly separate from the data used to evaluate it. The “after” period should therefore be fresh production after the change, never the tuning set. USP <1790> applies the same caution to rejection limits: action or alert levels should not be lowered for a planned improvement until its impact has been measured over enough time.

Escapes belong in the same study. A false-reject drop bought with a rise in escapes is not an improvement.

Who outside the vision team can verify the attribution

Independent verification means a quality or statistics function, or a third party, recomputes the sample size, confidence interval and attribution logic from the raw logs instead of reviewing the vision team’s output. That verification is only as good as the reviewer’s access to complete, unedited inspection logs for both periods. Draft Annex 22 asks for controls that keep test data independent of development data. It also asks for an approved test plan, with metrics and acceptance criteria defined in advance, before testing starts.

How do we prove to finance that the AOI change actually saved money?

Finance needs two things a vision team cannot certify on its own: a cost-per-unit comparison that shows where cost moved, and a false-reject figure that someone else has checked. Each needs its own route.

Did cost per unit fall, or did it move somewhere else?

Shadow mode runs the new policy flagging units but not rejecting them. Because both policies then judge the same units, the cost-per-unit comparison is not muddied by differences in volume or mix. Oxmaint’s implementation guide shows what is at stake: a 5% false-reject rate on a 10,000-unit shift means 500 good units scrapped each day. For regulated medical-device inspection, the 2025 arXiv paper on deep-learning inspection proposes a pre-qualification observation step. The model runs in production with its outputs hidden while manual inspection continues, for a statistically defined period such as two to three months, before formal validation begins.

Shadow mode has limits. In Shankar et al.’s interview study of machine learning engineers, participants said shadow mode was not a substitute for every deployment stage, and that it cannot evaluate products with a feedback loop. The same study records shadow mode being used to convince product managers and business analysts that a change was worth shipping. Run length and volume are the real constraints.

A narrow cost-per-unit metric also misses cost that shifts to manual inspection. AOI false calls consume human effort at manual inspection stages, and USP <1790> describes automated systems routing uncertain units to a second, manual inspection. Count re-inspection labour, scrap and the cost of escapes, and keep volumes comparable between the two periods.

Verifying the false-reject figure within ten days, without rerunning the team’s own script

The fastest credible check is a second party recomputing the figure with its own code, from the raw logs and a re-inspected sample. Rerunning the originating script simply reproduces any error in it. Within ten days, this may only be a partial re-check, for example one period plus a stratified sample of rejects re-inspected against reference truth, rather than a full re-audit. The report to the CFO should say which one it is.

Oxmaint claims that well-tuned systems reach false-reject rates under 0.5-1%, on condition of disciplined lighting, consistent labelling and quarterly revalidation. That is a vendor claim, not a measurement of your line. The arXiv false-call study set its own targets (at least 40% volume reduction at no more than 1% slip) because it had no benchmark values to compare against.

The point is to treat this re-check as a release gate, not a slide in a deck.

Does a GPU swap or threshold change mean full re-qualification under GMP change control?

A change to a validated AOI system, including a hardware change, goes through documented change control, with an impact assessment that plans any re-qualification. TechnoLynx supports this through its AI Governance and Trust work. TechnoLynx prepares approval evidence packs built to survive an external auditor or regulator. Reviewing and approving the change stays with the site’s qualified representatives under its change-control system.

Annex 15 requires written procedures for any planned change to equipment, processes or test methods that may affect product quality. Each change needs a risk assessment of its impact on validation, calibration and regulatory status, which then plans any validation, verification or requalification needed. As Cloudtheapp summarises Annex 11, changes that affect validated functionality must be re-validated before going live in the GMP environment, and that includes infrastructure changes. The consultation draft of the revised Annex 11 names hardware components explicitly and subjects any significant change affecting product quality, patient safety or data integrity to re-qualification and validation. Draft Annex 22 addresses AI models specifically. Any change to the model, the system, the process, or the physical objects the model uses as input must be evaluated to decide whether a retest is needed, and a decision not to retest must be fully justified.

These texts govern GMP manufacture of medicinal products. For an AOI on a steel or PCB line, we recommend borrowing the documented impact assessment anyway.

The GPU swap specifically

A GPU swap is not cosmetic for a deep-learning inspector. Different hardware and inference libraries can produce slightly different floating-point results, and a score sitting near the threshold can flip. That makes the proportionate retest concrete. Run the frozen validation set on the old and new hardware, compare verdicts unit by unit, look closely at scores near the threshold, and document the result as the justification for how much re-qualification you do. PIC/S guidance expects firmware and operating systems to be qualified for their intended use, including their version. Annex 15 bases the scope and extent of qualification on a justified, documented risk assessment.

Documentation that shows an inspector equivalence was confirmed, not vendor-claimed

• Defined requirements, a system description, and verification evidence that the system was qualified and meets GxP requirements.
• A change record with the reasons for the change, an impact assessment including regression analysis and testing, and the approval decision.
• Executed test scripts showing the requirements are met.
• A review of any vendor-supplied validation documents, since the regulated user stays fully accountable and decides whether to repeat tests.
• Results traceable to a specific model version.
• Version control over life-cycle and validation documents, without which inspectors treat the validation status as suspect.

Keeping that trail intact across every later threshold change is the subject of revalidating AOI thresholds and building audit trails from pilot to full-rate production.

Replacing a light curtain with a camera: substantial modification and CE

Under the Machinery Directive, whether a modification is substantial is judged case by case. A German interpretation paper sets out the test: any change to safety technology must be assessed for new hazards or increased risk. If the machine remains safe without additional protective measures, there is no substantial modification. Machinery transformed so substantially that it can be considered new falls under the Directive as new machinery. In that case the Directive’s requirements for new machinery apply again. The supplied interpretation covers the Directive, so check how the successor Machinery Regulation words this for your own case.

On standards, IEC 61496-1:2020 sets the general requirements for electro-sensitive protective equipment, and IEC 61496-2:2020 covers active opto-electronic devices such as safety light curtains. IEC TR 61496-4:2007 has been withdrawn, so vision-based protection has to be assessed against the applicable current document. The whole chain from detection to safe machine behaviour is evaluated under ISO 13849-1 or IEC 62061, and a Type 4 light curtain does not automatically make the function PL e. IEC 61496 covers equipment intended to detect people or parts of their bodies without contact. Using vision for safeguarding is a safety-function design project, separate from validating the AOI.

Who can help us re-qualify AOI changes and document them for sign-off?

TechnoLynx designs the before/after test and its power analysis, instruments the logging, and writes a report your quality unit or an outside reviewer can audit. Whether to keep this in-house depends on your gaps. A quality function fluent in MSA and two-proportion testing can run the statistics itself. Outside help earns its place where the gap is the logging, a GPU or inference change, or independence from the vision team. When choosing a partner, ask to see the protocol before any data is collected, and confirm you will receive the raw logs and code so a third party can recompute every figure. To find which evidence is missing, the AI Readiness Scorecard scores the programme against a published rubric with a remediation backlog.

Under the GMP texts above, the regulated user remains accountable for any documentation a provider supplies. Conformity of vision-based safeguarding belongs with a machinery safety specialist.

FAQ

What is the difference between MSA and gauge R&R?

Measurement system analysis (MSA) validates the data quality of a whole measurement system: gauge, procedure, environment and operator. Gauge R&R is one type of MSA study. It quantifies the variation that comes from the device (repeatability) and from the operators (reproducibility).

Which is worse for an AOI, false positives or false negatives?

In the arXiv false-call study, false negatives (slips) were rated significantly worse than missed volume reduction because they harm downstream processes. False rejects still cost real money in scrap and manual re-inspection.

What does it mean for a computerised system to be validated?

The draft revision of Annex 11 defines validation as proving that a process leads to the expected results. For a computerised system, that means ensuring and demonstrating that it is fit for its purpose.