To qualify visual inspection automation, you show that the machine finds known defects at least as reliably as qualified human inspectors do. Both are scored on the same masked challenge set, against a pass criterion fixed before anyone runs the test. In practice you build a known-defect set, give each unit a probability of detection (POD) from repeated manual inspection, compare the machine on the units humans reject reliably, and size the number of inspection runs so that comparison has statistical confidence. The acceptance criteria go into a qualification protocol the quality unit establishes and approves.

The regulators agree on the benchmark. USP <1790>, EU GMP Annex 1 (section 8.32), the ECA good practice paper and the FDA draft guidance all expect automated inspection to be equivalent to or better than manual inspection. The awkward part is the comparator. Manual inspection is probabilistic, and detection rates below 100% are expected, especially for small or low-contrast defects. So the human reference is a distribution, not an answer key, and that shapes every statistical choice that follows.

Attribute agreement or a non-inferiority margin: which test should decide machine versus human?

Use both, because they answer different questions. Attribute agreement analysis describes how consistently appraisers classify units, within themselves, between each other and against a known standard. A non-inferiority test with a pre-specified margin produces the pass/fail verdict an auditor can recompute.

Here is how the agreement side works in Minitab’s implementation. Kappa against a known standard treats the standard as one more trial. It computes kappa for each trial against that standard, then averages the trial values. Fleiss’ kappa can also be computed within each appraiser across repeated trials, and between all appraisers. Reported side by side, these separate an inspector’s own inconsistency from disagreement across the panel. The machine enters the analysis as one more appraiser. What kappa cannot tell you is whether the machine is good enough. A high agreement figure has no threshold attached to it.

A non-inferiority test supplies that threshold. It tests whether one method is worse than the other by more than a small margin, and that margin is defined as the largest difference considered to have no practical importance. When higher detection is better, the hypotheses are H0: δ ≤ −NIM against H1: δ > −NIM. Rejecting the null supports the alternative that the machine is not worse by more than the margin.

For defect detection in injectables, the established form of this test is the Knapp and Kushner method. It runs a manual part and an automated part, then compares the two. Each unit is zoned by its manual POD:

• below 30% is the acceptance zone
• 30% to under 70% is the grey zone
• 70% or more is the reject zone

Manual reject zone efficiency (mRZE) is the mean POD of the reject-zone units. Automated reject zone efficiency (aRZE) is the machine’s mean detection rate on those same units. The pass criterion is aRZE ≥ mRZE, judged only on units that human inspectors reject at least 70% of the time. Grey-zone and acceptance-zone units can be run through the machine for information, but they do not decide the result. USP <1790> describes this 70% Reject Zone as industry common practice worldwide. The chapter also notes the cost of that threshold: particles of a size right at the reject-zone limit may still go undetected up to 30% of the time.

Which statistic answers which qualification question

Question you need answered	Statistic	What it gives you	What it cannot give you
Is each inspector consistent with themselves?	Fleiss’ kappa within appraiser, across repeated trials	Repeatability per rater	A verdict on the machine
Does the panel agree with itself?	Fleiss’ kappa between appraisers	Width of the human reference	Which rater is right
Do raters agree with the truth?	Kappa against a known standard, averaged over trials	Accuracy of each rater, the machine included	A pass/fail threshold
Is the machine at least as good as people?	Knapp aRZE ≥ mRZE on reject-zone units	The qualification verdict	Behaviour on grey-zone units
What margin is fair when no ground truth exists?	Pairwise human–human differences, bootstrapped	A margin derived from real rater spread	Immunity to outlier raters

The reference set: masked known defects with measured POD

POD for each defect standard comes from documented, repeated manual inspection. USP <1790> states that 30 to 50 inspections of each container give secure probabilistic data for particulate standards. The FDA draft guidance adds three conditions:

• defective units should typically make up no more than 10 to 20 percent of a test set
• their identity should be masked from the people being tested
• the quality unit should control the set so the test cannot be manipulated

The ECA good practice paper sets a similar range: 10 to 20 percent but under 30 percent randomly distributed defects, covering all known defects, with defective units invisibly marked.

One source disagrees. A 2025 arXiv assessment of deep-learning inspection for Class III medical devices suggests 25 to 50 percent defective units for test method validation. In the A3P Knapp study, the 10 percent defective kit was chosen to avoid biasing the manual inspection. For injectable products we would follow the pharmaceutical guidance, because the low defect fraction protects the human half of the comparison.

Representativeness is a particular concern for AI-based inspection. The ECA Academy warns that a neural network can learn the artefacts of how an artificial defect was made instead of the defect itself. How to compose, characterise and defend the set is a design problem of its own, covered in designing a regulator-defensible known-defect challenge set.

How many inspectors and challenge units give adequate power?

You calculate the number of runs rather than choose it. It follows from the manual baseline, how many units sit in the reject zone, and the performance you expect the machine to reach. Knapp and Abramson published tables giving the minimum number of inspections needed to compare two inspection methods at 95% confidence. The formula is I = N / RZN:

• I is the number of test-kit inspections
• N is the constant read from the table where the mRZE row meets the expected aRZE column
• RZN is the number of reject-zone units

On the human side, the A3P study used 5 authorised inspectors chosen to represent the wider panel, and each inspected the kit 10 times. That gives 5 × 10 = 50 results per unit, the top of the USP range.

The 2025 arXiv assessment of Class III medical-device inspection describes the Success-Run Theorem as a widely accepted sizing method for attribute validation where no false negatives are permitted. Methods that allow a set number of misses typically need much larger samples. The same paper gives 95% confidence and 99% reliability as example thresholds for high-risk inspection.

Sometimes the endpoint is a continuous paired difference rather than a zone efficiency. For that case the PASS documentation sets out the power calculation for a paired non-inferiority t-test. The test assumes normally distributed, symmetric paired differences. Single binary calls do not meet that assumption, so the test fits per-unit detection rates averaged over many repeats better than raw accept/reject decisions.

Worked example: sizing machine runs with the Knapp tables

The A3P case gives these figures:

• Base case: mRZE of 88% and 80 reject-zone units. The table constant is 914, and 914 / 80 = 11.42, rounded up to 12 machine inspections of the kit.
• Fewer reject-zone units: at the same 88% mRZE and aRZE, 30 reject-zone units would need 31 inspections.
• Weaker baseline: with 80 reject-zone units and an mRZE/aRZE of 75%, the requirement rises to 21 inspections.

The number of reject-zone units has a large effect on how many kit inspections are needed. The ECA paper also sets a floor: the qualification set should be inspected at least 10 times during machine qualification. We would plan for whichever is larger, the table result or that floor.

When the human inspectors disagree with each other

Treat inspector disagreement as the measured width of the human reference, not as noise to throw away. It limits how tight any equivalence claim can be. The Knapp approach already builds this in, because repeated inspections under controlled lighting and pacing are what let you assign a reject probability to each standard unit.

A ground truth that depends on expert interpretation is harder to define. A 2025 arXiv study of neonatal seizure detection lets each human rater take a turn as the reference. Pairwise human–human differences then define the empirical performance range, and the 95% confidence intervals from that range set the non-inferiority margin. Its Average κ variant works like this:

1. Bootstrap the inter-rater agreement among humans.
2. Take the mean minus the 2.5th percentile as the margin m.
3. Substitute the AI for each expert in turn.
4. Accept equivalence if the lower 5th percentile of the change in kappa exceeds −m.

That variant was the most robust of the tests the authors compared. The study is of neonatal EEG, not vial inspection. We think the logic transfers, but that is our judgement, not a published inspection precedent.

The choice of agreement statistic matters under severe class imbalance. In the same study, Gwet’s AC1 stayed above 0.9 at majority-to-minority ratios above about 10:1, even when no minority-class sample was classified correctly. Defects are the minority class, so we would report kappa-family statistics rather than AC1.

Melchore (2011) notes that about 69% of companies in a 2008 PDA preliminary survey used a manual baseline to validate automated inspection, and that a manual baseline must be consistent to give a meaningful validation.

Keep two limits in view. The result is only as valid as the challenge set is representative of production defects. And this is a comparison of decision accuracy, not an assessment of throughput, latency or deployment readiness.

Reconciling an AQL sampling plan with the comparison study’s sample size

An AQL sampling plan is a lot-acceptance rule. It is not a power calculation for the human-machine comparison, so size any shared run to whichever of the two requirements is larger.

ANSI/ASQ Z1.4 decides whether to accept or reject a lot from a sample, against an Acceptance Quality Limit specified in advance. USP <1790> defines the AQL as the defect rate at which 95% of lots are accepted, which makes it a measure of the risk of falsely rejecting good batches. The unacceptable quality limit (UQL), where 90% of lots are rejected, is the better measure of patient risk. Typical visual inspection AQLs are:

• critical defects: 0.010–0.10%
• major defects: 0.10–0.65%
• minor defects: 1.0–4.0%

Z1.4 calls the choice of inspection level a business decision rather than a statistical one.

The resulting samples are small. A C=0 plan needs only the lot size and a risk-based AQL. In a QS Compliance example, a lot of 250 at AQL 1.5 requires 19 randomly selected pieces with zero defects found. USP <1790> notes that for most batch sizes between 3,201 and 150,000, the plans suggest 20 units with an accept number of zero, based on an AQL of 0.65%. At the top of the critical range (0.10%), a 20-unit sample contains on average 20 × 0.001 = 0.02 defective units. A sample that almost never contains a critical defect cannot tell you whether the machine catches critical defects as well as people do.

The sizing sequence we would follow:

1. Compute the AQL sample from lot size, inspection level and AQL per defect category.
2. Separately, compute the per-defect-class count the comparison needs, driven by the rarest class you must make a claim about.
3. Take the larger of the two as the run size.
4. Seed enough instances of each defect class to meet the comparison requirement, even if that oversamples relative to the bare AQL.
5. If a class is so rare that one combined run becomes impractical, run a dedicated enriched seeded batch for it.

Can the AQL check and the human-machine comparison share one seeded vial run?

Yes, provided each defect class has enough seeded units for the agreement statistics and the run’s conditions represent normal production. Keep in mind what the AQL step means on an enriched run. It says nothing about the quality of a production lot. Whether seeded defects behave like natural ones has to be evaluated. The ECA Academy holds that the deciding factor is not whether samples are artificial or real, but whether they reliably represent how the real product behaves under the chosen inspection strategy.

After go-live, routine AQL sampling becomes the continuing comparison. The ECA paper describes the manual AQL inspection of every batch as a man-machine comparison, which amounts to batch-to-batch revalidation of the product. For release, it gives these minimum AQLs:

• critical: 0
• major: AQL 0.65 or 1
• minor: AQL 4 to 6.5

USP <1790> adds that acceptance sampling catches unexpected defects that were not part of the automated system’s development and training. If a batch has to be reinspected, the FDA does not recommend more than one reinspection. Any samples that fail the AQL reinspection are counted with all other rejects in the batch reconciliation.

Qualification is a snapshot, so the comparison has to keep running

The ECA Academy says plainly that qualification is only a snapshot of performance at one point in time. In its view, long-term reliability depends more on continuous monitoring, trending and process control. During routine operation, the ECA paper expects automated performance to be shown within the defect detection limits set during PQ, before and after each inspection run, using a test set with the defects used in qualification.

Annex 1 asks for challenges at regular intervals but does not fix a frequency. A function test before and after the batch is the usual practice. Extra checks are triggered after maintenance, after a restart following a crash or software reset, and whenever drift is suspected. Requalification should ideally be annual, or every two years at the latest, and should include a statistical trend analysis of routine performance data. For machine-learning systems, the 2025 arXiv paper also recommends monitoring feature distributions at inference time to detect concept drift.

In a two-sample test comparing an earlier period’s data with a later one, a p-value below 0.05 is evidence that the two did not come from the same distribution. If a stratum has shifted, in its images or its defect-class mix, an apparent machine improvement may just reflect a change in what reached the line, so stratify the test and report each stratum next to any pooled figure.

How many lots does a shadow run need before Quality signs it off?

USP <1790>, Annex 1 and the FDA draft guidance set the equivalent-or-better benchmark. We would set each line’s parameters from:

• its defect base rates
• the confidence its CSV risk classification demands
• the measured human baseline

Agree them with Quality before the run starts, not after.

LynxBenchAI publishes hardware measurements that inform the choice of GPU for the line the shadow run covers, and the first step is to check its published results for that hardware before the run starts; it does not run your inspection workload.

The 2025 arXiv paper describes the shadow phase. The model runs in production with hidden outputs while manual inspection stays active, for a statistically defined period (2 to 3 months is its example). Results are then compared with the human decisions, and validation goes ahead only if performance matches expectations. The paper also cites GHTF guidance that sampling plans used in PQ must be statistically justified where inspection is a critical process.

We would run the design steps in this order:

1. Estimate defect base rates on the target line, including how often the rare defect types occur.
2. Choose a confidence level and a detectable difference that match the site’s risk classification.
3. Compute the lot count and the stratification by defect type that give that power.
4. Seed rare types where production alone would take too long to supply enough of them.

Set the agreement threshold relative to measured human-human agreement, as in the bootstrap margin described earlier. A machine threshold set above what inspectors achieve with each other is a bar no system can be shown to meet, and it has no scientific justification.

Precedent helps, but it does not transfer automatically. A3P describes the Knapp and Kushner method as the most commonly used AVI qualification method. A points-to-consider paper on machine learning in AVI states that ML implementation does not require changes to current AVI qualification strategies. Even so, the FDA draft guidance warns that inspection procedures carried over from another site or product may not always be suitable.

A lean requalification protocol for a GPU or runtime change

When the model is frozen and only the GPU or inference runtime changes, rerun the locked known-defect challenge set on both stacks and compare per-unit pass/fail outcomes. Do not rely on a vendor’s summary statistics for this. LynxBenchAI publishes hardware measurements that inform the choice of the new GPU, and the first step is to check its published results for that hardware; it does not run your inspection workload.

Revalidation after a change is expected. Validation obligations for manufacturing software require full validation before use and revalidation after changes. Freezing a model and putting it through current validation strategies gives good assurance of reliable production performance. Inspection results should be traceable to a model version and checked for reproducibility, and final test sets should stay locked.

If an equivalence dossier gives only summary statistics, ask for the raw per-unit, per-trial data before accepting it. An unchanged aggregate detection rate can hide a defect class that moved in one direction while another moved the other way. Grey-zone units are borderline by definition, so look there first for flipped calls.

The approach used for a vial-label inspection system transfers to a new detection task only when defect taxonomy and imaging conditions are similar enough. Where they are not, the challenge set has to be rebuilt for the new task.

Can the known-defect challenge pattern be used outside pharmaceutical inspection?

Yes, as a template rather than a citable standard. These parts carry across:

• a masked known-defect set
• repeated human inspection
• a margin fixed in advance
• a machine compared on what humans reliably catch

The three adaptations below need different answers.

Lab-lit training images, sodium-lit mill floor

Measure domain shift with a distributional metric rather than by eye. Maximum mean discrepancy compares two image distributions through their kernel mean embeddings. Simpler tests such as Kolmogorov-Smirnov struggle with high-dimensional data. The size of the effect can be large. In a 2025 VISAPP study, a Faster R-CNN defect detector’s mAP50 fell from 84.2% on its original test set to 45.6% under simulated particle drift. The same authors caution that subtler drift, such as gradual changes in sensor calibration, may be harder to detect.

We would check that any divergence metric actually predicts the accuracy drop before trusting it. Size the mill-floor validation set from the expected defect prevalence and the confidence interval you need. We would treat colour-space augmentation as a supplement to real sodium-lit images, not a replacement.

A citable framework for a standards board

The citable precedent is pharmaceutical. The Knapp reject-zone method appears in USP <1790>, and A3P describes it as the most commonly used AVI qualification method. Outside pharma, do not assume a standard protocol exists. The neonatal seizure study found exactly that in its own field, with each study defining its own criteria. For a board that has not seen the pattern before, we would present it as an adapted pharmaceutical method and commission an independent audit of the adaptation.

Known-narrative challenge sets for GxP text generation

The structure carries over to a narrative-drafting feature in the form of a known-narrative challenge set: documents seeded with known factual errors, omissions and tone problems, each scored by human reviewers and by the system. The hard part is redefining “defect” so that each error type has its own acceptance criterion. Use URS/FS/DS documents to scope what the challenge set must exercise.

Is there an IQ/OQ/PQ template a two-person validation team can actually run?

No template works unchanged, but a small team does not need a full CSV programme. The core for a bounded inspection system is:

• a scoped URS/FS/DS
• an OQ script built on the known-defect challenge set
• a PQ with statistically justified sampling

The obligations behind it are specific. Inspection software must be validated for its intended use, with documented protocols, verification results and version control. PQ typically involves test method validation under normal use conditions. The lifecycle in the 2025 arXiv paper draws on GAMP 5 Appendix D11, ISO 13485 and Annex 11, and recommends a Dataset Specification Sheet covering class distributions, acquisition setup and labelling process.

The FDA draft guidance lists what a qualification protocol should identify:

• the test sets
• test duration
• grading method
• how results are documented
• acceptance criteria
• actions on failure
• requalification method and frequency

Who can we bring in short-term to set up the qualification framework?

TechnoLynx builds documented, representative defect-library qualification protocols for automated visual inspection systems. We treat it as a release gate for a production AI system, with its own validation artefacts. Building in-house makes sense when qualification will recur often enough to justify the investment. When choosing a partner, ask to see a pre-registered margin and raw per-unit data from past work. Our AI engineering for life sciences and healthcare page describes the engagement. Where a GPU change is part of the project, LynxBenchAI publishes hardware measurements to inform that choice. It does not run your inspection workload.

Some things fall outside this engagement. Approving the qualification protocols remains the quality unit’s role. Whichever route you take, someone in-house has to own requalification and change control after go-live.

What an auditor should be able to recompute from the qualification report

A defensible report lets a reviewer redo the verdict from raw data, without having to trust a summary. Before sign-off, check that it contains:

• the POD for every challenge unit, with the number of manual inspections and inspectors behind it
• the zone assignments, mRZE, aRZE and the aRZE ≥ mRZE result
• the margin, test and reference rule, dated before unblinding
• the power calculation or Knapp table lookup that set the number of machine runs
• within-appraiser, between-appraiser and known-standard kappa
• per-defect-class results, not only aggregates
• two-sample drift test results for any pooled historical data
• the function-set and AQL results that keep the qualification true after go-live

If any line is missing, the report shows the machine passed once under conditions nobody can reconstruct. It does not show the system is equivalent to people, and that difference matters most when the defect mix next changes.