AI inspection equivalence testing comes down to one design: measure the inspectors you have now, fix an adjudicated reference standard, pre-register a non-inferiority margin and a paired test, then run machine and humans on the same units until you have enough defect-positive cases to reach the planned power. A high agreement percentage between machine and inspectors does not show the machine is correct. FDA’s statistical guidance on diagnostic tests is explicit that agreement does not mean correct, and that agreement figures change with the prevalence of the condition in the study set. Regulators set the bar the same way: the EU’s draft GMP Annex 22 says an AI model’s acceptance criteria should be at least as high as the performance of the process it replaces, so that performance has to be known first.
The order of work for an AI inspection equivalence study
Each step produces something the next step depends on, which is why the order matters more than any single statistic.
- Baseline the human process. Run an attribute agreement study that gives each inspector’s miss rate, false-alarm rate and agreement, and the agreement across inspectors.
- Fix the ground truth. Have more than one subject-matter expert confirm the status of every unit in the test.
- Set the margin and the test. Choose the non-inferiority margin before any results exist.
- Size the run. Work out the sample size from the primary variable, the test statistic, the hypotheses and the type I and type II error rates.
- Lock the plan. Write the intended use, metrics, acceptance criteria, test data reference and test script into a test plan that is approved before testing starts.
- Run, analyse, report. Report the 2x2 table and the accuracy or agreement measures with two-sided 95% confidence intervals, and document and justify any deviation from the plan.
How good are our inspectors today, in numbers?
The human baseline comes from an attribute agreement study, the pass/fail form of Gage R&R: several inspectors judge the same set of good and bad units repeatedly, independently and in random order, and the results are scored for effectiveness, miss rate, false alarm and kappa.
LynxBenchAI measures how AI hardware runs the public PyTorch stack, which helps choose the device the inspection model runs on; it does not score a team’s model against its inspectors.
Designing the attribute agreement study
Capvidia’s 2023 MSA guide recommends three appraisers (two at minimum) and 50 parts (30 at minimum), with every part judged three times by each appraiser. Its acceptance table rates an appraiser as acceptable at 90% or higher effectiveness, a miss rate of 2% or lower and a false-alarm rate of 5% or lower, with kappa at 75% or above. ASTM F3263 frames the same idea as two error rates: the primary criterion is that the escape rate β (accepting a non-conforming unit) stays at or below its threshold, and the secondary criterion is the false-alarm rate α (rejecting a conforming unit). F3263 runs this as a single blind study with two sampling plans, one made up entirely of non-conforming units for the escape rate and one made up entirely of conforming units for the false-alarm rate.
A few design details decide whether the baseline holds up. F3263 recommends three or more operators so that reproducibility is not overestimated, and notes that adding inspectors adds trials and statistical power without needing more unique parts. Inspectors should know the product, because unfamiliar inspectors make the test overly conservative and tend to raise false positives.
Kappa is the proportion of times appraisers agree, corrected for chance, divided by the most they could agree after the same correction. Minitab’s documentation cites the AIAG view that a kappa of at least 0.75 shows good agreement, with values around 0.90 preferred.
Two caveats apply to any baseline. A small or unrepresentative panel under-powers both the agreement study and the equivalence test that follows it. Inspector performance also drifts, which is why the USP <1790> draft expects human inspectors to be requalified at least once a year.
Should field returns stand in for the manual miss rate?
No, not as the primary baseline. The USP <1790> draft gives returns a more useful job: characterising particles found in returned product helps explain how a defect got past the original 100% inspection. Use returns as a diagnostic signal, and use the agreement study as the number.
Turning the baseline into a machine-versus-human benchmark
The fair comparison puts the model through the same units, against the same reference, as one more appraiser. For an automated system that works with no inspector involvement, F3263 says one inspector is enough, but recommends controlling other sources of variation such as lighting, temperature, humidity, shift and part orientation. Annex 22 lists suitable metrics for an accept/reject model as including a confusion matrix, sensitivity, specificity, accuracy, precision and F1 score.
“At least as reliable as humans” should then mean three separate things, each with its own pre-agreed limit: machine sensitivity no worse than the human baseline by more than the margin, a false-reject rate within its own limit, and kappa against the reference reported as supporting evidence rather than as the pass criterion. Which units go into the set, and in what proportions, is a design problem of its own, covered in our guide to designing a regulator-defensible known-defect challenge set.
When certified inspectors disagree on the same unit, what counts as ground truth?
Ground truth for an equivalence study is a reference standard with an inspectable label history, raw reader labels preserved after adjudication. FDA defines a reference standard as the best available method for establishing whether the target condition is present, and says it can be one test or a combination of methods.
The distinction matters because of what happens without one. If the model is compared only with the inspectors’ calls, FDA’s guidance says the results should be reported as positive and negative percent agreement, not sensitivity and specificity. Worse, if the new test is better than the non-reference standard, agreement will be poor, and no statistical method can tell that case apart from a genuinely inaccurate new test. An AI system that catches defects your inspectors miss can therefore show poor agreement with them.
Majority vote, adjudicator, or a repeatability study on the raters?
When no single test can settle the target condition, an expert panel is often used as the reference standard, according to a 2024 scoping review of diagnostic accuracy studies. The same review found that at least 8% of the studies it reviewed did not assess all participants the same way (differential verification), which is the trap to avoid when some units get a lab test and others get a panel vote.
The choice between methods depends on the pattern of disagreement and on the stakes:
- Majority vote can hide a bias that several inspectors share.
- An independent adjudicator (a fourth expert, or a lab or metrology method) is the safer choice for critical defect classes.
- An attribute agreement study on the raters themselves helps find why they disagree, because it separates each appraiser’s repeatability from the reproducibility between appraisers.
F3263 prefers more than one subject-matter expert to confirm each sample. It also warns that a sample the experts cannot agree on as good or bad will only cause problems in testing, and recommends samples that are clearly “just passing” or “just failing” instead. Do not force every case into positive or negative: define which cases are excluded, deferred, kept as uncertain or resolved with more evidence. Annex 22 expects test labels to be verified to a very high degree of correctness, for example by several independent experts, validated equipment or laboratory tests.
Two rules keep the reference honest once the study is running. FDA’s guidance says resolver results must not be used to revise the original 2x2 table, because the revision rests on unverified assumptions, and the apparent agreement from a revised table is always greater than or equal to the original. For a locked evaluation set, define who may access cases, how corrections are authorised and whether model output is hidden from readers, and keep every post-hoc correction in an audit trail.
Can we trust the labels on an 18-month-old reference panel?
Not without re-verifying them first. Set the re-grading trigger from the panel’s own risk: elapsed time, heavy handling, a line change or a change to the defect criteria. F3263 notes that changes to visual design or process standards need evaluating for their impact, and says an attribute validation should run on the most recent visual standards.
Whatever method re-verifies the panel (a metrology lab, a coordinate measuring machine, a fresh expert panel) needs a known accuracy better than the tolerance the panel claims. Keep the original labels alongside the re-graded ones, so that every metric can be traced back through the full label history.
How long and how large must the shadow-mode parallel run be?
A shadow run is long enough when it has collected the number of defect-positive units the pre-registered power calculation asks for. The calendar matters only as the rate at which defects turn up.
TechnoLynx can run this phase as release-readiness validation within its Production AI Reliability work: setting the margin and defect base-rate assumptions, running the inspection model alongside the inspectors without touching their decisions, and producing the equivalence analysis and go/no-go evidence.
LynxBenchAI does not size the run or score the model against your inspectors, but its published results show how each AI accelerator runs the public PyTorch stack, which helps when choosing the device the inspection model will run on during the shadow run; the first step is to check those results for the devices you are considering.
Same inputs, no feedback, rollback agreed in advance
Shadow mode means the AI runs in parallel with the existing human workflow, processing the same inputs without influencing outcomes, while humans keep making the decisions. A vendor write-up from Brightlume sets out the architecture: every input is mirrored exactly, outputs are logged and timestamped, and the model must not see the human decision in real time, because that feedback would contaminate the validation data. The blinding should run both ways, with inspectors never seeing the model’s call. Rollback triggers belong in writing before the run starts, not after the first bad day.
Vendor agreement thresholds do not replace a pre-registered margin, because agreement does not mean correct and sensitivity is estimated only from defective units.
Worked example: how sample size moves with margin and pairs
NCSS’s PASS documentation tests equivalence of two correlated (paired) proportions by checking whether a 100(1−2α)% confidence interval for their difference lies entirely inside the equivalence bounds, an approach it says regulators often recommend. Its tables show how sensitive the required size is to the assumptions:
| Scenario from the PASS tables | Pairs | Power |
|---|---|---|
| Both methods 80% accurate, ±5-point bounds, discordance input 0.05 | 468 | 90% |
| Same, discordance input 0.10 | 881 | 90% |
| Both 50%, ±10-point bounds | 50 | 2.6% |
| Both 50%, ±10-point bounds | 200 | 86% |
Sensitivity is estimated only from units that actually have the defect, so a sensitivity margin needs defect-positive pairs, and the production volume needed is roughly those divided by the defect rate. On a line with a low defect rate, we would let a seeded challenge set supply most defect-positive pairs, with live production confirming behaviour on real variation. The method is covered in our piece on qualifying automated visual inspection against human inspectors with known-defect challenge sets.
What to do when ten days cannot reach power
Run the power calculation before the window opens. If ten days of production cannot supply enough positives, there are four options: extend the shadow run, enrich it with a seeded challenge set, make a lower-confidence decision that is flagged as such to the customer, or keep manual inspection until the data exist. What is not an option is stopping early because the numbers look good: ICH E9 says trials should stop early only for ethical reasons or if power is no longer acceptable, with any interim analysis planned before unblinded data are seen. A short run that “found no significant difference” proves nothing either, because ICH E9 calls it inappropriate to conclude equivalence from a non-significant test of no difference.
The customer’s own threshold for a convincing result matters too. Agree the threshold with them in writing alongside the power calculation, so the two cannot drift apart later.
Borderline disagreements during the run
Send borderline cases through the same adjudication protocol as the ground truth, logged separately from the primary table. Watch the rate as well as the individual cases: a cluster of borderline disagreements in one defect class can show the model is not ready even when overall sensitivity looks fine. ICH E10 adds a human-side warning: when everyone knows all arms are active, there is a tendency to classify borderline cases as successes, which shrinks observed differences and flatters non-inferiority.
What evidence does a pharma or medical-device regulator expect from the study?
In pharmaceutical visual inspection, the expectation is evidence that the automated system meets or surpasses human inspection. FDA’s guidance on visible particulates says automated machines should be validated to that standard, using test sets prepared and approved by quality assurance in which the identity of defective units is masked from those being tested. The quality unit should also approve qualification protocols that define the test sets, grading method, acceptance criteria and actions for failures.
The Knapp method: non-inferiority in pharma inspection
The USP <1790> draft treats visual inspection as probabilistic, and says automated equipment should be validated against the compendial manual inspection process, with an expectation of equivalent or better performance. The Knapp-Kushner method makes that measurable by comparing manual and automated inspection results. Each defect standard is given a probability of detection (POD) by repeated manual inspection, and the draft suggests 30 to 50 inspections per container, ideally by several inspectors. In an A3P 2023 case description, a test kit with 10% defective units was inspected ten times by each of five inspectors, giving 50 results per unit.
Units are then sorted by POD: below 30% is the acceptance zone, 30% to 70% the grey zone, and 70% or more the reject zone. The manual reject zone efficiency (mRZE) is the mean detection rate of reject-zone units, and the automated system passes when its automatic reject zone efficiency on those same units is at least as high: aRZE ≥ mRZE. The Knapp-Abramson tables give the minimum number of reject-zone inspections needed to show this at 95% confidence, with the minimum number of test-kit inspections calculated as I = N/RZN. The draft limits defective units to around 10% of each test set to prevent rejection bias.
One limitation is easy to miss. A 70% POD threshold means particles at that size can still be missed up to 30% of the time. Matching mRZE means matching a human process that is itself imperfect, which is why the baseline work above has to be done carefully.
What Annex 22 adds for AI models
Annex 22 asks for test data that represent the full sample space of the intended use, stratified and including all subgroups and rare variations. The test set and each of its subgroups must be large enough to calculate the metrics with adequate statistical confidence, and technical or procedural controls must keep test data out of development, training and validation. FDA’s diagnostic guidance makes the same point about difficulty: leaving out intermediate, hard cases gives an overly optimistic picture of performance.
Outside inspection, thresholds for a device such as a sleep-apnoea detector, and the detail expected about training choices, are questions to put to the relevant authority directly rather than answer from another domain’s template.
What non-inferiority margin on sensitivity is defensible for a critical defect class?
The margin is the largest loss of sensitivity that can be judged acceptable, specified in the protocol before the study runs. A margin chosen after the results are known is not sufficient to show non-inferiority, according to FDA’s non-inferiority guidance.
FDA’s guidance states the hypotheses as H0: C − T ≥ M against Ha: C − T < M, where C is the control, T the test and M the margin. In an inspection study, C is the human baseline sensitivity and T the machine’s. Non-inferiority is shown when the upper bound of the two-sided confidence interval for C − T falls below M. Using a 95% interval corresponds to a one-sided alpha of 0.025.
The guidance separates two margins: M1 is the entire effect of the active control, and M2 is the largest clinically acceptable difference. For inspection, M1 corresponds to everything the human process catches compared with no inspection at all, and M2 is the share of that you are prepared to give up, which for a critical defect class should be very small. ICH E10 says the margin cannot exceed the smallest effect the control would reliably be expected to have, and that it rests on both statistical reasoning and clinical judgement.
FDA’s non-inferiority guidance cites 10% to 15% margins as typical in antibiotic trials, and a 2022 arXiv analysis of binomial non-inferiority methods studied 10% and 15% margins, noting FDA guidances that justify 10% or 12.5% from historical evidence. Those margins come from antibiotic and clinical-response trials, so they are context rather than a target for a defect class that can harm a patient. The USP <1790> draft classes a defect as critical when it may cause a serious adverse reaction or death, and we would agree that margin with QA and the regulator. No statistical method determines it on its own. False-reject rate deserves its own, usually looser, limit, because it is mainly a cost problem rather than a safety one.
Which paired test should we use: McNemar-based, Tango or exact?
Use a paired test, because every unit is judged by both the machine and the inspectors. NCSS’s documentation separates the two forms: equivalence is two-sided (accuracy about the same), and non-inferiority is one-sided (the new procedure is no worse than the reference). One paper on equivalence tests for paired data critiques earlier McNemar-based sample-size work on the equivalence of sensitivities. The same paper argues that a small probability of discordance is not, on its own, evidence of equivalence. McNemar-type tests assume matched pairs, so a design where machine and humans see different units needs an unpaired two-proportion method instead.
The choice of method also affects type I error. The 2022 arXiv analysis warns that some asymptotic methods are anti-conservative, meaning they fail to hold the nominal error rate. For independent groups, it found an exact-based method stayed closer to the nominal level than the popular asymptotic ones.
| Your situation | Design choice |
|---|---|
| Machine and inspectors judge the same units | Paired non-inferiority test on the difference in sensitivity, one-sided alpha 0.025 |
| Goal is “about the same”, in either direction | Two-sided equivalence: the interval must sit inside both bounds |
| Machine and humans see different units | Two-proportion method for independent groups, chosen so it holds type I error |
| Only inspector calls available, no reference standard | Report positive and negative percent agreement only; no accuracy claim |
| Pharma particulate inspection | Knapp reject-zone comparison, aRZE ≥ mRZE, sized with Knapp-Abramson tables |
The fourth row follows FDA’s agreement guidance, and the last row follows the A3P description of the Knapp method. The other rows are our reading of the paired-test literature above.
How do we pre-register the analysis plan so an auditor cannot call it post-hoc?
Write the full analysis into an approved protocol before anyone sees unblinded results, and follow it. ICH E9 says the design and principal features of the analysis should be specified in a protocol written before the trial begins, and that adherence to it determines how much confidence the results deserve. The protocol should state clearly that non-inferiority is its explicit intention and specify the margin. Redefining the primary variable after unblinding will almost always be unacceptable, and only analyses foreseen in the protocol count as confirmatory. Criteria for excluding units from the per-protocol set should be documented before the blind is broken. ICH E9 notes that the full analysis set is not conservative in a non-inferiority trial, and advises planning both a full and a per-protocol analysis. One-sided tests carry half the conventional two-sided alpha.
If you also want to claim superiority, plan for it now. With one primary endpoint, FDA’s guidance allows a non-inferiority trial to go on to test superiority without inflating type I error. In practice, we would version, sign and timestamp the plan, store it where the test team cannot edit it, and have an external statistician review it before the run. The audit-trail controls around the data itself are a governance question of their own, and they sit alongside this plan rather than inside it.
Who can run an AI inspection equivalence study with us, and should we build it in-house?
TechnoLynx builds the engineering side of an AI inspection equivalence study: the shadow-mode harness that runs your model next to your inspectors at line speed, the power and sample-size work behind the margin, and the analysis and evidence package your quality team signs against. This is the release-readiness validation that our Production AI Reliability work covers, treated as an engineering discipline separate from model accuracy. The output is evidence a quality director or auditor can read and defend, built as an engineering artefact.
The limits are firm. We do not give regulatory sign-off, safety certification or legal advice, and we do not replace your in-house QA function. A team that already has a statistician experienced in measurement system analysis, and an inspection model that already runs at line throughput, can reasonably run the study itself and bring in an external statistician to review the plan. For the margin on a critical defect class and the evidence format, the regulator or notified body remains the authority to consult, whoever does the engineering.
FAQ
What is the difference between measurement system analysis and Gage R&R?
Measurement system analysis assesses how much variation the measurement process itself introduces, looking at the device, the people and the method. A Gage R&R study estimates how much variation comes from equipment (repeatability) and how much from operators (reproducibility).
What is the difference between the chi-square test and McNemar’s test?
The chi-square test of independence compares proportions between separate groups, while McNemar’s test compares two paired measurements taken on the same subjects, using only the discordant pairs. A study where machine and inspectors judge the same units is paired, so McNemar-type methods fit it and a plain chi-square test does not.
