Shadow mode testing of an AI inspector only holds up if the protocol is fixed before the first unit is scored. That means five things agreed in advance: a non-inferiority margin agreed with Quality; a sample size and run length powered to test that margin on each line; a written rule for settling AI-versus-human disagreements; a check that every verdict is paired with the right physical unit and arrives before the reject point; and a pass/fail criterion for each line. If any of these is set after someone has looked at the data, the verdict becomes negotiable, and a cutover decision cannot afford that.
In a silent or shadow trial, the model runs live in its intended setting, in parallel with the standard process, and nobody acts on its outputs. What separates a defensible run from a long demo is the statistics and the plumbing around it, and both have to be designed per line. We treat the run as a reliability artefact, not an accuracy score. Evidence that the system behaves on this line, under this load, against these inspectors, is a different deliverable from a good number on a held-out test set.
Shadow mode testing protocol: what the test plan fixes before day one
The European Commission’s draft Annex 22 on AI in GMP asks for acceptance criteria defined by a process subject matter expert and approved before testing starts. It also asks for an approved test plan covering the metrics, the test data, the test script and how the metrics will be calculated. For an inspection cutover, we would expect that plan to pin down at least the following.
| Element | Fixed in advance | Why it cannot wait |
|---|---|---|
| Non-inferiority margin | The largest acceptable loss in detection against the human baseline, per defect class | A margin chosen after the results can be quietly fitted to them |
| Sample size and run length | Defective and good units needed per class, converted to weeks from line volume | The sample size is planned from the margin, so the margin has to exist first |
| Defect-mix coverage | Challenge units covering every known defect class | Qualification sets are expected to contain all known defects |
| Discordance rule | Adjudicator, decision rule and log fields | The adjudicated outcome must be recorded separately from the original labels |
| Pairing and timing | Unit identity through the pipeline, and a verdict deadline at the reject point | A detection that arrives outside a defined time window can be scored as a miss |
| Per-line verdict | For example, no missed defects and false rejections at or below 2.5% | The IPA best-practice document states automated-inspection acceptance in these terms |
| Rollback triggers | The conditions that revert a line to manual inspection | A trigger defined mid-incident becomes a crisis decision |
How long a shadow run lasts is set by statistical power, not the calendar. It is a power calculation against the pre-set margin, driven by the defect base rate and the confidence required, and the result is divided by the units each line inspects per week. If the true defect rate turns out lower than planned, the run gets longer, so the plan should include a checkpoint where the sample size is re-estimated.
What sample size and run length does each line need to show the AI is not worse?
Each line needs enough defective units in each defect class for the upper confidence bound on the AI’s detection shortfall to fall below the pre-set margin. It also needs enough good units to test a false-rejection limit. Run length is whichever of those two counts takes the line longer to produce.
Why non-inferiority, not superiority
A non-inferiority test asks whether a new method is worse than an active control by more than a pre-stated margin. That margin is the largest difference you will accept in exchange for secondary benefits such as lower cost or burden. This is the real question behind replacing human inspection: nobody needs the AI to beat the inspectors, only to lose no more than an agreed amount. Setting the margin to zero turns a non-inferiority test into a one-sided superiority test. Superiority is the wrong frame: a failed superiority test does not establish parity.
Pharma inspection practice already measures automated inspection against the human inspector. The A3P good-practice paper on visual inspection makes the central element of validating an automated system the demonstration that it is at least as good as a human inspector on detection rates. The USP 〈1790〉 briefing expects equivalent or better performance than compendial manual inspection. The draft Annex 22 says a model’s acceptance criteria should be at least as high as the performance of the process it replaces, which means that performance has to be known first.
Setting the margin the way the FDA sets it
The FDA’s non-inferiority guidance defines two margins. M2 is the largest clinically acceptable loss, usually taken as a fraction of M1. Non-inferiority is shown when the upper bound of the two-sided 95% confidence interval for the control-minus-test difference sits below the margin. This corresponds to a one-sided alpha of 0.025. In the FDA’s reteplase example, pooled streptokinase trials gave a 2.6% mortality benefit, with its 2.1% lower bound taken as M1. A clinical decision then required the new drug to rule out losing more than half of that benefit, an M2 of 1.05%. A typical M2 is often half of M1, partly because the sample needed to preserve a larger share becomes impractically large.
We read M1 on an inspection line as what human inspection catches over no inspection, defect class by defect class. M2 is the detection loss Quality is willing to accept. For critical defects, M2 should be close to zero.
The FDA also warns that a non-inferiority result can only be interpreted if the control actually had its expected effect during the trial (assay sensitivity). For inspection, this means a parity result proves nothing if the shadow period saw few real defects, or if the inspectors were missing them. Running a test set with known defects before and after the run, as the A3P paper describes for routine operation, shows the system’s detection stayed within its qualified limits.
There is another route to a margin when no clean ground truth exists. One neonatal seizure-detection study (2025) derives the margin from the variability among human raters and calls the AI equivalent if it falls within that range.
Worked example: turning a margin into weeks
The detection rate is estimated only on defective units, so defect throughput sets the run length.
- At an assumed 0.5% defect rate for a class: 300 ÷ 0.005 = 60,000 units, or 3 weeks.
- At 0.05%: 300 ÷ 0.0005 = 600,000 units, or 30 weeks.
The rarest class you need to certify sets the run length. In one lot-acceptance sampling study (2025), the required sample size rose from 18 to 303 as the actual defect rate moved from 0.04 to 0.08, closer to the claimed 0.10. The FDA notes that when the control effect is small, a non-inferiority study may need a very large sample or may not be practical at all.
Can the sample size come down without weakening the evidence?
Yes, through the three routes below. Each has to be written into the test plan before the run, not adopted halfway through.
The first is enriching the detection evidence with qualified challenge sets. The A3P paper describes product-specific sets with 10–20% defective containers (and under 30%), containing every known defect, inspected at least 10 times during machine qualification. The IPA best-practice document spreads defects at 5% to 15% of a kit, with at least one rejection per category. The USP 〈1790〉 briefing limits defective units to about 10% of a test set to prevent rejection bias. It assigns units to the reject zone at a manual probability of detection of 0.7 or higher. Challenge sets concentrate defects far above the line’s base rate. The live run still has to show the system works in real conditions, the primary objective of a silent phase.
The second is group sequential testing. Jennison and colleagues set out designs with interim analyses that can stop early for inferiority, non-inferiority or superiority. The price is a larger maximum sample. In one of their examples, a design testing superiority and non-inferiority together needs at most 308 units per arm, against 258 for a fixed non-inferiority design. They also found that adding one more interim analysis matches or beats the gain from adaptive group sizes.
The third is active testing. Kossen and colleagues (ICML 2021) choose which points to label in a way that keeps the test-loss estimate unbiased while reducing its variance. It fits best where adjudicated labels, not units, are the scarce resource.
Any deviation from the approved plan has to be documented, investigated and justified. The FDA also notes that when only one study will be run, a confidence level stricter than 95% (for example 99%) can be required. Cutting the sample so far that a rare but critical class is underrepresented saves weeks and loses the sign-off.
Disagreements inside the baseline: human versus legacy AOI
If the human inspector and the legacy AOI disagree on a unit, settle it with a pre-specified rule before using it to score the AI. The AI’s own verdict must never be the tie-breaker. The FDA’s CADe guidance asks submitters to describe how multiple truthers’ interpretations are combined, and how inconsistencies are handled. It also asks that truthers be different people from the readers in the performance assessment. Both consensus styles have costs. Unanimous consensus gives confident labels but throws away the disagreements, so the share of discarded data should always be reported. Majority consensus keeps everything but carries disagreement into the reference.
One pass/fail verdict per line, with the breakdown attached
A single parity verdict per line should travel with the statistics behind it. The breakdown should give sensitivity, specificity, PPV and NPV alongside any summary metric. It should track false rejects and defect escapes separately, because aggregate accuracy can hide a shift between them. The run should hold enough cases per important cohort, such as a defect class, to give each its own confidence interval. Parity on line 1 says little about line 2 if lighting, products or inspection rate differ. Pharma practice treats light intensity and inspection rate as critical process parameters fixed during qualification. A Nature Health scoping review of silent trials found that performance consistently drops when moving from retrospective to live evaluation. So each line gets its own run.
When the AI and the inspector disagree, who adjudicates and how is it logged?
A named adjudicator applies a fixed rule and records the result in a fixed format. All three are agreed with Quality and approved before the run starts, so a disputed unit becomes a logged decision rather than a debate. Thresholds and acceptance gates set after the results are in invite the organisation to pick a convenient target.
Store each party’s first, independent judgement separately, with reasons, before anyone discusses the case. For each discordant unit, record the original AI label, the original human label, the adjudicated outcome, who decided, any evidence added, the rule clause applied and a timestamp. An inspection exception log adds part and lot identification, station and camera, software or recipe version, the original image and region of interest, and the confidence score. Tiered review gives the rule its escalation path. In a multicentre polyp-detection trial protocol, two pathologists assess each sample, a third reviews any discrepancy, and a central pathologist decides when inconsistency remains.
Leave room for a third outcome. Some images do not contain enough evidence to settle the disposition, and in those cases neither the system’s output nor the operator’s opinion should count as ground truth. Mark them inconclusive and send them to independent review.
Beware the convenient rule that the human call always wins. It keeps the line running, but it cannot show the AI is right on any disputed unit. Under it, every real defect the AI caught and the inspector missed gets scored as an AI false reject. The polyp protocol avoids this by counting the AI’s extra detections as their own endpoint. If the discordance rate is high, classify the reasons before blaming either side, since a disagreement may expose an ambiguous rule or a missing document.
Is the AI verdict computed before the unit reaches the reject point?
Only measurement can answer this. Tag each unit at capture, carry that identity through batching and inference to the verdict, and compare the tail of the verdict latency with the unit’s transit time from camera to reject gate.
LynxBenchAI publishes device measurements that show which inference hardware leaves enough latency headroom, so the first step is to check the candidate devices there before tracing the line’s own pipeline on site.
When the question is which inference hardware leaves enough latency headroom, LynxBenchAI is the measuring tool: its published results compare devices, while the line’s own pipeline still has to be traced on site.
TechnoLynx does this as engineering work on the target hardware. The team defines the identity requirements linking units, frames and verdicts for a batched, multi-camera pipeline. It instruments the capture, batching and inference stages without materially distorting latency. It then confirms pairing integrity and verdict arrival against transit time, and leaves the tracing in place as the line speeds up.
Batching is where the risk enters. In the open-source vision-rt pipeline, cross-camera batching reached 2,264 fps across 32 cameras on one RTX 5070 Ti, 34% more than one context per camera. Its author also notes that live cameras only get batches of about one unless the batcher waits. The throughput gain is paid for in waiting time. NVIDIA’s DeepStream documentation for nvstreammux describes partial batches, and warns that some downstream plugins can add latency when more than one frame per source is batched. One multi-view vehicle inspection system routes 11 synchronised cameras to task modules and reports 285 ms latency end to end.
Two rules follow. First, compare the p99 verdict latency with transit time, not the mean, because the unit that slips past the gate is the one in the tail. Second, write the deadline into the scoring rule. The polyp protocol counts an AI detection only if it appears within a one-second window. On a line, a verdict that arrives after the reject point should count as a miss. Pairing can also break for reasons that have nothing to do with batching, such as clock skew or dropped frames between cameras, and profiling overhead can distort the timings it measures. Timing integrity says nothing about whether the verdict is right. Showing that the deployed engine still matches the one you validated is a separate task, covered in qualifying and re-verifying the edge inspection stack.
Who can verify our pipeline’s pairing and timing, and should we do it in-house?
TechnoLynx verifies a pipeline’s pairing and timing so that the shadow-mode comparison rests on correctly paired, on-time data. An in-house team can do this work if it has GPU pipeline profiling skills and time on the target hardware. A specialist makes sense when batching, multi-camera fusion and edge deployment are new to the team at once. When choosing a partner, ask whether they will instrument your actual pipeline rather than quote vendor figures. Our AI engineering for manufacturing and automotive work covers the pipeline side. For hardware choice, LynxBenchAI publishes device measurements, but it does not run your inspection workload.
The engagement covers the pipeline, not the quality decision itself. The process subject matter expert defines the acceptance criteria, documented and approved before testing starts. The statistical design can be done by an in-house statistician.
After cutover, which verdicts should go to human review instead of automatic reject?
Route by risk. A unit is auto-rejected only when model confidence is high and an independent disagreement signal is low. Everything else goes to a person, with thresholds set per defect class on your own labelled data, and a sample of automatic decisions is audited on a schedule.
TechnoLynx frames this with the plant: it defines defect severity classes, acceptable miss and false-reject rates, and reviewer capacity. It then builds a second-pass check, such as cross-camera agreement, on top of the existing model. It runs that check on the edge hardware within the line’s latency budget, and validates candidate thresholds on held-out labelled data before deploying them with drift monitoring.
A single global confidence cut-off is too blunt. A multi-camera rig already produces per-view detections with their own confidence scores that can be compared. The second pass reduces reviewer load but does not replace review. When the model consistently confuses two defect classes, every view and every pass agrees, and agrees wrongly. Review intensity should follow consequence, reversibility and novelty. The draft Annex 22 notes that, depending on criticality and how much the model was tested, a human-in-the-loop process may require review of every output. In one silent trial in the Nature Health review, clinical stakeholders selected the decision threshold (75% PPV) themselves. That is the right owner for the choice. The A3P paper describes batch-to-batch revalidation through the manual AQL comparison. Thresholds that are too tight send most units to review and defeat the purpose of automating. Thresholds that are too loose let escapes through.
If one line fails parity, can the passing lines cut over while it stays in shadow mode?
Yes, provided the lines really are independent. Cut over the lines that pass, keep the failing line on manual inspection with the AI still in shadow, and extend its run under a documented plan. The silent phase exists so teams can decide on local evidence whether to discard, improve or deploy a model. One trial in the Nature Health review planned four months and extended to six.
Independence is the catch. If all three lines feed one GPU server that batches frames across cameras, line 2 is not isolated: its frames share the queue and its fixes ship in the same build. Before claiming partial cutover is possible, check that each line can be switched, rolled back and versioned on its own.
Rollback needs the same advance agreement as the parity test. Define the triggers, such as a defect escape found in AQL sampling, false rejections above the agreed limit, or a set number of downstream complaints. Keep the manual inspection crew staffed and qualified for the whole trial so that reverting takes hours, not a hiring cycle. Pharma practice offers a useful model: reinspection runs only under a Quality-approved procedure that limits how many times it may be repeated. Extending shadow mode on one line deserves the same limit, because a parallel run costs inspector time and cannot go on indefinitely.
Is a benchmark or challenge set enough before handover?
No. A fixed challenge set calibrates the system’s detection against known defect standards. The gating evidence is a shadow run against the target environment’s own live data. The silent-trial literature finds performance drops from retrospective to live evaluation, and passing an offline set does not rule that drop out.
Challenge sets give you the rare-defect evidence. The live run on each line gives you the false-rejection rate, the timing and the disagreements that decide whether the line is ready. A plant that has both, gathered under a plan signed before the first unit, can cut over one line and hold back another without anyone reopening the argument.
