A signed-off threshold is valid only under the conditions it was validated under. Keeping it trustworthy therefore takes a documented, change-triggered revalidation process, not a one-off qualification. Whether a change needs a bounded recheck or full requalification depends on its assessed impact on the validated state. GHTF process validation guidance (2004) says a new piece of equipment repeats installation qualification while most of the operational qualification stands. The EU’s draft Annex 22 (2025) asks that any change to an AI model, its system or the physical objects it uses as input be evaluated for retest, and that any decision not to retest be fully justified. The parts carry over between domains: a trigger list, a golden-sample reference set with pass/fail criteria, monitoring between changes and a rollback rule. What differs is the statistic being protected.

How do we pick a cost-weighted operating point on the ROC curve?

The cost-weighted operating point is the threshold that minimises expected cost, given what a false reject and a missed defect each cost. It is not the threshold that maximises accuracy. Each threshold applied to a scoring classifier produces a different point in ROC space, so choosing a threshold means choosing a point on the curve. Elkan (2001) showed that, in the two-class case, standard learners make their decisions at an implicit probability threshold of 0.5. He also showed that a 2x2 cost matrix has essentially one degree of freedom from a decision-making point of view. If correct decisions cost nothing, the optimal threshold is p* = C_FP / (C_FP + C_FN). Assume a missed defect costs twenty times as much as a false reject: then p* = 1 / (1 + 20), which is about 0.048.

That formula assumes the scores behave like calibrated probabilities. Hernández-Orallo, Flach and Ferri contrast two of the threshold choice methods they study. The score-driven method trusts the scores as probabilities. The optimal method trusts only the ranking and reads the threshold from the slopes of the ROC convex hull built on validation data. Elkan makes the same practical point: when scores are ranked correctly but poorly calibrated, determine the threshold empirically.

This matters later for revalidation. A signed-off threshold encodes two things: a cost ratio and a score distribution. A camera swap that shifts the score distribution changes the expected cost at the same threshold value, even if nobody touched the cost ratio. The calibration study that sets this baseline is covered in calibrating and defending an AOI model’s decision boundaries.

Which changes should force an unscheduled threshold re-check?

Any change that could affect the system’s output should trigger a documented change-impact assessment. That includes the camera model, lens, lighting, firmware, the inference hardware or runtime, the model itself and the physical objects the model inspects.

TechnoLynx builds this as a standing procedure, not a one-off. TechnoLynx works out which change categories need their own trigger and scope, and defines the golden-sample set and pass/fail limits for each station type. TechnoLynx then embeds the checklist in the maintenance work-order and change-control flow, and runs it after a real hardware change to confirm it catches known failures, such as false-reject spikes, before the line is re-enabled.

The trigger list should cover at least these categories:

• Lighting. LEDs lose 20 to 30 percent of their output over their lifetime. iFactory’s lighting checklist reserves 20 percent intensity headroom at commissioning and investigates any drift beyond 5 percent. A lighting retrofit almost always exposes camera position and mount decisions that also need revising.
• Ambient and seasonal light. A station commissioned in winter can meet low spring sun through a skylight that never appeared in its training images.
• Camera geometry. Cognex’s patent on calibration validation notes that camera position and lens distortion change on a production line through thermal expansion, vibration and accidental focus changes.
• Inference stack. ONNX Runtime’s documentation says TensorRT engine caches must be invalidated when the model, the ONNX Runtime version, the TensorRT version or the hardware changes.
• Materials and suppliers. GHTF guidance says that when a raw material supplier changes, parts of OQ and PQ may need to be redone, because the interaction with the process may not be fully understood.
• Quality trends. GHTF lists negative trends in quality indicators as a revalidation condition, alongside design changes and transfers between facilities.

No trigger list will be complete. Annex 15 (2015) asks that the possibility of small changes over time be assessed. Monitoring, covered below, catches what the trigger list misses.

Bounded change-control check or full requalification?

The impact of the change sets the scope. FDA’s 2011 process validation guidance says that, depending on how a change might affect product quality, additional design and qualification activities could be warranted. It also requires each change to carry a description, a rationale, an implementation plan and quality unit approval before it is implemented. Annex 15 asks that the scope of qualification be decided by a justified, documented quality risk assessment. In unregulated manufacturing, a functional acceptance test equivalent to OQ may be enough, while FDA-, USDA- or IATF 16949-regulated lines need full IQ/OQ/PQ documentation.

For automated optical inspection on board assemblies, IPC-9716 sets requirements that cover threshold limits, measurement system analysis, maintenance and verification protocols.

Scoping a change: a decision table

Change	Typical scope	Basis
Like-for-like camera, lens or light replaced, same product	Bounded: repeat installation qualification for the new part; repeat PQ elements as the impact requires	New equipment repeats IQ; most OQ stands; some PQ is repeated depending on impact
Camera firmware update	Evaluate for retest and document the decision	Any change to the system should be evaluated for retest (draft Annex 22)
Lighting geometry retrofit (dome, backlight)	Revise camera position, working distance and fixturing alongside the lighting	Retrofits expose camera and mount decisions
Product or board design change that affects the process	Revalidation, scope set by the impact assessment	Design changes that affect the process are listed as a revalidation condition
Material or supplier change	Parts of OQ and PQ	Interaction with the process may not be understood
TensorRT or ONNX Runtime upgrade on the same device	Rebuild the engine, then evaluate for retest	Engine caches must be cleaned up and rebuilt when the runtime or TensorRT version changes; any system change is evaluated for retest (draft Annex 22)
New edge module	Rebuild the engine, calibrate INT8 on the target and validate on the device	Engines should be validated, and INT8 calibrated, on the deployment device
Model retrain or update	Quality gate before serving, compared against the previous version with tight thresholds	Google’s ML Test Score
Line relocation or site transfer	Revalidation evaluated and documented, scope set by impact	Transfers between facilities are listed as a condition that may need revalidation

We would keep the lighter procedure for hardware-only changes. A product or board design change still needs the fuller path, however much schedule pressure there is to avoid it.

What a golden-sample set needs to keep requalification consistent

A golden-sample set should be fixed and versioned, so every requalification compares like with like. FDA’s draft guidance on visible particulates describes such a test set: a mixture of good and defective units, approved by quality assurance and drawn from a library built from lifecycle samples, simulated defects and purchased samples. The library should include examples from the lower detection limits found in threshold studies. Draft Annex 22 asks that test data be stratified, include all subgroups and reflect common and rare variations.

iFactory’s guidance for automotive vision adds operating conditions. It recommends capturing every shift and season, every paint colour, images from before and after lamp changes, and parts from every supplier. It also says to keep augmented images out of the golden set and to record the lighting state with each model version. We would also weight the set towards parts whose scores sit close to the threshold, because those are the decisions that flip first when conditions change.

We treat a golden panel as a valid shortcut only once it has been shown to agree with full-dataset performance. To qualify it, run the panel and the full labelled set on the same baseline and confirm that false-accept and false-reject rates match within agreed bounds. For pass/fail agreement with the reference classification, Minitab’s guidance cites the AIAG view that a kappa of at least 0.75 indicates good agreement, with values around 0.90 preferred.

Can a Gauge R&R study from the previous variant be reused? Only if the parts being measured have not changed. The NIST/SEMATECH e-Handbook frames the question for a new camera setup: a gauge study should test whether gauges agree with one another and whether that agreement holds across artifacts and over time. Its design uses more than two check standards measured on more than two days for each gauge. It estimates bias as each gauge’s consistent difference from the average across gauges. A replacement camera on an unchanged variant can therefore often be handled as a gauge-to-gauge bias comparison on check standards rather than a full repeat study.

Worked example: a reference-object recheck instead of full recalibration

Cognex’s patent US9734419B1 describes a camera-calibration check that we would use as the model for threshold rechecks:

1. At commissioning, store the calibration parameters and the statistics of the discrepancies between where features were found and where they were expected.
2. Later, image a calibration object of known pattern and compute the same discrepancies again.
3. Set the acceptance threshold from the stored statistics. One example in the patent is the mean of the calibration discrepancies plus three standard deviations.
4. If a discrepancy exceeds the threshold, flag recalibration and identify which camera needs it, so the effort goes only to that unit.
5. Optionally, mount the object permanently in the scene, where it is visible whenever no part is present, and run the check at set intervals or on events.

What makes this robust is that the pass limit comes from baseline statistics recorded at sign-off, not from someone’s judgement on the day of the change.

Does moving inference from CPU to GPU, or to a new edge device, mean revalidating the threshold?

Yes: the new path needs at least a documented check on the deployment device. In most cases, though, it does not mean restarting OQ from scratch. The GHTF logic for new equipment (repeat IQ, keep most of OQ) depends on the new path producing equivalent decisions, and that is exactly what an inference change can break. PyTorch’s documentation states that results may not be reproducible between CPU and GPU executions, even with identical seeds, nor across releases or platforms. Shanmugavelu and colleagues show that floating-point non-associativity causes run-to-run variability. They note it can mask errors in threshold-based correctness tests and can prevent certification for commercial applications.

Bit-for-bit identity is therefore the wrong bar for a GPU path. The right bar is equivalent decisions on the golden set within a tolerance agreed in advance, with the band of scores near the threshold examined closely. Draft Annex 22 asks for configuration control of a tested model and for measures that detect unauthorised changes. Pinning the GPU, driver and runtime versions in the validated configuration follows directly from that. We would not rely on a contract clause fixing the hardware bill of materials for the life of the system. A pre-agreed equivalence pathway for future swaps holds up better.

Re-quantising a TensorRT model for new edge hardware

TensorRT engines are built for specific settings and specific GPUs, and they are not portable across devices. Ultralytics’ Jetson guide says that Orin modules sharing the same architecture are not automatically portable. It advises validating every engine on the deployment device and running INT8 calibration on the target device. One integration guide reports that an engine built on an AGX Orin will not load on an Orin NX.

Guidance on reusing calibration data across devices is mixed. NVIDIA’s TensorRT documentation says a calibration cache is portable across devices when calibration happens before layer fusion, and is generally not portable across TensorRT releases. Ultralytics notes that TensorRT 11 removed the IInt8Calibrator interface altogether. Given that narrow window, we would calibrate on the target and treat the result as a new artefact to validate.

Three practical details reduce the burden. Calibration is deterministic when the same inputs are given in the same order on the same device, so a versioned calibration set gives the same scales on that device. NVIDIA’s experiments suggest around 500 images are enough to calibrate ImageNet classification networks. Some layers cannot run on the DLA and fall back to the GPU. We treat keeping rare defect classes accurate under INT8 as a separate quantisation problem from this procedure. The wider runtime side is covered in our article on edge deployment and runtime qualification for machine vision.

Sign-off, evidence and the work order

Annex 15 defines change control as a formal system in which qualified representatives of the relevant disciplines review changes that might affect validated status. For a vision station, we would have the maintenance technician run the checklist, a process or quality engineer review the golden-set result, and the quality unit approve where the line is regulated. iFactory’s lighting checklist shows the evidence standard: every phase produces a saved image, a lux reading or a signed line item, and the installer and line supervisor sign at completion.

The record should hold:

• the trigger
• the change-impact assessment
• golden-set results compared with the baseline
• the versions of model, engine, driver, firmware and lighting state
• the decision, including a justified decision not to retest

The work order should not close without the requalification record attached. How that record is packaged for an auditor is covered in our article on validating models before they go live, from risk tiering to evidence packages.

Six months after sign-off, how do we know the threshold still holds?

You know by monitoring the model’s performance metrics and whether its inputs remain within the validated sample space. Triggered rechecks and a calendar backstop cover what monitoring might miss. FDA calls this stage continued process verification: ongoing assurance during routine production that the process stays in a state of control, which requires a system for detecting unplanned departures.

TechnoLynx sets up this loop under its Production AI Reliability offer. That covers drift metrics tied to your validated escape rate, triggers that invoke the revalidation SOP, and tests showing the triggers fire under simulated hardware and model changes. The work starts by framing the problem against your existing model and threshold baseline.

Draft Annex 22 asks for regular monitoring of model performance to detect changes such as deteriorating lighting. It also asks for monitoring of whether inputs are still within the model’s sample space. Once something has fired, how large the change is decides between full recalibration and a light check. Ultralytics makes the same distinction for retraining: incremental updates for gradual change, full retraining when the data has changed drastically.

Can monitoring tell us the escape rate has drifted from what we validated?

It can, though we would not rely on escape rate alone. Breck and colleagues at Google (2017) recommend setting quality thresholds from the bounds seen at launch and notifying a responder as soon as quality drifts outside them. They also note that validation data is always older than serving data.

Leading signals close part of that gap:

• Input and score shift. Rabanser and colleagues (NeurIPS 2019) note that ML systems tend to fail silently under distribution shift. They found that shift detection based on a classifier’s own outputs worked surprisingly well across a broad set of shifts.
• Reference targets. iFactory measures a fixed target’s brightness, contrast and sharpness every shift against its commissioning values. Drift raises a maintenance alert, not a retraining job, which separates hardware faults from model faults.
• Control charts. GHTF guidance treats a point outside control limits as evidence that the process has changed and needs investigation.

Breck and colleagues say alert thresholds need careful tuning to balance false positives against false negatives so that alerts stay actionable. They also distinguish slow degradation over many versions from sudden drops in a new version. This is why we treat production AI reliability as an engineering discipline separate from model accuracy: the deliverable is a set of artefacts that catch regressions before customers do, not a dashboard.

Rollback belongs in the same plan. Rollback criteria should be agreed before any change: revert to the previous model, engine and threshold together, or route decisions to human review while the recheck runs. For model updates, the new version should run in shadow alongside the old one, with decisions compared before switching. This only works if the workflow was designed to tolerate faults and the organisation accepts human review.

Who can set up change-triggered revalidation and drift monitoring for a signed-off threshold?

TechnoLynx keeps a signed-off threshold valid through hardware, firmware and model changes. That means the trigger list with a bounded scope for each change type, the golden-sample set and pass/fail limits, drift metrics tied to the escape rate you validated, and the rollback rule.

On building in-house versus bringing in a specialist: if your team already owns the model, the inference stack and a working change-control system, most of the remaining work is embedding. When choosing a partner, ask how they would prove a trigger fires on a simulated camera or model change. A procedure nobody has exercised is a document, not a control. The engagement runs under Production AI Reliability and starts with framing the problem against your current baseline.

The scope has limits. This work is not safety certification, regulatory sign-off or a replacement for in-house QA, and your quality unit still approves every change. It does not suit proof-of-concept systems that have no signed-off baseline yet. Other routes are realistic too. An in-house validation team can work from ISPE’s GAMP 5, a risk-based framework for scoping revalidation after a change. The equipment vendor’s own requalification service fits when the change stays inside their hardware. A validation consultancy fits when the gap is regulatory documentation rather than engineering.

A maintenance plan an instructor can run without touching the model

For a classroom or training rig, the workable plan is a frozen model and taxonomy, a scheduled audit using a reference image, and a checklist trigger after any physical change.

TechnoLynx sets up these change triggers, and the tests showing they fire after a simulated hardware or model change, under Production AI Reliability, starting by framing the problem against your existing model and threshold baseline.

Locking the model so it never needs routine retraining is reasonable as long as the lighting and the set of objects stay within the bounds it was validated on. Configuration control, as draft Annex 22 frames it, is what makes the lock real, because it includes measures to detect unauthorised changes. The instructor’s diagnostic would borrow iFactory’s production check: a reference target compared with its baseline brightness and sharpness. It should show a plain pass or fail, never a metric the instructor has to interpret. The routine is short:

• after replacing a part, run the reference check
• after re-pointing the camera, run it again
• after reloading the frozen model, confirm the version and run the check
• stop and escalate on any failure

Retraining comes in when performance drops significantly or drift is detected. For production lines, iFactory recommends logging lux quarterly against the deployment baseline. A rig with windows nearby may need a shorter interval.

Change thresholds an underwriter can apply the same way every time

Where an insurer or fleet owner has to decide whether a hardware or software change needs new evidence before renewal, we would recommend a tiered policy. In that policy, a sensor supplier swap or a core model update would require revalidation evidence. A minor firmware patch classified as low-risk would get a streamlined attestation. GHTF treats a supplier change as possibly requiring parts of OQ and PQ to be redone, and draft Annex 22 requires any change to a model or its system to be evaluated for retest.

TechnoLynx can write the engineering side of such a policy. That covers cataloguing the change types for the fleet’s inspection systems, setting the mandatory-versus-streamlined thresholds, drafting evidence templates and piloting them against recent change cases. The categories should be reviewed periodically as new hardware appears. Without a fleet-wide monitoring and audit trail, it is hard to verify that underwriters apply the checklist consistently.

The same discipline, a different statistic in every domain

The checklist, golden set, monitoring cadence and rollback rule carry over between domains. The sufficiency criteria do not, and each use case needs its own. For a vision station, the protected statistic is the escape rate and false-reject rate at the signed-off threshold. For an LLM swap, it is the pass rate on a fixed, versioned evaluation set against a bar agreed in advance. For a forecasting model, it is the error on a fixed backtest window as seen by the downstream decision. For GxP narrative generation, draft Annex 22 sets the floor: acceptance criteria at least as high as the performance of the process the model replaces.

In every domain, the hard question is the same. It is how large a move in the protected statistic counts as a failure, and that number has to be written down before the change, not argued over after it.

FAQ

What does AOI stand for in inspection?

AOI stands for automated optical inspection. IPC-9716 sets requirements for AOI systems on printed board assemblies, covering lighting, calibration, threshold limits, measurement system analysis and verification protocols.

What is continued process verification?

Continued process verification is Stage 3 of FDA’s process validation lifecycle. It means gaining ongoing assurance during routine production that the process stays in a state of control.

What does gage R&R mean?

Gauge R&R (repeatability and reproducibility) studies characterise the errors that affect measurements made by gauges and instruments in production.