Statistical Confidence Bounds on Defect Escape Rate for Edge Vision

You can retire manual inspection on an edge vision system once statistical confidence bounds on the defect escape rate, not the observed rate, fall below a limit you agreed in writing before testing began. That written agreement needs three numbers: a one-sided upper confidence bound the escape rate must stay under, a lower confidence bound the field recall must clear, and an accept/reject threshold margin shown to hold under sustained thermal load. The reason is arithmetic. Zero escapes in 500 inspected parts looks perfect, but by the rule of three the 95% upper limit on the true rate is still about 3/500, or 0.6%.

A recall figure from a validation set describes accuracy, which is model quality. The bound, the sustained-load test and the signed protocol are reliability artefacts, and production AI reliability is its own engineering discipline, separate from accuracy. The hard part is agreeing what “good enough” means, measured on which parts, at what temperature, and who has the authority to decide.

Can we fix the cycle-time and accuracy protocol with the client before the deadline?

Yes. Write a joint qualification protocol, structured like a factory acceptance test, and have the client sign it before the first validation part runs. The protocol fixes the line’s cycle-time target, the escape-rate and recall bounds, the test sample, the environmental conditions and the pass/fail rule.

LynxBenchAI publishes per-device results on the public PyTorch stack, which give a broad indication of the candidate edge device before the cycle-time figure is fixed; the first step is checking the candidate device’s results there, bearing in mind that they do not measure your inspection model.

TechnoLynx frames the cycle-time and accuracy targets with the client, chooses the sample composition and sustained-run duration, builds the qualification harness on the target edge hardware, runs the validation and reports throughput and accuracy evidence against the agreed criteria. The first step is framing the cycle-time and accuracy targets with the client.

Factory acceptance practice explains why the order matters. Buyer and supplier agree what “pass” means before the test date (specifications, tolerances, cycle times) and write it into the FAT protocol. Without that document, pass and fail turn into a negotiation on the test floor instead of a documented outcome. Medical-device verification goes further. Acceptance criteria must be defined before the first sample is evaluated, and auditors compare protocol approval dates with test execution dates to catch criteria written after the results were known. Those rules sit within medical-device design verification under 21 CFR 820.30(f). The discipline still carries over, because a sign-off threshold chosen after seeing the data proves nothing.

A protocol that holds up at the deadline covers these items:

Cycle time at true line speed and part mix, measured over a sustained run rather than a short burst.
Escape-rate upper bound, with its confidence level, statistical method and sample-size justification; ISO 13485 verification planning asks for the same justification.
Recall lower bound and the interval method used to compute it.
Test sample drawn from production-representative parts. If non-representative articles are used, the reason has to be documented and the test may need repeating. The sample also needs a representative range of defects, with enough in each category for statistical analysis.
Environmental conditions, specified and recorded throughout the test. On edge hardware this includes enclosure ambient and device temperature.
Pass/fail logic for each criterion, as a measurable threshold rather than “runs correctly”, and saying whether all samples must pass or x of y.
Deviation and retest rules: a failed test needs root-cause analysis before any retest, never retesting until it passes.

The sign-off evidence should include a comparison with what the line does today, as well as the bounds. For inspection-method changes in a regulated quality system, the central piece is an equivalence study on paired samples, ideally including seeded known defects, showing the new method is at least as capable as the old one. That claim needs a documented baseline of the manual method: current SOPs, historical defect rates by category, and past escapes. Independent confirmation matters too. Relying only on the supplier’s own test data removes most of the value of an acceptance test.

Factory and site testing ask different questions. FAT asks whether equipment works as built, and SAT asks whether it works here, with real power, real upstream equipment and real production materials. For an edge vision system, we would put the accuracy criteria in the on-line run. Lighting, part presentation and enclosure heat are part of what is being qualified.

The written protocol does not freeze requirements forever. If the client changes the part mix or line speed, renegotiation is legitimate, but it goes through a documented change rather than an argument in the last week. Missing metrics cause the same trouble later: any performance metric left out of the protocol surfaces as a gap after installation. FAT is frequently written into the purchase contract as a condition for shipment and payment release, which makes it contractually binding even though no regulation mandates it. After sign-off, the same evidence feeds the evidence package and runtime safeguards for an unattended edge vision deployment, which covers what must keep running once inspectors have gone.

Will borderline accept/reject calls flip when the edge GPU throttles?

You find out by running the inspection model until the edge device throttles, recording the scores of parts near the threshold, and setting the accept/reject margin at or above the worst drift observed. If the drift stays within tolerance, the threshold stays where it is.

TechnoLynx defines the borderline score range and the thermal stress profile, runs sustained-load tests on the target edge GPU, compares borderline-call stability across repeated runs, and locks the margin and its monitoring into production. LynxBenchAI helps with choosing the device in the first place. Its published results show how each device runs the public PyTorch stack your models are built on, so they are broadly indicative for a custom PyTorch classifier, but it does not run or measure your inspection model.

NVIDIA documents the mechanism. NVIDIA’s TensorRT documentation says thermal throttling starts when GPU temperature reaches a predefined threshold of around 85°C on most GPUs, and that nvidia-smi dmon shows temperature climbing during inference until the clock drops. The same documentation warns that GPUs with low power limits, such as the NVIDIA T4, power-throttle by design. Poor cooling can lower their stabilised clock before any thermal throttle happens. It also notes that clock changes make measurements taken over very short windows (around 20 ms) unstable, and that locking the clock at a lower frequency stabilises them.

In a study of a passively cooled Raspberry Pi 5 running YOLOv8n (an edge CPU platform, not a GPU), the uncontrolled fastest state saturated at 84.8°C and throttled 1823 times over 30-minute sustained runs. The same authors found that the passive operating envelope closed at ambient temperatures of 27°C and above, where even the lowest frequency state rose past the throttle threshold. They used 82°C as a conservative reference below the 85°C throttle limit when defining thermal margin. A lab bench at 21°C and a cabinet beside a curing oven are different test conditions.

No source cited here measures how inspection scores drift under throttling, so the margin has to come from your own runs on the target device.

Sizing the threshold margin from a sustained-load run

Select a borderline set: validation parts, both good and defective, whose scores sit in a band around the accept/reject threshold.
Score the set cold to establish a baseline, then run the production workload in the real enclosure at the worst expected ambient until temperature plateaus and clocks drop, logging device telemetry throughout.
Re-score the borderline set at intervals during the hot run, recording per-part score deltas against the cold baseline along with cycle time and dropped frames.
Repeat the run several times; set the margin to at least the worst observed drift, times a safety factor agreed in the protocol.
Characterise each GPU model and each enclosure design separately.

A wider margin has a cost. Every part pushed into the uncertain band becomes a false reject or a human review, so the margin and the false-reject budget belong in the same protocol line.

How do statistical confidence bounds on the defect escape rate work, and how many parts do they take?

A statistical confidence bound on the defect escape rate is the highest true rate still consistent with what you observed, at a stated confidence level. It is safer than a point estimate because a small sample with zero or few escapes always produces a point estimate that flatters the system. The exact one-sided bound has a minimum probability of covering the unknown true rate equal to the target confidence level.

LynxBenchAI publishes per-device results on the public PyTorch stack that help choose the edge hardware, and the first step is checking the candidate device there; it does not measure your inspection model, so the bound still comes from your own parts.

The maximum tolerable escape rate is a separate number and is set first. It reflects the downstream harm of a missed defect, not the sample size you happen to have. FDA process-validation guidance says the confidence level can be chosen from a risk analysis of the attribute being examined. Medical-device verification practice likewise ties higher-risk requirements to higher reliability and confidence, and so to larger samples.

Computing the upper bound: Clopper-Pearson and the rule of three

Use the exact binomial (Clopper-Pearson) method for rare events. Fritz Scholz’s notes on confidence bounds prefer its conservative approach when bounding accidents or undesirable defects. With k escapes in n parts, the 95% upper bound is qbeta(0.95, k + 1, n - k) in R. In Scholz’s worked example, 25 events in 912 trials give a point estimate of 0.0274 and an upper bound of 0.0381. Excel provides the same Beta quantile through BETAINV.

Avoid the textbook normal-approximation bound. Its minimum coverage is zero: for true rates close to zero, a sample with no escapes makes the bound zero as well, so it cannot cover the true rate. The NIST/SEMATECH handbook notes that when n is small (say below 30), normal-approximation bounds may not be accurate enough, and an exact method is easy to compute.

With zero escapes the arithmetic simplifies. The rule of three says that if none of n items shows the event, you can be 95% confident the true rate is at most 3/n. Hanley and Lippman-Hand (JAMA, 1983) report that above n = 30 the rule agrees with the exact calculation to the nearest percentage point, and that below 30 it slightly overestimates the risk.

Use a one-sided bound, not a two-sided interval. When small rates are the concern, a two-sided interval spends half the miss probability on a lower bound nobody needs. At 95%, matching the one-sided zero-event bound with a two-sided interval needs about 23% more samples (m = 1.2314n), all of them also escape-free.

Worked example: parts needed with zero escapes at 95% confidence

Assumptions: parts are independent, the true escape rate is constant over the test, and the test sample represents the production defect mix. The exact zero-failure size is n = ln(1 − C) / ln(1 − p), the binomial reliability formula with R = 1 − p.

Target escape rate p	Rule of three (3/p)	Exact one-sided, n = ln(0.05)/ln(1 − p)	Two-sided equivalent (×1.2314)
5% (R95/C95)	60	59	73
1%	300	299 (2.9957 / 0.01005 = 298.1, rounded up)	369
0.1%	3,000	2,995 (2.9957 / 0.0010005 = 2,994.2, rounded up)	3,688

The 59-sample row matches the R95/C95 attribute plan used in device verification. The 0.1% row matches Scholz’s example of running 3,000 trials to establish 0.001 as a 95% upper bound “and hope for the best, namely 0 events”. If an escape does turn up, recompute the bound with qbeta, which handles nonzero counts.

Be careful what n counts. If “escape rate” means the share of defective parts the classifier misses, n is the number of defective parts inspected, not the number of parts. On a line with a low defect rate, 3,000 defective parts can mean a very large production run, or a seeded-defect library built for the purpose. If “escape rate” means escapes per part shipped, the bound is easier to reach, but it also depends on the defect rate staying where it was during the test. Choose one definition in the protocol and keep it.

Keeping a human double-check while the bound is still wide

Compute the upper bound from current field samples. If it is at or below the tolerable maximum, a reduced safety margin can be considered; if it is above, keep the human double-check and keep collecting. With zero escapes the bound shrinks roughly as 3/n, so doubling the escape-free sample halves it. Wide bounds can hold the gate shut for a long time. That conservatism is intentional.

The double-check also counts the escapes. A field escape is only known if something downstream catches it, so removing the human check too early removes the instrument that measures the bound.

Human inspection is a weak reference point. Swain and Guttmann (1983) estimated the minimum human error rate at about 10⁻³ for simple accept/reject tasks, and higher for complex ones. Error rates of 20% to 30% are frequently quoted across inspection tasks, and even 100% inspection does not find every defect. Detection can also deteriorate by up to 40% within 30 minutes of inspecting. The safety net works best when it is designed for that. Two inspectors outperform one in every arrangement studied except splitting the batch in parallel. Humans tend to do better at deciding than at searching, so the machine should search and the human should decide on what it flags. In one study, a machine aid with low misses and high false alarms gave the best system performance and the highest trust scores.

Track the escape-rate bound and the false-reject rate as two separate controls. A drawdown that cuts false rejects can quietly push escapes up, and a single “accuracy” figure hides the trade.

What lower confidence bound on field recall should gate removing inspectors?

No universal number exists. The gate is a minimum lower confidence bound on field recall, fixed in advance, that is at least as good as the manual baseline measured on the same parts, and it is checked at each stage of a phased drawdown rather than once. A point estimate is the wrong basis. Sampling error can make a point estimate of recall overstate or understate the truth, leading to work being accepted as thorough when it is seriously incomplete. In William Webber’s study of recall confidence intervals, the recall estimator’s bias is greater for low prevalence, small samples and low recall.

The method matters. Webber concludes that naive binomial and normal (Wald-type) intervals are biased or unstable and should be avoided. For a simple binomial recall (defects caught out of defects present), the Jeffreys interval uses a Beta(0.5 + r, 0.5 + n − r) posterior. A beta posterior with a Jeffreys prior gives an accurate one-tailed interval on a binomial proportion. Where recall is estimated by sampling the accepted and rejected segments separately, Webber recommends the half-prior beta-binomial interval as the most balanced.

Small numbers say less than they seem to. With every defect caught, the 95% lower bound on recall is (0.05)^(1/n), approximately 1 − 3/n when n is large. Five defects out of five gives an exact lower bound of about 0.55 (0.05^(1/5) = 0.549). Scholz’s corrosion example, also 5 successes in 5 trials, concludes that jumping to strong conclusions “has no basis”. A lower bound of 0.99 needs roughly 300 defective parts, all caught (1 − 3/300 = 0.99). Webber argues that wide bounds should be reported anyway: where uncertainty is high, showing it is all the more important.

Phased drawdown: go/no-go at each stage

Stage	Inspection arrangement	Gate to advance	If the gate fails
Shadow	Classifier runs; inspectors check every part; disagreements logged	Recall lower bound and escape upper bound meet the protocol on paired samples, including seeded defects	Retrain or adjust threshold; stay in shadow
Double-check	Inspectors review all classifier accepts in the borderline band plus a random audit	Bounds still met on field data from this stage; thermal margin confirmed on the production enclosure	Hold the stage; keep collecting
Reduced	Human review of flagged and borderline parts only; scheduled audit sample	Field escape bound below the agreed maximum; false-reject rate within budget	Return to double-check
Removal	Audit sampling and drift monitoring only	Final-stage evidence package signed by the named authority	Revert to the previous stage

The whole table is our recommended structure, not a standard. Each stage only moves the bound if it produces enough representative data. A short stage with a handful of defects barely changes the estimate, so stage length should follow the sample-size arithmetic rather than the calendar.

Who signs off the go-live decision, and on what evidence?

The go-live decision belongs to the client’s quality or safety owner, named in the protocol before testing, and not to the engineering team that built the classifier. In a regulated quality system, the final accept/reject decision of a vision system is almost certainly classed as high process risk, which calls for scripted testing, boundary testing against a curated defect library and full records. FDA guidance recommends that a statistician or someone trained in statistical process control develop the data collection plan and the statistical methods.

The evidence package the signatory reviews contains the signed protocol with its approval date, the escape-rate upper bound and recall lower bound with methods and sample counts, the equivalence study against the manual baseline, the sustained-load thermal results, each drawdown stage’s results, and any deviations with their impact assessments. Deviations are acceptable to regulators when they are documented with an impact assessment that concludes the data remain valid.

Who can help us qualify an edge defect classifier before the inspectors come off the line?

TechnoLynx builds and runs the qualification for an edge-deployed defect classifier in manufacturing and automotive production. TechnoLynx agrees the cycle-time and accuracy protocol with your quality team, builds the test harness on your target edge hardware, runs the sustained-load and escape-rate validation, and supports sign-off and post-go-live monitoring. The protocol, statistics and thermal characterisation are where an outside check adds independence. When choosing a partner, ask to see a qualification protocol written before testing and a sampling plan with its arithmetic shown. For the hardware decision, LynxBenchAI is the measuring tool: published results per device on the public PyTorch stack.

TechnoLynx supports sign-off but does not sign the release: authority stays with your quality owner. TechnoLynx’s scope is the engineering, not regulatory sign-off, so a line under FDA or notified-body oversight still needs its own qualified reviewer.

FAQ

What does thermal throttling mean on an edge GPU?

The driver lowers the GPU clock once the chip reaches a predefined temperature, around 85°C on most NVIDIA GPUs, to prevent overheating. Low-power cards such as the T4 can also power-throttle before they get that hot.

Does a higher confidence level mean a smaller margin of error?

No. For a fixed sample, a higher confidence level gives a wider interval, and holding the bound fixed takes more samples. For a zero-event bound at 95%, moving from one-sided to two-sided (halving the miss probability on the upper end) needs about 23% more samples.