A feasibility audit has cleared computer-vision inspection on your line. The next question is not which vendor, it is how big the pilot has to be — and that is the question industrial teams most often answer with a calendar rather than with the defect distribution. Four weeks on one line, collect whatever comes past the camera, report a blended accuracy figure. The number that comes out of that pilot is not wrong so much as unusable: it cannot be broken down per defect class, it carries no stated confidence, and it was measured under whichever subset of the line’s lighting and fixturing conditions happened to occur inside the window.
Sizing a pilot properly means fixing four things before the first image is captured: which defect classes are in scope, how many confirmed instances of each you need, which operating conditions the window must span, and what result counts as a pass. All four come out of the feasibility audit’s findings, not out of a project schedule.
What does sizing a CV inspection pilot before scaling actually involve?
The pilot exists to produce one thing: evidence strong enough to fund or refuse scale-up. Everything in the sizing decision follows from that.
Two claims are worth stating plainly, because they are where the naive and expert approaches diverge.
A detection rate is only meaningful per defect class, and only against a known number of confirmed instances of that class. A pilot that saw four scratches and six hundred solder voids has measured solder voids and guessed at scratches — yet a blended figure hides exactly that asymmetry. Report per class or do not report.
The second claim: a pilot window that does not span the line’s real lighting, shift, and fixturing variance measures the pilot’s conditions, not the line’s. We see this pattern regularly — a system that held 97% through a two-week day-shift pilot drops sharply on the night shift when the overhead plant lighting contribution changes, or after a fixture is re-seated during a changeover. That is not model degradation. It is a pilot that never sampled the condition.
Sizing is therefore a sampling problem with a cost constraint attached. Too small and the numbers cannot carry a capital request. Too large and you are re-buying evidence the feasibility audit already gave you for free.
Deriving the sample requirement per defect class
The audit hands you a defect-class catalogue with a feasibility verdict per class. That catalogue is the scope list — the pilot inspects the classes rated feasible, and only those. Borderline classes belong in a documented “observe but do not claim” bucket: instrument them, log what the model does, make no rate claim. Attempting a defensible rate on a class the audit rated borderline is the fastest way to end a pilot with a number nobody will sign against.
For each in-scope class, work backwards from the claim you need to make. If the business case rests on catching 95% of a given defect and you want a confidence interval narrow enough to be worth quoting, a handful of instances will not do it — you need tens of confirmed positives, and the tighter the interval the more you need. Then divide by the class’s natural occurrence rate on the line to get a window length. This is where sizing usually breaks: the rare classes, which are frequently the expensive ones, imply windows measured in months at natural rates.
The practical response is seeding. Confirmed defective parts pulled from QA quarantine, reintroduced under controlled conditions and logged, close the gap on rare classes without waiting for production to supply them. Seeded instances measure detection; they do not measure false-positive rate, which has to come from the natural stream at production throughput. Keep the two populations separate in the data model from day one — merging them is a mistake that is very hard to unwind at analysis time.
The sample-sizing and benchmark-design discipline here is the same discipline we apply in porting and performance assessment work, where the question is also how much measurement is enough to support a decision.
What the pilot window has to cover
Sample counts set a floor on duration. Variance coverage sets a different floor, and the binding constraint is whichever is longer.
The site survey profile tells you what conditions exist: illumination stability across shifts, seasonal or ambient light ingress, part-presentation repeatability in the fixture, conveyor speed range, vibration, and the changeover events that reset fixturing. Turn that profile into an explicit coverage checklist and track percentage covered as a live pilot metric, not a retrospective observation.
| Sizing dimension | Derived from | Minimum the pilot must produce |
|---|---|---|
| In-scope defect classes | Audit feasibility verdict per class | Per-class rate for every class rated feasible; no rate claimed for borderline classes |
| Confirmed instances per class | Target detection rate + acceptable confidence interval | Tens of confirmed positives per class; seeded parts permitted, logged separately |
| False-positive measurement | Natural production stream only | FP rate at the throughput the line must hold, not at a reduced pilot rate |
| Window length | max(sample requirement, variance coverage requirement) | Both floors satisfied, not just the shorter one |
| Variance coverage | Site-survey lighting / shift / fixturing profile | Stated % of catalogued conditions sampled, with the uncovered ones named |
| Cost per inspected unit | Pilot operating point + incumbent manual baseline | Per-unit inspection cost against the manual workflow it partially displaces |
The criteria that must be written down first
A pilot without pre-agreed pass conditions becomes a negotiation about its own results. Fix these in writing, signed by the plant and the supplier, before capture starts:
- Per-class detection rate thresholds, each with the confidence interval width that makes it acceptable.
- A false-positive ceiling expressed in operational terms — false rejects per shift, or rework hours per thousand units — measured at production throughput. A percentage alone is not actionable to a line manager.
- The throughput the system must sustain while holding those rates, since the confidence threshold that hits the detection target and the one that hits the FP ceiling are rarely the same setting.
- Which conditions must be covered for the result to count, and what happens if a condition never occurs during the window.
- Who confirms ground truth, and how disagreements between the model and the inspector are adjudicated and recorded.
These artefacts have a second life. If scale-up is approved, the success criteria and the false-positive ceiling become the reliability envelope the hardening work is measured against — which is a good reason to write them as if someone will hold you to them, because someone will.
The cost model, sized alongside the pilot
The pilot must return a cost figure, not just an accuracy figure. Two comparable quantities: cost per inspected unit at the pilot’s operating point, and the cost of the incumbent manual workflow for the same defect classes.
The second one is where the model usually goes wrong. Manual inspection is rarely fully displaced — sampling, rework triage, and audit tasks remain with people, so the baseline is the portion of the manual workload the in-scope classes actually represent. False-positive rework hours belong on the CV side of the ledger at the FP rate the pilot measured, not at the rate the vendor quoted. Our detailed treatment of that comparison lives in the total-cost analysis of CV versus manual inspection; the pilot’s job is to supply that model with real inputs rather than assumptions.
Include the pilot’s own cost as a sunk line item and state what a second pilot run would cost. That number is the strongest argument for sizing correctly the first time: re-running a pilot to recover evidence the first attempt failed to gather typically costs more than the extra weeks and seeded parts that would have prevented it.
Where sizing meets scope
Pilot sizing is not a project-management task bolted onto a vision engagement. It is the point where the feasibility audit’s findings become a measurement plan, and it is the part of the work we scope most carefully in our computer vision engagements — usually as a defined step between audit and build, described in more detail across our services.
The honest remaining uncertainty is the rare-defect tail. Seeding closes part of the gap, but a class that appears twice a year on a line will not have a well-bounded detection rate at the end of any pilot a plant is willing to fund. Naming that class as unmeasured, and deciding in advance whether it stays with human inspectors, is a better outcome than quoting a rate built on three samples.
Frequently Asked Questions
What does sizing a CV inspection pilot before scaling mean in practice? Pilot scope determines whether your computer vision inspection trial generates statistically valid ROI data or wastes engineering resources on inconclusive results. It means deciding, before capture begins, which defect classes the pilot will claim results for, how many confirmed instances of each it needs, which line conditions the window must span, and what result counts as a pass. Those decisions come from the feasibility audit’s defect-class catalogue and lighting/fixturing profile, not from a project calendar. The output is a measurement plan, not a schedule.
How many confirmed defect instances per class do you need before a detection rate is defensible? Enough that the confidence interval around the rate is narrow enough to quote — in practice tens of confirmed positives per class, with the exact figure driven by the target rate and the interval width you are willing to accept. A handful of instances yields a number with an interval so wide it cannot support a capital decision. Seeded parts from QA quarantine are a legitimate way to reach the count on rare classes, provided they are logged separately from the natural stream.
How long should the pilot window run to cover the line’s real lighting, shift, and fixturing variance? Long enough to satisfy two independent floors: the sample requirement per class, and coverage of the conditions catalogued in the site survey — shifts, ambient light changes, conveyor speed range, and at least one changeover that re-seats fixturing. The binding constraint is whichever floor is longer. Track coverage as a live metric and name explicitly any condition the window never sampled.
What success criteria and false-positive ceiling should be fixed in writing before the pilot starts? Per-class detection thresholds with their acceptable confidence-interval widths, a false-positive ceiling expressed operationally as false rejects per shift or rework hours per thousand units, the throughput that must be sustained while both hold, the required condition coverage, and the ground-truth adjudication process. Fixing these in advance prevents the pilot from becoming a negotiation about its own results, and the same artefacts become the reliability envelope if scale-up proceeds.
How do you build the pilot cost model against the incumbent manual inspection workflow? Compare cost per inspected unit at the pilot’s operating point against the portion of the manual workflow the in-scope defect classes actually represent — not the full manual inspection cost, since sampling, rework triage, and audit tasks stay with people. Load false-positive rework hours onto the CV side at the rate the pilot measured, and include fixturing, lighting, and integration effort. State the cost of a second pilot run as well; it is the clearest measure of what mis-sizing costs.
Which defect classes should be excluded from the pilot even when the feasibility audit rates them borderline? Borderline classes should be instrumented and logged but excluded from rate claims. If imaging physics, achievable pixel resolution, or fixturing repeatability leave a class uncertain, a pilot-scale sample will not resolve it — it will produce a number with an interval too wide to act on while consuming window time the feasible classes need. Observe them, report what was seen, claim nothing.
What evidence does a pilot have to produce for a scale-up decision to be fundable? Per-defect-class detection rates with stated confidence, a false-positive rate measured on the natural stream at production throughput, the percentage of catalogued line variance the window covered with uncovered conditions named, and a cost-per-inspected-unit figure set against the incumbent manual baseline. Missing any one of these usually forces a second pilot run to recover the gap.
Three numbers determine pilot scope
Start with your current monthly defect volume, target false-positive rate, and the minimum accuracy gain that justifies deployment cost. Revisit it when your workload shifts.
