A quoted CV inspection accuracy figure is a measurement of a vendor’s image set, not a prediction about your line. Until that number has been re-measured on your own defect classes, under your lighting and at your throughput, it should be treated as unverified — and the business case built on top of it should be treated as provisional.
This is the failure mode we see most often in industrial vision procurement. It is rarely dishonesty. A vendor demo genuinely achieved 98.7% on the data it was scored against; the images were captured under fixed illumination, with parts presented squarely, and the defect mix reflected what was easy to collect. The line does none of those things. The gap between those two numbers is not noise — it is structural, it is predictable in direction, and it is measurable before any hardware is committed.
Where does the vendor benchmark number actually diverge from site-measured accuracy?
Four mechanisms account for most of the delta, and they compound rather than average out.
Long-tail defect classes. Curated benchmark sets over-represent the defects that are easy to photograph and under-represent the ones that are rare, ambiguous, or destructive to produce on purpose. A model scored at 98% blended accuracy can sit near-perfect on the three dominant classes and near-random on the two classes your QA records show cause the most escapes.
Presentation variance. Benchmark images are captured with the part where the optics expect it. On a conveyor with a loose fixture, pose and standoff distance wander, defocus creeps in, and a defect signature that occupied 40 pixels in the demo occupies 12 at the far edge of the field.
Illumination instability. Plant lighting drifts across shifts, ambient light leaks change with the season and the loading-dock door, and specular surfaces flip between showing a defect and showing a highlight. We treat this as a first-order variable, not an environmental footnote — the physical conditions that decide it are covered in more depth in the CV inspection site survey.
Throughput pressure. Accuracy quoted without a stated inference budget is quoted at leisure. On the line, the confidence threshold and non-max-suppression settings that hold the false-positive rate to something the rework station can absorb are not the settings that maximise the headline score. A model that must decide inside a 25 ms window with the frame-rate the belt dictates is a different operating point than the one that was benchmarked.
Two claims worth stating plainly, because they are what the rest of this argument rests on:
A CV inspection accuracy figure is only actionable when it is reported per defect class, with the false-positive rate at the throughput the line must hold. A single blended number cannot be acted on, because it does not tell you which defects are in scope and which are not — and scope is decided per class, not per part.
The benchmark-to-site accuracy gap is measurable before pilot spend, not after. It is a data and measurement exercise, not a discovery that requires committing cameras, edge hardware, and integration engineering first.
What a blended number hides that per-class reporting shows
The same system, reported two ways. The left column is what appears in a demo; the right is what the buyer needs.
| Vendor blended report | Per-class site report | |
|---|---|---|
| Detection rate | One figure, all defects pooled | One figure per defect class, as the plant’s QA taxonomy defines them |
| False positives | Often absent, or quoted at the accuracy-optimal threshold | Stated at the throughput and threshold the line will actually run |
| Data source | Curated image set, fixed lighting, controlled presentation | Buyer’s own defect instances, captured across shifts and fixtures |
| Confidence | Point estimate | Band, with the number of confirmed instances behind each class |
| Decision it supports | “The system works” | “These classes are in scope; these are not; here is the acceptance threshold” |
| Failure it hides | Long-tail classes averaged away by dominant classes | — |
The right-hand column is not a harder report to produce. It is the same measurement, disaggregated, with the false-positive rate attached and the sample size disclosed. Most of the resistance to producing it is not technical.
Re-measuring a quoted figure before the pilot
The sequence below is what we run when an industrial engineering lead brings us a vendor accuracy claim they cannot verify. It is scoped as a feasibility audit rather than a pilot precisely because it precedes hardware commitment; our wider approach to computer vision engineering sits behind it, and the audit itself is described among our engineering services.
- Rebuild the defect catalogue from the plant’s own records. Not the vendor’s class list. Escapes, rework tickets, and scrap codes define which classes matter and their relative frequency.
- Count confirmed instances per class. A detection-rate claim on four confirmed instances is not a claim. Establish, per class, how many labelled instances exist and how many more must be collected before a rate means anything.
- Capture under real variance, not best case. Span shifts, both ends of the conveyor’s speed range, the fixture in its worn state as well as its ideal one, and whatever ambient light the bay actually receives.
- Re-score the vendor model on that set. Same weights, same pipeline, buyer’s data. Report per class, with the confidence threshold fixed at the operating point the line requires — not tuned per class to flatter the result.
- Attach the false-positive rate at throughput. Convert it into rework-station load, in parts per shift. That is the number the plant manager can veto.
- Set the acceptance threshold before the pilot. Per class, in writing, agreed with the vendor: detection rate floor, false-positive ceiling, and the sample size that will be used to judge it.
Step 6 is the one most often skipped, and it is the one that decides whether the pilot produces a decision or an argument. When acceptance criteria are written after the data arrives, the criteria move.
The named-technology detail matters here too. Whether the model is a YOLO-family detector exported through ONNX and run under TensorRT, or a segmentation-based approach, the export and quantisation path can shift the operating point between the benchmarked model and the deployed one. Re-measurement has to happen on the deployed artefact, not the training-time checkpoint.
What this costs you when it is skipped
The visible cost is a failed pilot: hardware bought, integration hours spent, and a system that catches the dominant defect class while the escapes that motivated the project keep escaping. The less visible cost is the credibility one. A vision project that over-promised once makes the next proposal harder to fund, even when the next proposal is sound — and in our experience that second-order damage outlasts the write-off.
The under-recognised outcome is the good one: sometimes the re-measurement says no. Two of the five classes are physically out of reach at achievable pixel resolution, the business case does not survive without them, and the correct decision is not to pilot. Reaching that conclusion for the cost of an audit rather than the cost of a deployment is a result, not a disappointment. Which defect classes are reachable in the first place is its own question, and the feasibility boundaries by defect class treat it directly.
Frequently Asked Questions
What does “CV inspection pilots over-promise on accuracy” mean in practice on a production line?
Stripped down, CV Inspection Pilots Promise is the following. CV Inspection Pilots Promise has one honest answer. It means the accuracy figure the project was funded against was measured somewhere other than the line. In practice the pilot detects the dominant, high-contrast defect classes at close to the quoted rate and performs far worse on the rare classes, while producing more false positives than the rework station was staffed for. The system is not broken; the promise was scoped to conditions the line does not reproduce.
Why does vendor benchmark accuracy differ from site-measured accuracy, and where does the gap come from?
The gap comes from four compounding sources: long-tail defect classes that curated sets under-represent, part-presentation variance on a real conveyor and fixture, illumination that drifts across shifts and reacts to surface finish, and the confidence threshold being forced to a different operating point by throughput and false-positive limits. Each is a property of the buyer’s environment, which is why the delta is site-specific rather than a fixed discount you can apply to any quoted number.
How do you re-measure a quoted accuracy figure on your own defect set before committing to a pilot?
Rebuild the defect catalogue from your own QA and scrap records, count confirmed labelled instances per class, capture images spanning real shift, speed, and fixture variance, then re-score the vendor’s deployed model — post-export, post-quantisation — on that data at the threshold the line requires. Report per class with the false-positive rate attached. This is the scope of a feasibility audit and does not require the pilot hardware to be bought.
Why is a single blended accuracy number misleading, and what does per-defect-class reporting show instead?
A blended figure lets strong performance on frequent, easy classes mask near-failure on rare ones, and the rare ones are often exactly the escapes that justified the project. Per-class reporting shows which classes are in scope, which are not, and how many confirmed instances sit behind each rate — turning one unusable number into a scoping decision you can defend line-by-line.
How should false-positive rate at production throughput be reported alongside detection rate?
As a pair, at a single fixed operating point, expressed in units the plant uses: detection rate per defect class, and false positives converted into parts flagged per shift and the rework hours that implies. A detection rate quoted without its false-positive rate at production throughput is not a specification — the threshold can always be moved to make one look good at the other’s expense.
What data volume and defect representation do you need before an accuracy measurement is trustworthy?
There is no single universal count, but the requirement is per class rather than in total: enough confirmed instances of each class that the rate carries a usable confidence band, and coverage spanning the shift, lighting, speed, and fixturing variance the line genuinely exhibits. If a class has only a handful of confirmed instances, the honest output is “not yet measurable” rather than a rate.
What acceptance thresholds should be agreed with a vendor before a CV inspection pilot starts?
Per defect class: a detection-rate floor, a false-positive ceiling stated at production throughput, the sample size the judgement will use, and the conditions under which data will be collected. Agree them in writing before hardware is committed, because thresholds negotiated after results arrive tend to relocate toward the results.
If your business case still rests on a number you have not re-measured, the open question is not whether the system works — it is which of your defect classes the number was never covering.
Vendor demos use cleaner data than you have
Controlled lighting, consistent part positioning, and pre-filtered defect categories inflate accuracy numbers by 15-30% compared to production floor conditions. If CV Inspection Pilots Promise is on your roadmap, the next step is to map it onto your own constraints rather than copy a reference architecture.
