Model performance evaluation for an inspection model means measuring each station against its current inspection baseline, with results separated by defect class and inspection condition. A single blended accuracy number hides exactly the places where the model breaks. Aristek Systems gives the cleanest example: on a line where 0.5% of parts are defective, a model that accepts every part scores 99.5% accuracy and catches no defects at all. What replaces the blended score is a set of error rates (escape rate and false reject rate) broken out by station and defect class and measured against the inspectors the model is meant to replace or support.
Model performance metrics that matter on a defect-detection line
Two numbers usually matter more than accuracy on a production line. Escape rate is the number of defective parts accepted divided by the total number of actual defects. False reject rate is the number of good parts rejected divided by the number of good parts. Aristek Systems recommends reporting them alongside defect prevalence, recall, precision and the number of samples tested, because those six figures together let a reader see whether a result means anything.
No single accuracy figure applies across production lines. Each station needs to be measured against its current inspection baseline, with results separated by defect class and inspection condition. That baseline has a ceiling of its own. Attribute agreement analysis compares inspectors with each other and with a reference standard, and if inspectors agree with one another only 85% of the time, labels built on their judgements limit how consistent the model can ever be shown to be.
On a defect line, the rarest critical defect is the class a blended score is least able to tell you about.
A high aggregate score can sit on top of a failed line
Barocas et al. (Microsoft) describe an evaluation set of 100 points, 90 from group A and 10 from group B. A system that gets every A right and every B wrong reports 90% accuracy, while its accuracy on group B is zero. Swap “group B” for “camera 4” or “the cracked-weld class” and the same arithmetic applies to an inspection line.
Medical imaging research reports the same effect. Oakden-Rayner et al. call it hidden stratification and report relative performance differences of over 20% on clinically important subsets that were never identified during training or testing. D’Amour et al. show a deeper version: models trained by the same pipeline with only a different random seed reach near-identical held-out accuracy, yet behave very differently under distribution shift. On some ImageNet-C corruption tasks the variation was an order of magnitude larger than the variation in standard test accuracy. In their medical imaging example, AUC varied more on a held-out camera type than on the standard test set.
Compression adds another layer. Hooker et al. found that pruned and quantised networks keep comparable headline metrics while diverging on a narrow subset of the data, and that the damage falls disproportionately on the under-represented long tail.
Production AI reliability is a separate engineering discipline from accuracy: its deliverables are the evaluation harnesses, regression suites, drift detection and validation packages that catch these failures before customers do. A dashboard showing one number is not one of those deliverables. For designing the metrics themselves, see our page on segment-specific, overfitting-aware evaluation metrics. This page covers the validation work around them.
Which subgroups should a stratified evaluation break results out by?
Break results out by defect class and by the conditions that change the image the model sees. A defect detection model does not inspect a product. It inspects an image taken under specific lighting, from a specific camera position, at a specific point in production. The strata below are where we would start on a manufacturing inspection line.
| Stratum | Why it changes results | What a shift there looks like |
|---|---|---|
| Line or station | A model trained on one camera and lighting setup can lose accuracy on another, even for the same part | One station’s escape rate climbs while the others hold |
| Camera position, lens, calibration | A camera moved by a few millimetres or a replaced light source can invalidate validation results | Drops after maintenance or a dirty lens |
| Lighting condition | In one pen-component detection study, accuracy ranged from 91.5% back-lit to 95.1% front-lit across four lighting conditions | Night-shift or ambient-light changes |
| Part variant, SKU, material lot, supplier | The golden set has to cover product variants and material lots, plus shifts and suppliers where production spans them | Rejects rise with no change in part quality |
| Shift | Ambient light differs between day and night unless the station is shrouded | Split by shift to narrow the cause |
| Defect class | Acceptance thresholds differ by class and quality risk | A class falls below its own threshold |
| Hardware target and precision | TensorRT may run each layer in a different precision, which can reduce accuracy | Per-class deltas between FP32 and INT8 builds |
The primary challenge for stratified evaluation is small sample sizes. A structured-regression study of intersectional subgroups notes that a standard per-group estimate has variance that scales with the inverse of the group’s sample size. In their 5,000-patient example, only 6 patients fell into one three-way intersection, and a quarter of the 32 subgroups had fewer than 10 observations. Sample sizes quickly get small once subgroups intersect, as line × camera × part variant × shift does.
Three options help. Merging small cells is common, but it can mix heterogeneous groups into a catch-all. Structured regression and shrinkage estimators give much more accurate point estimates for small groups, and the structured approach kept well-calibrated confidence intervals while making them up to 20% narrower than the standard method. Barocas et al. suggest considering a range of values for each additional factor. Factors that always occur together cannot be separated (if every poorly lit image also shows the same part variant, the evaluation cannot say which one caused the failure), and controlled variation is the approach that most conclusively shows whether a disparity exists. We would always report each cell’s sample count and interval next to its rate, so that a cell with ten samples is never read as a verdict.
Proving every line, site and hardware target matches the pilot
A model that passed on the pilot line has not passed on line 2 until line 2 passes its own validation. Each new station needs its own validation against the inspection process it replaces or supports.
TechnoLynx does this work through the AI Porting & Deployment Pack. TechnoLynx defines what “equivalent performance” means for each line or site, rebuilds and calibrates the inference engine for the new target, and runs per-line comparisons and shadow-mode evidence against the baseline before cutover. The first step is agreeing which baseline (human inspectors or the pilot-line model) each line has to match.
Pilot accuracy does not transfer automatically
Aristek Systems’ advice is to start AI inspection at one station and add others only after each new station passes its own validation. A medical AI review by Zhang et al. describes the general mechanism as cross-environment variation: a model applied to a new environment meets a different data distribution, made worse by differences in device settings and practice.
Once validation passes, camera and lighting positions get documented and locked, and a shroud blocks ambient light changes between shifts. Locked positions do not replace the validation pass: each new station still passes its own. A retrofitted line still has its own wear, part mix and installation tolerances, so it still gets its own check before cutover.
Running the parallel period with human inspectors
A parallel run (shadow mode) is how you build confidence before cutover. Both systems process the same production traffic, one stays live, the other’s output is recorded but not acted on, and a comparison layer logs where they agree and where they diverge. For inspection, the inspectors stay live and the model runs in shadow. The parallel-run pattern sets three rules: define the acceptable divergence rate before the run starts, expect a declining trend, and investigate a stable non-zero rate before cutting over. It also recommends setting a maximum duration up front, because an open-ended parallel run becomes a permanent operating cost.
The comparator should ignore legitimate differences and alert only on meaningful ones. On a line, that means telling model errors apart from inspector disagreement, using the agreement baseline described above. On high-volume lines, routing a representative sample (the pattern suggests 10-20%) through the shadow path gives statistical confidence without doubling the load. We would size the parallel period by the samples each stratum needs under the worked example below. That makes the number of samples per stratum, rather than a calendar figure, the planning unit.
A NIST guide on comparing instruments covers instruments that give binary responses and demonstrates equivalence testing built on an indifference zone. Sample-size planning there starts from the minimum difference you need to detect, the power you need and the Type I error you will accept. When counts are too small for normal approximations, Fisher’s exact test on a 2×2 table is the nonparametric option. When the same parts pass through both systems, a paired analysis is more efficient.
Identical labels do not require bit-for-bit identical outputs. We would set a defined equivalence margin on accept/reject decisions per defect class as the bar. The exception depends on what sits downstream of the model. One ARM study found FP32 output hashes disagreeing on 100% of images while 0% of predicted labels did, and anything that consumes continuous scores (calibrators, anomaly scorers, audit logs) inherits that drift.
Porting to Jetson, a new GPU or INT8 is a re-validation task
Treat a port as a new model that has to earn its results again. It is not a copy of the old one. TensorRT can run each layer in FP32, FP16, BF16, FP8 or INT8, and by default it picks whichever precision performs best, which sometimes costs accuracy. NVIDIA’s guidance is to compare layer outputs against golden values from a reference such as ONNX Runtime and, for INT8, to recalibrate with a more representative calibration set. NVIDIA’s Polygraphy compares two runners’ outputs by error quantile (default 0.99), L2 norm and cosine similarity.
Karimov et al. report that static INT8 TensorRT engines for YOLO models ran about 1.5-3.3x faster with a 3-7% drop in mAP50-95 on clean COCO data, and that calibrating on degraded images did not consistently improve robustness. On ARM CPUs, one 2026 study found INT8 QDQ outputs byte-identical across Cortex-A53, A72 and A76, while switching only the execution provider made FP32 outputs disagree on every test image. The authors conclude that precision, not microarchitecture, is the axis that changes behaviour on those devices. Their result covers ARM CPU runtimes, and we would not extend it to a GPU engine without testing. On that study’s small CNN, INT8 label changes clustered in the low-confidence region.
TensorRT engines are built for a specific GPU. If accuracy differs between two builds of the same model, a bad tactic choice for a layer is one cause NVIDIA names, while repeated runs of one built engine should give bit-accurate outputs. That gives a useful diagnostic on the line. If detections change while the same engine file runs, run-to-run variation is not the cause. Reusing a calibration cache on a new target still needs the per-class check. Our page on proving kernel equivalence after porting or precision changes covers numerical tolerance bands in depth.
Choosing which edge device to port to is a separate question. LynxBenchAI’s published results show how each device runs the public PyTorch stack that teams’ own models run on, so they are broadly indicative for custom PyTorch models. They do not measure your model, your accuracy or your line, so the per-line validation above still applies. The device comparisons are on the LynxBenchAI results pages.
Who can validate our inspection model across lines and hardware before we switch off manual checks?
TechnoLynx builds the validation evidence a team needs before trusting an inspection model on new lines, sites or hardware. That work covers per-station and per-class error rates against your inspectors, a time-boxed shadow run with a divergence threshold agreed in advance, and re-validation after porting to Jetson, a new GPU or INT8. The engagement runs through the AI Porting & Deployment Pack, part of TechnoLynx’s AI engineering for manufacturing and automotive, and starts by defining what equivalent performance means for each line.
A team that already has a held-out set, a harness and labelled data per line may only need a review of its statistics rather than a build. Supplier acceptance tests are another route: VDI/VDE/VDMA 2632 Part 3 sets out procedures for assessing the classification performance of machine vision systems at acceptance.
Comparing a licensed SDK, a VMS detector or a custom pipeline on your own footage
Our recommendation for choosing between a licensed tracking SDK, a bundled VMS detector and a custom pipeline is one vendor-neutral harness. It should run every candidate on your own camera feeds and report detection or tracking accuracy and added latency as two separate results. TechnoLynx builds such harnesses, agreeing the metrics and annotation protocol, sampling footage across your venue conditions, instrumenting per-frame latency on the target hardware, and handing the harness over for re-runs when a vendor ships an update.
LynxBenchAI publishes results showing how each candidate device runs the public PyTorch stack, which indicates device-level throughput for the harness but not your accuracy; the first step is checking the LynxBenchAI results for your target hardware.
Accuracy should cover detection recall and precision, tracking continuity and ID switches, scored against human-annotated clips. Latency should cover per-frame and end-to-end timing under production-like load. Demo-reel figures were not measured on your rigs, lighting or content, so they cannot stand in for this comparison. The result has limits. It applies to your venue only, it goes stale as SDKs change, harness latency can differ from latency inside the full broadcast chain, and annotation quality caps how far the comparison can be trusted.
The same harness idea applies when the two candidates are FP32 and INT8 builds of one detector. Replace the single aggregate mAP figure with a table of accuracy deltas per object class and per distance band, and re-run it after every requantisation or calibration-data change. Compression disproportionately impacts performance on the under-represented long tail. Which classes and distances are most sensitive depends on the architecture and the training data, so measure it rather than assume it.
How do you build a held-out defect set and acceptance criteria for each cell?
Confirm each cell independently. Give each one a held-out defect set drawn from its own cameras and lighting, gate it on the same false-negative target the pilot cell met, and roll out further only once every cell passes on its own. Treat each cell as a possible covariate-shift domain, where the input distribution changes even though the relationship between inputs and labels stays the same. TechnoLynx can design these per-cell confirmation sets and run the gating tests on each cell’s target hardware.
LynxBenchAI publishes results showing how each device runs the public PyTorch stack, an indication of how a cell’s target hardware runs the model before its own gating tests; the first step is checking the LynxBenchAI results for that hardware.
The held-out set has to be a golden set that remains completely separate from training data. It should contain every relevant defect class, limit samples, product variants and material lots, and reflect shifts and suppliers where production spans them. Rare defects, borderline parts and unusual lots belong in it, because those cases show how the model will behave on the line.
Set acceptance thresholds per defect class. A critical defect may need an escape rate close to zero, while a cosmetic defect can trade missed defects against false rejects differently. Pharmaceutical visual inspection guidance also sets acceptance criteria per defect class. Acceptance criteria for each defect class are based on the probability of detection observed during test-set qualification, with a recommended false-reject target of under 5% of good units. In that guidance, test sets hold roughly 10% defective units to avoid rejection bias. Its typical AQL ranges run from 0.010-0.10% for critical defects to 1.0-4.0% for minor ones. Expectations need setting before any of this. Near-100% accuracy is almost never plausible, so the workflow needs fault tolerance and a human-in-the-loop path for uncertain cases.
Worked example: how many samples a rare-defect claim needs
These are planning approximations. By the textbook rule of three, zero failures in n trials gives an approximate 95% upper bound of 3/n. So 3/n ≤ 0.001 gives n ≥ 3,000 defective samples of that class, per cell. That number is why we would build critical-class sets from seeded or challenge samples rather than wait for natural defects.
Assumption two: you want to estimate the 0.5% prevalence from the Aristek example. Tuyl and Tuyl’s work on rare events recommends judging the margin of error relative to the proportion, with a relative margin between 0.1 and 0.5. A normal approximation gives n ≈ z²(1−p)/(ε_R²·p). At 95% confidence (z = 1.96), p = 0.005 and ε_R = 0.5, that is 3.84 × 0.995 / (0.25 × 0.005) ≈ 3,060 parts. At ε_R = 0.1 it is about 76,400. In the paper’s COVID-19 case study, the simple Wald interval had notably poor coverage.
The right fix for a failing cell depends on the cause of its failure. Bias-driven gaps (systematically shifted predictions) call for a different fix from variance-driven gaps. Variance often signals too little or too poor data for that group and calls for collecting more data rather than tuning. Different shift types need different corrections, which is why we quantify each cell’s shift before choosing a fix.
The same discipline for forecasting and field vision: segments, crops and seasons
Load forecasts and agricultural vision models need the same breakdown: report error by the segments that matter operationally, not one blended number. NIST’s AI Risk Management Framework says accuracy measurements should be paired with test sets representative of expected use and may include results disaggregated by data segment. TechnoLynx frames the stratification scheme against your actual deployment distribution and implements or adapts the per-bucket evaluation pipeline on your PyTorch workload, reconciling bucket-level results with the aggregate figure.
LynxBenchAI publishes results showing how each device runs the public PyTorch stack these workloads run on, an indication for choosing hardware but not a measure of your model; the first step is reviewing the LynxBenchAI results for your candidate devices.
Forecast error by feeder type, region and meter penetration
Break forecast error out by feeder type, region and meter-penetration bucket, so that rural, low-penetration feeders show up as their own row and are not averaged into the headline. A weighting caveat applies. If the buckets are combined with weights that do not match the deployment distribution, the recombined figure can still hide failure.
Generalisation across crop, season and region
A claim that a crop or field model generalises needs a minimum trial matrix (crop × season × region, adding growth stage or lighting where they change the image) with enough samples per cell for a usable interval. D’Amour et al. argue for stress tests built around each application’s real requirements, rather than reliance on standard benchmarks. We would reuse an existing cross-domain harness only when the new domain’s shift is comparable.
Platforms that multiplex third-party models need per-vendor, per-condition thresholds
Per-model, per-condition accuracy breakdowns are achievable and more useful than one platform score, but they depend on testing each vendor’s model yourself. NIST notes that third-party systems complicate risk measurement: a developer’s metrics may not match the deployer’s, and the developer may not disclose its methods. We would run the same degraded-condition scenario set (fog, glare, rain) against each model independently. Where a vendor shares its own data, use it as a cross-check rather than a substitute.
Swapping out a model that fails in fog without breaking a client integration needs a stable, model-agnostic interface and a pre-validated swap process. Even then, the replacement’s latency and output details need re-validation with the client.
How do we keep evaluating the model after go-live without enough labels?
Keep the stratification running after launch. Monitor per site, camera revision or batch. Use label-free signals continuously, use labelled checks whenever outcomes arrive, and set alert triggers per stratum rather than one global threshold. The evidence below comes from a clinical audit, and its mechanism is one we would apply to an inspection line.
LynxBenchAI publishes results showing how each device runs the public PyTorch stack, an indication of how a new site’s or camera revision’s hardware runs the model before per-stratum monitoring; the first step is checking the LynxBenchAI results for that device.
Calibration broken out by site, scanner firmware or batch
A decade-long audit of a frozen clinical prediction model found discrimination stable (AUC 0.76 at development, 0.74 and 0.75 later) while calibration drifted. The calibration intercept moved to −0.54 and −0.31, with both confidence intervals excluding zero. A discrimination-only check would have rated the model healthy. The audit’s monitoring loop treats an intercept confidence interval that excludes zero as the statistical drift signal, acts only when a pre-specified magnitude bound is crossed, and references the calibration slope to its deployment-time value rather than to 1. One more finding matters for fleets: a correction estimated in an earlier period did not transport, and reversed the bias from over-prediction to under-prediction (intercept −0.31 to +0.23). A threshold tuned for one camera revision is a stale correction for the next, so we would set baselines per revision.
Until labels mature, watch the outputs. Zhang et al. list softmax entropy, output confidence, class-wise prediction frequencies and prediction stability as signals for unsupervised monitoring. Thresholds can be set from prior experiments or tuned to keep false alarms at an acceptable rate (they give 1-5% as an example). Windows typically hold hundreds of samples or one to three months of data, with smaller windows reacting faster and larger ones giving more stable readings. Rabanser et al. found that, across the shifts they tested, two-sample tests on representations from pre-trained classifiers performed best for detecting dataset shift without labels. On a line, a rising reject rate is itself a signal: split the results by SKU, shift and material lot, which usually narrows the cause within a day. Our page on revalidation, monitoring and rollback for deployed models covers what happens once an alert fires.
Separating drift from a site that is simply different
Some distribution shifts between sites are benign, and drift detection has to tell them apart from harmful ones. The audit’s monitoring framework checks case-mix on each incoming window, using a population stability index. It confirmed the over-prediction was systematic by finding it within each access stratum, not only in the pooled figure. The inspection equivalent is to condition on part mix, lot and shift before calling anything drift. Not every distribution shift harms performance, and treating benign shifts as alarms wastes effort and can get a system switched off.
The audit’s authors describe the end state well: a model in continued use becomes a monitored process rather than a one-off validated artefact. For an inspection model, a planned change (new supplier, SKU, lighting or camera) triggers a defined assessment against the golden set before it reaches full production.
FAQ
How do you check for data drift without labels?
One practical method trains a domain classifier to separate reference samples from current ones. An AUC near 0.5 suggests no shift, and a high AUC suggests clear separation. Two-sample tests on pre-trained representations performed best in one comparative study.
Is precision more important than accuracy for inspection?
On a line, escape rate and false reject rate usually matter more than accuracy, reported alongside recall and precision. Which one gets the stricter threshold depends on each defect class’s quality risk and the cost of each type of error.
