An inspection system stays reliable through change when four things are settled before the model is built. The defect taxonomy describes how defects form, not how one supplier’s material happens to look. It carries a version number and a change log. The lighting and data have been tested against that taxonomy before anyone trains a model. And every model swap should go through a shadow run, a sign-off on defined evidence and a rollback staged in advance. Feasibility and taxonomy come first, then lighting, then labels bootstrapped from the inspections already happening, and only then the release gates that let a model change without breaking detection.
A detector trained on a vague category, under lighting nobody controlled, is unstable however good its architecture is.
Start with feasibility, not a model
Building formal defect models and putting them into tools is very expensive. That is why one field study of software defect elicitation argues that you first identify the relevant defects for the specific context and assess how often each occurs and how severe it is, then use that assessment to decide whether the investment is worth making. The same study built its taxonomy by coding each defect with its cause and effect, then organising the codes into a hierarchy. It stopped interviewing once no new defects appeared, and treated that saturation as its sign of comprehensiveness.
They contrast this with Orthogonal Defect Classification, which aims to be independent of any one product or organisation. For an inspection line we would take the minimal route. A small set of categories that the line’s engineers recognise is easier to keep stable than a general scheme imported whole. Our companion piece on building a defensible defect taxonomy and baseline from messy, local data covers the baseline side of this work in more depth.
Writing defect criteria that survive a change of steel supplier
Criteria survive a supplier change when each category is defined by the physical mechanism that produces the defect, such as dross adhesion versus localised overheating on a cut edge, and not by the surface texture of one steel grade. A design study of material standards puts a related requirement first in its list: materials in the same grade should behave in similar ways regardless of supplier.
A new die or a new batch of steel on its own should not reopen the taxonomy. If a defect fits an existing mechanism-based category, it is classified there even when it looks different, and only a genuinely new defect mechanism triggers a controlled version change.
The Sandia literature review on visual inspection reports that, without clear defect definitions, inspectors in the telecommunications industry formed personal criteria. Those criteria drifted over time, so products rejected at one point were accepted at another. The same review notes that comparing an item against a standard turns an absolute judgement into a comparative one, and inspection tasks that use standards tend to detect defects better.
There are two limits to mechanism-based categories. A different alloy family can bring failure modes that the feasibility study never saw. Categories drawn too broadly can also merge defect types that matter for downstream grading, so versioning discipline has to be balanced against whether the taxonomy is still useful on the line.
Should the defect standard carry a version number?
Yes. Version the standard and publish a change log beside it. The material-standards study asks that the structure of a standard be motivated, that the reasons for changes be stated, and that a description of each revision be published along with the standard. The standard also needs a written approval route: who can add or split a category, and what evidence they need to see first.
IPC-A-610 gives a model: when text and pictures disagree, its written criteria take precedence over the illustrations, and it sets an explicit order of precedence between the procurement agreement, the master drawing and the standard itself. A part type the standard did not anticipate is then handled by rule, not improvisation, as the table below sets out.
Decision table: does this new observation change the taxonomy?
| What the line sees | Taxonomy action | Who approves |
|---|---|---|
| A new die or material batch, same physical defect mechanism, different visual texture | No change; classify under the existing category and add the images as examples | Inspection lead, recorded in the change log |
| A defect produced by a mechanism no category covers | Minor version: add a subclass through change control | Taxonomy owner, with sample evidence |
| A part type whose defects cannot be described by the current classification logic at all | Major version: revise the classification logic, re-label the affected data | Taxonomy owner and quality sign-off |
| Disagreement between written criteria and reference images | Written criteria govern; fix the images | Taxonomy owner |
Will our fixed station lighting show what separates the defect types?
The direct test is to image known defect samples under the current fixed lighting and check whether the features that separate the defect types are clearly visible. If those texture and depth cues are washed out or ambiguous, the lighting should be redesigned, for example with low-angle illumination, before the pilot starts.
The Sandia review notes that some defects show up better under diffuse lighting and others under directional lighting. For textured or reflective surfaces, low-angle darkfield lighting is a recognised option. Edmund Optics describes how a low-angle ring light raises contrast on features that scatter light while leaving smooth areas dark, which makes fine surface detail visible. A liquid-vaccine inspection study calls correct lighting a prerequisite, since under natural lighting low contrast makes anomalies infeasible to identify by human or automated inspection.
One machine-vision lighting guide treats poor lighting as a data problem, not just a camera problem. It quotes a widely cited industry benchmark: if just 10% of training images are inaccurate, correcting the model afterwards needs three times more data. Lighting conditions also change over time. The guide notes that replacing a fluorescent fixture with an LED changes colour temperature and wavelength enough to shift how the camera sees the part. It adds that operator lights switched on and off independently introduce variation between shifts. Its recommendation is a dedicated, controlled light source for each inspection.
Test across the full range of parts the station handles, not a single sample. A setup that separates one pair of defects may also fail on categories added in a later taxonomy version. That makes a lighting check part of every minor version bump. The wider question of whether a detection setup is repeatable under lighting and sensor variation is covered in validating feasibility and repeatability of vision-based defect detection before production.
Can technician inspection findings bootstrap the first labels?
Yes. Technician walk-through findings and pass/fail calls can seed a first labelled dataset while manual inspection carries on unchanged. One condition comes first: the taxonomy has to be defined and its feasibility checked, or the findings have no stable categories to map onto.
Published work supports starting with weak labels instead of waiting for a clean annotation effort. In a study of mixed supervision for surface-defect detection, the authors found that a large set of weakly labelled data mattered more than a fully labelled set ten times smaller. They also found that adding just 5–10% fully labelled data often gave almost the same performance as labelling everything fully. They note that in existing industrial production control, weakly labelled data can be obtained at no cost.
The vaccine-vial study shows the next step: growing the seed set through self-training. The team hand-labelled 14,000 training and 6,000 evaluation samples from 160 vials. Ten 3D-ConvNets trained on that set reached mean AUROC scores of 0.94 for positive samples and 0.93 for negative ones. To add labels automatically, they used a voting rule: a sample was labelled positive only when more than one classifier predicted it positive with above 0.8 certainty and the classifiers’ saliency maps agreed on where the anomaly was. On held-out anomaly-free vials, the rule kept false positives to 4.5%. After 217,888 self-labelled samples were added, AUROC rose to 0.96 for both positive and negative samples. Automatic labelling needs a defined level of strictness to keep noisy labels out.
Existing inspection records can be noisy: on a PCB line, nearly 20% of components that automated optical inspection (AOI) flagged as defective turned out not to be defective when an operator checked them under a microscope. Technician notes also need checking for their own vocabulary, missing severity grades and gaps on rare defects. A reconciliation pass should map those notes onto the formal taxonomy before the labels are trusted. Manual inspections running in parallel should then keep supplying better labels to refine each dataset version. How many images per class to collect and where to source them is a separate planning question, beyond the bootstrap mechanism described here.
Keeping inspection reliable while the model is retuned
While the model is retuned, a documented shadow or parallel scheme can cover the gap. The model’s output should be spot-sampled against trained human inspectors, escalating to full manual inspection only if agreement falls below a threshold agreed in advance.
TechnoLynx can define the spot-sampling rate and shadow-mode protocol so the interim process produces auditable evidence, and can specify the agreement and escalation thresholds against which that process is judged sufficient. This sits within TechnoLynx’s AI engineering work for manufacturing and automotive lines.
The human reference that spot-sampling checks against is itself imperfect. The Sandia review cites a minimum error rate of around one in a thousand, mainly for simple accept/reject tasks, and error rates of 20% to 30% frequently quoted across many inspection task types. It also names missing data on inspector accuracy as one of the weakest links in a quality control system, because optimal sampling procedures can only be set once that accuracy is known. The acceptable miss rate for the interim period therefore has to be agreed with quality and compliance before the period starts, not inferred afterwards.
Interim reviewers brought in at short notice should be tested against a master list of defects. The review recommends that test samples include a representative range of defects, with enough in each category for statistical analysis. Shift design also has to allow for the vigilance decrement the review describes. The review recommends that inspectors spend no more than 30 minutes at a time inspecting, or alternate with other tasks. Where a re-inspection station exists, a field guide on automotive edge inspection notes that even a 5–10% sample of decisions joined to downstream outcomes gives statistically meaningful accuracy estimates.
This is a stop-gap that holds only until the retuned model has been revalidated.
Do we need a shadow run before cutting over to an INT8 inspection model?
Yes. A quantised or retrained model should run in shadow alongside the active model before it touches a reject decision, with rollback defined before the canary begins. Production AI systems regress in ways unit tests do not catch, including silent quality drift after a model update and failures on rare classes.
AWS’s SageMaker documentation describes shadow testing in the same terms. A copy of each live request goes to the candidate, and only the production variant’s response is returned. The automotive edge-inspection guide treats it as non-negotiable that no candidate prediction reaches the IO adapter or the programmable logic controller (PLC) that drives actuation during shadow mode. The shadow log is then reviewed for score divergence, label disagreement rate, tail latency and miss patterns per defect class before the candidate can move to canary. The guide’s typical canary scope is one station running one product family for at least a full eight-hour shift or a defined count such as 10,000 parts, whichever comes first.
For INT8 specifically, a published account of large-scale low-precision inference describes a continuous accuracy monitor. The monitor quantises each retrained snapshot and evaluates the FP32 and INT8 models side by side to catch degradation early. The INT8 model is rolled out to all traffic only once online experiments meet the launch criteria.
What is the rollback path if drift appears after go-live?
The rollback path is the higher-precision model, kept loadable, with rollback defined, tested and rehearsed before the canary begins. Google’s ML Test Score treats quick, safe rollback to a previous serving version as a key part of incident response. Compatibility is part of the artifact contract. A systems-design guide on quantisation warns that if the quantised candidate changed output schemas, thresholds or downstream assumptions, simply switching model files may not restore the former system.
The triggers deserve the same care. In the automotive guide, a single confirmed defect escape during the canary triggers an immediate rollback with no grace period and a mandatory incident review. The same guide suggests retraining triggers for slower drift: a Population Stability Index (PSI) above 0.20 on an input feature sustained over a shift, in-field precision or recall more than 3 percentage points below validated figures on a critical class, or a confidence-boundary rate above 15% of frames. It also notes that a shift in confidence distribution often comes before visible accuracy loss, for example after a lighting change, a fixture adjustment or a new part variant. That ties model monitoring back to the lighting and taxonomy controls above.
Who signs off an inspection model swap, and what evidence should they see?
A named quality authority should sign off, through the review gate the plant already uses, and should see evidence against thresholds set before the shadow run. The automotive guide records each promotion with the operator ID, a timestamp and the hardware-in-the-loop run that authorised it. The quantisation guide says the release gate should fail closed: missing evidence, an unmeasured target device or an unusable rollback blocks release, however good the model-size result looks.
A sign-off pack we would expect to see:
- Shadow-run divergence and label-disagreement rates against the active model
- Per-defect-class recall on critical classes, with no regression
- p99 latency from frame acquisition to PLC write, within the station’s budget on the hardware-in-the-loop rig
- Manifests for both artifacts, including calibration data and the higher-precision rollback checksum
- A rehearsed rollback procedure and a named person who can stop promotion
Where an external regulation also sets a threshold, the stricter of the internal policy and the external rule should govern. Shadow runs also miss rare events that only appear at full exposure, so post-cutover monitoring is part of the evidence, not an afterthought. This is engineering evidence for a release decision, not safety certification or regulatory sign-off.
How often should the defect taxonomy be reviewed?
Review the taxonomy on a fixed cycle, such as once a year, and also whenever a signal says the world has moved. Useful signals are a new material or die family, a lighting or fixture change, a sustained rise in confidence-boundary frames, or drift alerts. Surfaces genuinely change: a 2024 study on drift detection for industrial images points out that tool wear and grinding can alter surface characteristics enough to cause large differences in model performance.
The review should involve the people who see defects first-hand, and that needs a standing process rather than occasional contact. One vocational college uses an advisory board of field professionals to keep its curriculum aligned with employers’ expectations. A DACUM panel of expert workers, followed by a gap analysis against current content, is one documented way to turn employer input into specific changes. Two cautions apply. Input from a single employer can skew the taxonomy towards one company’s practices, and participation may lapse without someone managing the relationship.
The same discipline outside the factory: damage severity from claim photos
Feasibility-first thinking transfers to any classification built on images nobody controlled, and insurance photos of roof and siding damage are a clear case.
Can policyholder photos support damage severity classification at all?
That cannot be answered honestly without testing the insurer’s own claim photos. A cheap sample audit comes first: manual review of real photos against the severity distinctions, plus a simple baseline classifier, before any pilot is funded. If most photos are usable and the baseline shows signal at the key severity thresholds, proceed. If not, change the photo capture process first.
One published deployment shows which photos a damage model struggles with. In a damage-assessment system for social media imagery during Hurricane Dorian, the authors found the system was weak on foggy or blurry scenes, low-light scenes and flooding photographed from afar. Against expert review, it reached 74% accuracy on the severity task. The CrisisMMD dataset includes a “don’t know or can’t judge” class for low-quality images. An audit should count how often claim photos fall into such a class. It should do so separately for roof and siding, and across regions and seasons, because a small sample can hide that variation. A baseline classifier is only a rough feasibility proxy, not a sign of production readiness.
Can we agree one severity taxonomy before anyone builds or buys a model?
You can, and you should. Fix one severity taxonomy and labelling protocol before labelling starts, revisiting it only through a controlled change cycle like the one on the factory side. Senior adjusters hold the domain judgement, so they should author it. An external consultant is worth adding only when internal consensus stalls or alignment with outside standards is needed, and the vendor should not define the ground truth it will be measured against. The software defect-elicitation field study above ended with a validation workshop in which project partners reviewed the results before any tools were built, which is a sensible model for adjuster sign-off.
For siding, one published assessment framework classifies damage on two independent axes. Severity runs from cosmetic through moderate functional and severe functional to structural. System depth runs from cladding-only through weather-resistive barrier involvement to sheathing and framing. The same source notes that Section R703 of the International Residential Code sets minimum performance thresholds for exterior wall coverings. It also notes that adjusters, code inspectors and contractors routinely reach different conclusions about the same structure because they apply different investigation standards. That divergence is a reason to fix the insurer’s own definitions before labelling. Any mapping from a roofing severity scale to an industry standard has to be built and checked case by case. Finally, a taxonomy finer than the photo evidence can resolve will only produce inconsistent labels, so detectability should limit how granular it gets.
FAQ
Is quality control the same as inspection?
Not quite: inspection is one activity within quality control. The Sandia review describes it as a careful examination for flaws, made up of set-up, presentation, search, decision against standards, and the accept or reject response. Quality control planning also depends on knowing how accurate that inspection is, for example to set sampling procedures.
