A credible defect taxonomy labeling standard does three things a written description cannot. It ties each category to measurable criteria and reference images, including the borderline cases inspectors argue about. It says who decides when the criteria still do not separate a part. And it comes with blind agreement evidence showing that different inspectors apply it the same way. Attribute agreement analysis (the attribute form of Gage R&R) tests such categorical calls with agreement percentages and the kappa statistic. Once you have the evidence, it can support a staged go/no-go investment decision, as long as the report states plainly what the study did not cover.

How do we capture what our inspectors look for before they retire?

Capture it while they inspect, not in an interview room afterwards. Researchers studying aerospace inspection, citing Johnson et al. (2019), report that up to 50% of the knowledge expert inspectors use is tacit and does not appear in standard operating procedures. To the person who holds it, that know-how is simply obvious, which is why it was never written down and why getting it out takes a deliberate method.

The aerospace study relied on observational methods. In think-aloud sessions, inspectors talk through their reasoning while they work, prompted with questions such as “How did you come to this decision?” and “Are there any alternatives to this course of action?”. Self-confrontation interviews then play back recordings of their own work so they can comment on habits they have stopped noticing. A simpler knowledge-elicitation question set asks: walk me through it, what goes wrong and how do you know, why do you do it differently from the procedure, and what would you tell someone in their first week. In the aerospace study, sessions ran on parts whose defects had already been measured and sentenced, which allowed richer self-confrontation with the inspectors.

A transcript alone is not what you want out of the sessions. You want the decision variables: the cues that catch an inspector’s attention, the attributes that separate two similar defects, and the zones of a part where tolerances tighten. In the aerospace study, think-aloud sessions brought out that raised material along the contour is distinctive of a nick, and that inspectors feel for it with a stylus. A recording only becomes usable once it has been turned into written procedure that sits alongside the existing documentation.

We would give the sessions to whoever will own the standard afterwards, paired with someone practised at keeping an expert talking while they work, because the owner is the one who has to defend each rule later.

A golden-sample panel is worth building even if no camera follows

A golden sample is an approved reference standard that defines a component’s target state, and in practice it is usually supplemented by borderline examples close to the acceptance limit. That reference is useful without any automation: it gives everyone a shared point of reference for borderline cases, complaints and process drift. If a vision pilot does go ahead, the same set of good parts, defect samples and borderline cases is what the camera and lighting setup gets designed against. The panel is also the physical record of what the retiring inspectors knew.

What should a defect taxonomy labeling standard contain beyond written descriptions?

It needs measurable criteria, a reference image for every rule, borderline examples marked good or bad, defined viewing and measurement conditions, and a disposition for each outcome. The operational definition comes first. If the acceptance criterion is a phrase like “no excessive porosity”, an agreement study only measures how differently people read “excessive”, so categories have to be defined in words, with photos or physical boundary samples, before anyone is tested against them.

ISO 5817 shows the pattern for weld imperfections: its limits scale with plate thickness, so continuous undercut depth may not exceed 0.05t at quality level B, 0.1t at C and 0.2t at D. The actual limits still have to come from your own process and your customers’ requirements.

A criterion also depends on the viewing conditions it is applied under. Methode’s visual quality standard, for example, specifies inspection under 80-120 foot-candles (861-1291 lux) of overhead white fluorescent light. It treats lighting intensity, viewing angle, viewing distance and inspection time as setup factors to adjust when inspection effectiveness comes out low. Acceptance also depends on how you measure (tooling, angle, access), so the measurement method belongs in the work instruction. For disputes, Methode’s standard points to a dot gauge to measure defect size, which turns an argument into a reading.

How many borderline photos per material and thickness? No fixed number covers the edge cases. Let observed disagreement guide the choice: sort trial parts by how much the ratings split, and the few parts causing most of the disagreement show which boundaries need boundary samples. Even a standard that is reproducible on paper has not earned trust until real inspectors have been tested on it, which is what the agreement study below is for.

Checklist for the standard document

• Category definitions taken from the taxonomy, with each imperfection named and referenced to a classification scheme where one exists, as ISO 5817 does with ISO 6520-1.
• For each category: what it looks like (photo or sketch), how it is measured, how it is recorded and what happens next (accept, repair or escalate).
• An example image for every rule, edge cases listed first, and a do-not-label list, on one page as in Ultralytics’ labelling guideline.
• Borderline cases kept as concrete reference cases, each marked good or bad with a justification.
• Viewing conditions (lighting, angle, distance) and the measurement tool for each limit.

Measuring inter-annotator agreement across shifts

Measure it with a kappa statistic on a shared set of images that every inspector labels independently. Cohen’s kappa handles exactly two raters, and Fleiss’ kappa, published in 1971, handles any fixed number of ratings per item, which is what most inspection studies need. Kappa compares observed agreement with the agreement expected by chance, (Po − Pe) / (1 − Pe): it is 1 for perfect agreement and 0 for agreement no better than chance.

The chance correction matters when most parts are good. If a process makes 95% good parts and two inspectors each pass 95% of parts at random, they will still agree 90.5% of the time (0.95 × 0.95 plus 0.05 × 0.05), so a high percent-agreement figure can mean almost nothing. If annotators each label different subsets of a larger pool, Krippendorff’s alpha is the more general choice, because it accepts missing ratings.

A calibration round on shared ambiguous images comes first

Yes, run a calibration session before anyone labels at scale. The Ultralytics guidance is to have everyone label the same 50 images and scale up only when agreement passes. A calibration round does two jobs at once. If trained annotators cannot agree, the guidelines are ambiguous and need revising, and the scores also show which individuals need retraining. Include borderline parts from the contested boundaries rather than a random sample of production. When a reviewer corrects labels quietly, annotators never learn what they got wrong, so reject with a reason and let the annotator fix the label.

What kappa score is good enough to freeze the taxonomy and start training?

There is no universal number. The right gate depends on what a misclassification costs downstream.

Convention	Threshold	Context
Landis and Koch (1977)	0.61-0.80 “substantial”, 0.81-1.00 “almost perfect”	The most quoted scale; the authors themselves called the divisions arbitrary
AIAG Measurement Systems Analysis, as reported by Minitab	0.75 or higher is good agreement; 0.90 preferred	Manufacturing attribute studies, applied within appraisers, between appraisers and against the standard
AIAG rule of thumb, lower bound	Below 0.40 is poor	Automotive attribute agreement practice
McHugh (2012)	Below 0.60 is inadequate	Interrater reliability in health research
Claru annotation glossary	0.70 minimum for classification training data	Production ML labelling
truelabel annotation glossary	0.80 or higher for production-grade labels	Production ML labelling; 0.60-0.79 described as moderate

For an inspection standard, we would treat the AIAG 0.75 as the floor within appraisers, between appraisers and against the standard, and move to 0.90 where a wrong call means a part escapes to a customer or a safety-relevant defect is missed. Many customers already set 0.90 for critical characteristics.

Read kappa by category as well as overall, because a low kappa on one class and a high one on another tells you exactly which boundary needs work. Three caveats keep the number honest. When one class dominates, a few disagreements can pull kappa down sharply while percent agreement stays above 90%, so the part mix has to be deliberate. Small samples produce confidence intervals wide enough to include “no agreement”, and McHugh’s heuristic is no fewer than 30 comparisons. McHugh also advises calculating both percent agreement and kappa.

The threshold matters because label consistency caps what a model can learn. A model trained on labels with 85% agreement will plateau near 85% accuracy regardless of architecture, because the disagreement is noise the model cannot resolve. That is why we treat the taxonomy and agreement work as the data-quality foundation that comes before any model selection: a better network cannot fix labels that people do not agree on.

How do we test that a new written standard actually reduces inspector disagreement?

Run a blind attribute agreement study, also called an attribute Gage R&R study. Each inspector judges the same set of parts at least twice, in random order, without knowing which part is which, against a reference answer an expert has fixed in advance. Attribute data carries little information per rating, so common guidance is 30 to 50 parts or more, with more needed when categories are many or a miss is serious. The parts should be chosen on purpose: clearly good, clearly bad, and a meaningful share of borderline parts on both sides of the limit.

Blinding protects the result from flattering the system. An appraiser who sees a previous reading tends to drift towards it, which makes repeatability look better than routine inspection really is. Operators who can see each other’s calls converge, and the study ends up flattering the system.

The analysis gives four answers: within-appraiser consistency, each appraiser against the standard, agreement between appraisers, and all appraisers against the standard. For pass/fail calls, it also gives each appraiser’s miss rate and false-alarm rate. Agreement between appraisers can be high while everyone is wrong, if they all share the same mistaken criterion, so the comparison against the standard is not optional. AIAG gates each appraiser at 90% effectiveness, a miss rate of 2% or less and a false-alarm rate of 5% or less, and the system’s verdict follows its weakest appraiser.

If your criteria include continuous measurements (dross height read with a gauge, say), a variable Gage R&R applies. AIAG bands treat under 10% GRR as acceptable, 10-30% as marginal and over 30% as unacceptable. Treat these as conventions, not automatic pass/fail rules. Acceptability depends on what the measurement will be used for and the consequences of a wrong decision.

To test whether the new written standard really fixes the disagreement, run the blind re-labelling trial with all the inspectors on the same photo set. Then compare kappa and miss rate against the baseline you measured before the standard existed. We would freeze the taxonomy only when the trial meets the threshold you agreed in advance. If it falls short, fix the definition, boundary samples or lighting rather than relying on retraining alone, and rerun with fresh parts.

Ambiguous parts: a review category, a named adjudicator, or both?

Both are tools, and we would choose between them by how often the ambiguity comes up. No written criterion will separate every dross case from every burn mark, so the taxonomy needs a documented tie-break procedure alongside it. If parts still split inspectors only rarely after the refined criteria are in place, keep the two classes and escalate disputes to a named adjudicator. If ambiguity keeps recurring, add a bounded “review” category with its own downstream handling, such as a hold for engineering sign-off, and route every case in it to the same adjudicator.

Published labelling practice escalates disagreements the same way. In the SurgScan surgical-instrument dataset, disagreements went to a majority vote first, and when discrepancies persisted a neutral reviewer made the final classification. Golden-sample practice defines in advance who has the final say in borderline cases (Quality or Production, for example) and how that decision is recorded. The adjudicator should be a named role, not “the inspector on shift”, so that repeated disputes build up into precedent instead of being argued afresh each time.

In high-stakes labelling workflows, the adjudicator’s decision becomes the training label and the patterns of disagreement feed back into guideline updates. That same source reports adjudication costing 2-5 times the initial annotation cost. We would carry the same feedback loop into a cost-effective human-in-the-loop labelling pipeline for vision models once labelling moves to volume.

If a review category is not time-boxed and audited, it becomes a dumping ground for anything inconvenient. A single adjudicator becomes a bottleneck and a single source of bias, so a backup or rotation is needed. And a precedent log only helps if the written criteria really are updated from it. Planning for a review path is not a weakness in the design.

How do we present agreement evidence in a feasibility case without overclaiming?

Present it as measured evidence with its conditions attached. Phrase the conclusion as “feasible with defined next steps” unless the evidence covers the full deployment scope.

TechnoLynx can frame the feasibility scope with you, define the defect taxonomy and evaluation criteria, run the feasibility assessment on representative data, and write the uncertainty, limitations and next steps explicitly into the evidence package. TechnoLynx can also support bringing in an independent reviewer and help draft the feasibility statement for a ministry or investor audience, within its AI engineering for manufacturing and automotive work.

What the agreement evidence package should contain

• The taxonomy version and the revision of the written standard the study tested.
• The study design: appraisers, number of parts, the good/bad/borderline mix, number of trials and how blinding was kept.
• Kappa overall and per category with the sample size, plus percent agreement, appraiser-versus-standard results and miss and false-alarm rates.
• The intended use, the reference basis, known limitations and the conditions under which the conclusion no longer applies.
• A traceable chain from the inspected feature to the raw trials to the result, because a reviewer will ask how each number was calculated.

Datasheets for Datasets and Model Cards both mandate disclosure of the annotation method, the number of annotators per item, the agreement metric and the threshold for accepting labels. A feasibility case that reports the same items will read as familiar to a technical reviewer.

What a ministry or state investment review expects

Each reviewing body sets its own procedure, as the two examples below show. Under Vietnam’s Law on Public Investment, for example, the project owner prepares a complete feasibility study report, a State Evaluation Council evaluates it, and the report is revised on the basis of the evaluation opinions before the investment decision. Washington State’s feasibility standard for IT investments (adopted July 2023) treats the feasibility study as the point for a go/no-go decision before significant funds are committed. That standard also asks for the rationale behind cost estimates, a summary of expected quantitative and intangible benefits, the medium and high risks with how each will be managed, and whether external quality assurance will be used. Agreement evidence fits into that structure as the risk evidence for the data the system will learn from.

Wording the statement for investors, and whether to add a third party

We would reserve “proven feasible” for evidence that covers the whole deployment scope. Using it earlier risks reputational and legal exposure if scaled deployment later contradicts it. Wert-Berater’s guidance puts the middle path well: conclusions are supported, limitations are acknowledged, and when a key assumption depends on future execution the report says so plainly. The same firm warns that a study finding no meaningful vulnerability in a complex project often deserves the most scepticism.

An independent review earns its cost when third-party scrutiny is inevitable. An independent study prepared to financing standards tends to reduce friction with lenders, agencies and investment committees, though it may not eliminate their questions. Before commissioning one, define the decision the report must support and who will use it, otherwise the result may be informative without being useful for the decision. The review adds cost and time, and its value depends on how independent and credible in the domain the reviewer is.

Staging the capital decision: pilot, augment or commit

The right default is a staged decision: a time-boxed pilot, a written success metric with a baseline, and an agreed condition for stopping, before a larger budget is committed. This applies whether the investment is an inspection cell for one material and thickness band or a fine-tuned language model for one content format.

Who can help us decide between a pilot and the full budget?

TechnoLynx helps teams make that staged decision on evidence. It frames the pilot scope (which subset, which format or segment) and its success criteria, runs cost-quality comparisons between hosted frontier models and candidate open-weight models where the investment is a language model, and measures generalisation risk per segment where failure-threshold data exists. The engagement ends with a go/no-go and sequencing recommendation with cost estimates for the board; for language-model decisions, the LLM Selection Pack builds the eval suite and comparison report behind it. Where the decision includes the hardware an open-weight model would run on, LynxBenchAI is the measuring tool, benchmarking throughput, latency, memory and cost per unit of work across accelerators.

The pack does not suit a programme whose open question is still labelling agreement, because no model comparison is meaningful until the labels are stable. Nor does it suit buyers looking for the cheapest hours of build effort.

Full budget, a smaller pilot, or more data first?

Run the pilot on a subset before committing the whole budget. The point to walk away is the stop condition you wrote down before the pilot started. A pilot that stays “almost done” for months is the failure mode itself. If the pilot clears its metric at an acceptable cost relative to the baseline, move to the next stage.

Ranking segments by generalisation risk and cost

The answer depends on what you have measured. Rank the candidate formats or segments by expected generalisation risk and by the cost of retraining versus reusing the existing model, then pilot the segment that buys the most information for the least money. We would want reliable per-segment failure-threshold measurements before funding the next pilot. Without them, the ranking is an informed guess and should be labelled as one in the board paper.

FAQ

Is tacit knowledge the same as tribal knowledge?

They overlap. Tacit knowledge is the know-how and intuition that comes with experience. “Tribal knowledge” usually means the judgement, workarounds and situational know-how that a group never wrote down because to its holders it seemed obvious.

When should you use weighted kappa or another statistic instead of plain kappa?

Weighted kappa gives partial credit for disagreements between adjoining categories. For ordinal ratings such as a 1-5 severity scale, Kendall’s coefficients, which account for ordering, are usually more appropriate than kappa alone.