An accuracy table is not a deliverable. It is one line item in a longer inventory, and on a production line it is not even the line item that decides whether the inspection model is still in service six months after go-live. What decides that is how fast a degradation gets detected, attributed to a cause, and reversed.
That is the gap this piece fills: not the argument for reliability work — our production AI reliability practice covers why pilot-condition evidence stops predicting line behaviour — but the itemised contents list. What belongs in the pack, what each artefact actually contains, what evidence proves it exists in usable form, and which artefacts have to be rebuilt per line versus written once and reused.
What does “reliability artefacts beyond accuracy numbers” mean in practice?
It means the pack answers a different question than the pilot did. The pilot answered does this model separate good parts from bad on a curated image set. The pack has to answer when the lamp above station 4 is replaced with a different colour temperature, how long before anyone knows, who is told, and what do they do.
Those are not the same question, and the second one is not answered by a better confusion matrix. It is answered by artefacts: a baseline capture that records what the line looked like when the number was true, telemetry that detects departure from it, a pinned version that can be restored, and a named person whose rota includes the model.
The first uncontrolled change is where the divergence shows. A carton redesign, a fixture shimmed by a maintenance shift, a conveyor speed step after a line refresh — an accuracy-only pack has no mechanism to notice that anything moved. The model keeps emitting confident decisions against a distribution nobody re-measured, and the rejection rate that quality reporting depends on quietly stops meaning what it used to mean.
The minimum artefact inventory
Seven artefacts. Fewer than seven and there is a specific failure the line cannot recover from without the original build team.
| # | Artefact | What it contains | Evidence it exists in usable form | Scope |
|---|---|---|---|---|
| 1 | Condition baseline | Captured images plus recorded lighting, fixturing, camera pose, lens, SKU set and takt at the moment the release number was measured | The capture set opens and reproduces the release metric within tolerance when re-scored | Line-specific |
| 2 | Drift telemetry | Sampled per-inspection summary records, score and rejection-rate distributions over time, alert thresholds | A dashboard or query that returns the last 30 days without a data engineer being involved | Line-specific config, reusable pipeline |
| 3 | Rollback runbook | Named trigger conditions, pinned last-known-good version, the switch procedure, expected line downtime, who authorises it | A rehearsal record — the rollback was executed on the line, timed, and signed | Template reusable, triggers line-specific |
| 4 | Version pinning and build provenance | Model artefact hash, training data snapshot ID, preprocessing config, runtime and driver versions (e.g. the CUDA and TensorRT versions the engine was built against) | Rebuilding from the recorded inputs produces a functionally equivalent artefact | Reusable practice, per-release record |
| 5 | Per-line accuracy and rejection record | Confusion matrix, false-reject and false-accept rates, and measured rejection rate reported per line and per SKU family, with the conditions attached | The numbers carry their measurement conditions; aggregate-only reporting fails this test | Line-specific |
| 6 | Failure-mode register | Known failure classes for this line, what each looks like in telemetry, and the remediation for each | Each entry names a detection signal, not just a description | Partly reusable, line-specific instances |
| 7 | Ownership and handoff record | Named accepting owner, on-call rota entry, escalation path, what training was given, acceptance date | The named owner can be paged today and knows the model is theirs | Line-specific |
The scorecard question we use when reviewing a pack is deliberately blunt: for each row, present, partial, or missing. Partial is the common answer and the honest one — a rollback runbook that exists as a document but has never been rehearsed is partial, not present.
What counts as evidence rather than assertion
A pack is asserted when its artefacts are described in a slide and real when someone other than the author can act on them. The distinction matters because the two are indistinguishable at a handoff review unless the review is designed to tell them apart.
The evidence column above is doing that work. A baseline capture that cannot be re-scored is a folder of images. Drift telemetry that requires a bespoke script from the original team is not telemetry the plant owns. A pinned version whose build inputs were not recorded is a binary nobody can reproduce after the next runtime upgrade, and it will fail at exactly the moment it is needed — during a rollback, under time pressure, with the line stopped.
In our experience the cheapest test is a dry run: take a plausible incident — the lamp change — and walk the accepting owner through detection, attribution, and reversal using only what is in the pack. Where they get stuck is the missing artefact. It takes an afternoon and it finds more than a document review does.
Line-specific versus reusable
Two artefacts are genuinely reusable across lines and plants: the rollback runbook template and the version-pinning practice, including whatever build-provenance mechanism the team standardises on. Everything else has a line-specific core, because everything else encodes physical conditions.
That distinction is worth holding onto during a rollout. Teams that treat the whole pack as reusable copy a baseline from line 1 to line 3 and then wonder why the drift alerts fire constantly on a line whose lighting was never the same. Teams that treat the whole pack as line-specific rebuild the runbook seven times and ship it late. The failure classes each artefact is designed to catch are shared; the thresholds and baselines are not.
Where accuracy fits once the rest is in place
Accuracy does not disappear from the pack. It changes status — from release gate to monitored quantity with conditions attached. The useful form is per line and per SKU family, with the measurement conditions recorded next to the number, so that when the number moves there is something to compare it against.
Aggregate accuracy across a plant is the reporting form most likely to hide a problem: one drifted line averages out against six healthy ones, and the rejection rate feeding quality reporting is corrupted without any single figure looking wrong. We treat aggregate-only accuracy reporting as a gap in row 5, not as a presentation preference.
Which gaps send a line back to manual inspection
The gaps that matter most are the ones that make an incident unrecoverable rather than merely uncomfortable. Missing telemetry (row 2) means the degradation is discovered by an operator noticing too many false rejects, typically weeks late. Missing rollback (row 3) means the only available response is switching the model off. Missing ownership (row 7) means nobody was going to act anyway.
Those three compound. A line with none of them experiences its first drift incident as an unplanned stop, resolves it by reverting to manual checks “temporarily”, and the temporary state persists — commonly within a quarter of go-live. With telemetry that localises the change and a runbook that restores a pinned version without re-validation, the same incident is hours rather than days, and the model stays in service.
Who signs off, and what a handoff review looks like
Sign-off splits by artefact. The build team signs rows 1, 4, and 6 — they are statements about how the model was made and measured. The plant signs rows 3 and 7 — they are commitments to act. Row 2 is joint: the build team certifies the telemetry works, the plant certifies it is monitored by someone. Row 5 is signed by whoever owns quality reporting, because that is where a corrupted rejection rate does its damage.
An acceptable handoff review is the dry run described above plus a rehearsed rollback, with the accepting owner driving both. Not a walkthrough by the vendor. If the review is a presentation, it is testing the slide, not the pack.
Frequently Asked Questions
What does “reliability artefacts an industrial CV inspection pack needs beyond accuracy numbers” mean in practice?
Accuracy alone tells you nothing about whether your model will survive contact with production. It means the pack must answer how a degradation is detected, attributed, and reversed — not just how well the model separated good from bad parts on a curated pilot set. Accuracy describes the model at one moment under one set of conditions; the other artefacts describe what happens when those conditions change.
What is the minimum artefact inventory for a line-side inspection pack, and what does each artefact actually contain?
Seven: condition baseline, drift telemetry, rollback runbook, version pinning with build provenance, per-line accuracy and rejection record, failure-mode register, and ownership/handoff record. The table above lists the contents of each. Below seven, there is a specific incident class the plant cannot recover from on its own.
What evidence proves an artefact exists in usable form rather than being asserted in a slide?
Someone other than the author can act on it. A baseline that re-scores, telemetry queryable without the build team, a rollback that has been rehearsed and timed, a build that reproduces from recorded inputs, and an owner who can be paged today.
Which artefacts are line-specific and which can be reused across lines or plants?
The rollback runbook template and the version-pinning practice travel. Baselines, telemetry thresholds, per-line accuracy records, failure-mode instances, and ownership records are line-specific because they encode physical conditions that differ between lines even in the same plant.
How do accuracy metrics fit into the pack once the other artefacts are present, and which metrics should be reported per line rather than in aggregate?
Accuracy becomes a monitored quantity with its measurement conditions attached rather than a release gate. Confusion matrix, false-reject rate, false-accept rate, and measured rejection rate should be reported per line and per SKU family; aggregate plant-level reporting hides a single drifted line inside healthy averages.
What gaps in the inventory most often explain a line reverting to manual inspection after go-live?
Missing drift telemetry, missing rollback runbook, and missing named ownership — usually together. Without them the first uncontrolled change becomes an unplanned line stop resolved by switching the model off, and that state tends to persist.
Who signs off each artefact, and what does an acceptable handoff review look like?
The build team signs the baseline, version pinning, and failure-mode register; the plant signs the rollback runbook and ownership record; telemetry is jointly signed; the per-line accuracy record is signed by quality reporting. An acceptable review is a dry-run incident walkthrough plus a rehearsed rollback, driven by the accepting owner rather than presented by the vendor.
If your pack has all seven and the accepting owner drove the rehearsal, the remaining question is a harder one: how often does the inventory itself need re-validating as the line changes underneath it?
Three non-negotiables for your Beyond Accuracy implementation
Start by auditing whether your current telemetry can even surface the failure modes that matter most in your deployment context. If Beyond Accuracy is on your roadmap, the next step is to map it onto your own constraints rather than copy a reference architecture.
