A rollback runbook for a line-side computer-vision inspection model is a written, rehearsed document with four mandatory parts: named trigger conditions tied to drift telemetry, a pinned last-known-good model version that can be restored without a rebuild, a defined degraded mode the line runs in while the model is out, and an evidence capture step that records what the model was seeing when the trigger fired. Anything less is not a runbook — it is an assumption that somebody will notice.
That assumption is the common state of affairs. Rollback gets treated as an implicit fallback: if the inspection model misbehaves, an operator will spot it, someone will disable it, and the line reverts to manual checks. Two things are missing from that plan. Nobody has defined the threshold at which the model is judged to be misbehaving, and nobody has rehearsed the switch. So the first real drift incident becomes an unplanned line stop, spent arguing about whether the model is actually wrong while scrap or escape rates accumulate.
What a rollback runbook for a line-side CV model looks like
The template below is deliberately short. A runbook that runs to twenty pages does not get read at 03:00 on a night shift.
| Section | Must contain | Common failure if omitted |
|---|---|---|
| Trigger criteria | Named, numeric conditions on drift telemetry already instrumented on the line, plus a manual trigger with a named authoriser | Rollback becomes a judgement call escalated to whoever is awake |
| Authorisation | Who may trigger on shift without waiting for the build team, and the on-call escalation path above them | Model stays live through the incident because nobody feels entitled to stop it |
| Restore action | The pinned last-known-good model version identifier and the exact command or HMI action that restores it | Restore turns into a redeploy of the whole line stack |
| Degraded mode | Which of manual inspection, shadow mode, or reduced-scope classification applies, and the throughput assumption for each | Line runs at unknown inspection coverage while a fix is attempted |
| Evidence capture | Frames or summary records from the trigger window, model version in force, product and SKU inspected, timestamps | Incident is undiagnosable and the rejection-rate record for that window is uninterpretable |
| Return-to-service | Conditions under which the new model goes back in, and who signs that off | Model is disabled permanently — the reversion-to-manual outcome |
Every row of that table is an artefact with an owner. The runbook is the index; the pinned version, the telemetry thresholds, and the evidence store live elsewhere and must be reachable from it.
How are trigger thresholds set without resorting to gut feel?
Thresholds come from the baseline the drift telemetry established during commissioning, not from a number someone liked. In practice this means a rejection-rate band per SKU measured across the commissioning shifts, a confidence-distribution shift measure on the same sampling policy the line runs in production, and an input-side check — mean luminance, focus score — that fires before the decision-side metrics move.
Two properties matter more than the specific statistic chosen. First, the trigger must be computable from telemetry the line already emits, because a trigger that needs a new data pipeline will never be armed. Second, single-metric thresholds miss compound drift: a lighting change and a fixturing tweak that are each inside tolerance can jointly push a model past its decision boundary without any individual monitor tripping. Triggers should therefore include at least one composite condition — for example, rejection rate outside band and input luminance shifted beyond the commissioning range — alongside the single-metric ones. The compound failure modes that make this necessary are catalogued in our wider work on production AI reliability, which is where the trigger set should be written against.
Pinning a version so restore is not a rebuild
Model-version pinning is the part teams most often discover they do not have. The test is blunt: can the shift team restore the previous inspection model without a container build, without network access to a registry outside the plant, and without the original build engineer? If any of those three is required, the runbook’s restore step is fiction.
What that usually looks like in a working deployment: the serving artefact is an exported, immutable model file — commonly ONNX, or a TensorRT engine built for the exact device on the line — held on local storage alongside its predecessor, with the active version selected by a config pointer rather than by which image was last pushed. The engine build is the awkward case, because a TensorRT plan is tied to a specific GPU and library version and cannot be rebuilt quickly on a stopped line; that is precisely why the known-good plan file is retained rather than regenerated. Two versions retained locally is the minimum we look for. The pointer switch, the version identifier, and the checksum belong in the runbook verbatim.
A restore that takes minutes changes the economics of the whole incident. A rehearsed rollback is what converts a drift incident from a multi-day investigation into a recovery measured in hours, with the line running on a known-good version or a defined degraded mode in the interim.
Choosing the degraded mode before you need it
“Degraded mode” is not one thing, and the choice has throughput consequences that production planning needs to agree to in advance.
- Manual inspection — full human check on the affected station. Highest confidence, lowest throughput; requires that the manual procedure still exists and that operators have been trained on it recently enough to execute it.
- Model in shadow — the model keeps scoring and logging but no longer gates accept/reject. Preserves the telemetry needed to diagnose the incident while removing the model from the quality decision. Best default when the failure is suspected drift rather than a hard fault.
- Reduced-scope classification — the model continues to gate only the defect classes whose behaviour is unaffected, with the rest routed to manual. Useful when telemetry attributes the drift to a specific class, and dangerous when it does not.
Shadow mode is the one most often skipped and most often the right answer, because it keeps the evidence trail intact. Disabling the model entirely loses exactly the signal needed to explain the incident afterwards.
Why the evidence step protects your quality data
The rollback record has a second job beyond diagnosis. It documents which product was inspected under which model version, across which window. Without that, the quality data spanning the incident is not interpretable — a rejection-rate figure that mixes two model versions and an unrecorded degraded-mode period cannot be defended to a customer or an auditor later. We treat the evidence capture step as a data-integrity control, not as an engineering courtesy.
Rehearsal, and what a line change does to the runbook
This is where Rollback Runbook Line Side gets specific. The rehearsal we ask for is modest and specific: a planned rollback during a scheduled changeover or maintenance window, executed by the shift team without the build team touching anything, timed end to end, with the resulting duration written into the runbook as the recovery expectation. Once at commissioning, then again after any change that alters what the model sees.
Line changes are the revalidation trigger. A lighting retrofit, a packaging redesign, or a re-shimmed fixture invalidates the baseline the thresholds were derived from, which means the trigger criteria are now describing a line that no longer exists. Re-baseline first, then re-arm the triggers, then confirm the pinned known-good version is still known-good under the new conditions — a model pinned against the old lighting is not a safe restore target.
The rollback runbook sits inside a wider set of artefacts — drift telemetry, reproducible build evidence, named ownership — that we cover in the reliability artefacts an industrial CV inspection pack needs beyond an accuracy table. The runbook is the one exercised after go-live rather than before it, which is why it is the one most likely to be missing.
The open question on most lines is not whether a runbook exists but whether its recovery time has ever been measured. If nobody can state, in hours, how long it takes to get this line back onto a known-good inspection model, the number in the plan is a guess.
Frequently Asked Questions
What does a rollback runbook for a line-side CV model look like in practice? It is a short document with six sections: trigger criteria, authorisation, restore action, degraded mode, evidence capture, and return-to-service. Each section names an owner and points to a real artefact — the pinned model version, the telemetry thresholds, the evidence store. Length is a liability; it has to be executable on a night shift.
What trigger conditions justify a rollback, and how are thresholds set against drift telemetry rather than gut feel? Thresholds derive from the commissioning baseline: a per-SKU rejection-rate band, a confidence-distribution shift measure on the production sampling policy, and an input-side check such as mean luminance or focus score. At least one trigger should be composite, because compound drift — lighting plus fixturing, each individually in tolerance — moves the rejection rate without tripping any single monitor.
What is the defined degraded mode while the model is out — manual inspection, shadow mode, or reduced-scope classification? All three are valid; the runbook must name which one applies and state its throughput assumption. Shadow mode is the strongest default for suspected drift because the model keeps logging while no longer gating accept/reject, preserving the diagnostic trail. Reduced-scope classification only works when telemetry has attributed the drift to specific defect classes.
How is model-version pinning implemented so a known-good version can be restored without a rebuild or redeploy of the whole line stack? The serving artefact is an immutable exported file — typically ONNX or a device-specific TensorRT engine — retained on local storage alongside its predecessor, with the active version chosen by a config pointer rather than by the last pushed image. The test is whether the shift team can restore it without a container build, without external registry access, and without the original build engineer.
Who is authorised to trigger a rollback on shift, and what does the on-call escalation path look like? The runbook names shift-level authority explicitly, so the rollback does not wait for the build team. Above that sits the on-call path for the line’s inspection asset, which should be the same rota that covers the rest of the station rather than a separate CV-team arrangement. Ambiguity here is why models stay live through incidents.
What evidence must be captured during a rollback so the incident can be diagnosed and the rejection-rate record stays interpretable? Frames or compact summary records from the trigger window, the model version in force, the product and SKU inspected, and precise timestamps for the trigger, the restore, and the return to service. This is a data-integrity control: without it, quality figures spanning the incident mix two model versions and an unrecorded degraded-mode period.
How is the runbook rehearsed and revalidated after a line change, and how often? Rehearsal is a planned rollback in a scheduled changeover window, executed by the shift team alone and timed end to end, with the measured duration written back into the runbook. Do it at commissioning and again after any physical change to lighting, fixturing, or packaging — those changes invalidate the baseline the triggers were derived from and may invalidate the pinned restore target too.
Next time Rollback Runbook Line Side comes up
Treat Rollback Runbook Line Side as an engineering problem with a measurable answer, not a positioning question. The teams that do tend to ship the boring, correct version first.
