Rollback is the part of a computer-vision inspection deployment that gets written down last and needed first. The usual arrangement is an implicit one: the previous model file sits on a shared drive, and everyone assumes an engineer can swap it back at the next shift change. That assumption holds until the moment a model regression is actually happening on a running line, at which point nobody can agree on whether the model is wrong, who is allowed to change it, or what the line inspects while the change is being made.
A designed rollback path is a different object. It has four parts: pinned versions that include the preprocessing and thresholds the weights were validated against, degradation triggers that authorise the rollback without a debate, a fallback inspection mode the line can hold while the model is out, and a named owner on shift who can execute it. Those four parts are what turn “we could revert” into a rehearsed operational path with a measurable duration.
This article covers rollback mechanics only. The broader hardening question — lighting drift, packaging revisions, conveyor variance, and the rest of the operating envelope that makes a pilot number stop transferring — belongs to our wider treatment of taking a CV defect-detection pilot from validation cell to production line.
What gets rolled back — and why “the model” is the wrong unit
The most common design error is treating the model weights as the rollback unit. On an inspection line, the decision a camera station emits is produced by a chain: acquisition settings, preprocessing (crop, resize, colour normalisation, gain compensation), the network itself, any post-processing on detections, and the decision thresholds that convert a score into a pass, a reject, or a review flag. Reverting the weights while leaving the current preprocessing and current thresholds in place produces a configuration that was never validated as a whole — a third state, neither the old known-good one nor the new suspect one.
The rollback unit is the pinned version: weights plus preprocessing plus thresholds plus the calibration record they were validated against, promoted and reverted as one artefact. This is the single most consequential structural decision in a rollback design, and it is cheap to make early — a versioned bundle in a container image or model registry entry, with the threshold values stored beside the weights rather than in a line-side config file that operators edit independently.
Named tooling makes this concrete. Exporting the inspection graph to ONNX and building the TensorRT engine as part of the pinned bundle means the reverted version brings its own engine rather than being re-optimised at rollback time against whatever driver and library versions the station happens to be running. In our experience with industrial CV deployments, rollbacks that re-run a build step on the line are the ones that overrun their window — a revert that needs a compile is not a revert, it is a deployment.
| Component | Pinned with the version? | Why |
|---|---|---|
| Model weights | Yes | The obvious unit; useless alone |
| Preprocessing (crop, resize, normalisation) | Yes | Silent accuracy shifts if mismatched to weights |
| Decision thresholds / score cut-offs | Yes | Determines the false-reject rate the line feels |
| Compiled inference engine (e.g. TensorRT plan) | Yes | Avoids a build step inside the rollback window |
| Camera acquisition settings (exposure, gain, white balance) | Recorded, reverted deliberately | Shared with the line; may have changed for good reasons |
| Line PLC reject logic | No — but the interface contract is fixed | Rollback must not change what the PLC receives |
What authorises a rollback, and who executes it
A trigger that requires a meeting is not a trigger. The point of writing degradation triggers down in advance is that the person on shift at 03:00 does not have to reason about whether a regression is real; they have to check a condition and act.
Useful triggers come in pairs — one input-side, one output-side — because they fail differently. False-reject rate crossing a pre-agreed control limit for a defined number of consecutive subgroups is an output-side trigger. Score-distribution shift beyond a set divergence threshold against the validation baseline is another. An input-side trigger fires on the acquisition conditions themselves: illuminance or white-balance statistics leaving the range the pinned version was validated across. These signals are the same ones a monitoring harness already produces; the design work here is not inventing them but wiring them to an authorised action. Our companion piece on instrumenting drift signals on a production inspection line covers how those signals are computed and baselined.
Authority is a separate axis from detection. The rollback owner is a role on shift — typically the line quality engineer or the maintenance technician on the automated inspection station — not a data scientist reachable by email. Two properties matter: the owner can execute the revert without a second approval when a written trigger has fired, and the owner cannot silently change thresholds outside a rollback. Ownership boundaries of this kind are exactly what tends to be left unassigned at handover, which is why we treat who owns the inspection model after go-live as its own design question.
What the line inspects while the model is out
Rollback that leaves the station with no inspection capability is a stoppage with extra steps. The fallback inspection mode is the part of the design that bounds the cost of a regression: it defines, in advance, what throughput and what inspection coverage the line holds during the window.
Three options are common, and the choice is contextual rather than ranked:
- Previous pinned model version. Best when the regression came from a model change and the prior version’s production accuracy is documented. Fastest, and preserves full inspection coverage — provided the previous version is still valid for the current part mix and board or packaging revision.
- Rules-based or classical CV check. A tightened OpenCV-style geometric or intensity check that catches the gross defect classes only. Lower coverage, but stable and independent of the model — appropriate when the regression’s cause is unknown and both model versions are suspect.
- Manual inspection at reduced rate, or 100% downstream sampling. Highest coverage per unit inspected, lowest throughput. Appropriate for a safety-relevant or high-value defect class where escaping a defect costs more than slowing the line.
The important discipline is that whichever mode is chosen, its accepted throughput and coverage are agreed with operations before the rollback happens — not negotiated during it. Rollback bounds the exposure window of a regression; it does not remove the possibility that the line runs slower, or stops, while the fallback is in force.
Rehearsal, logging, and the measurement that matters
The measurable outcome of a rollback design is time-to-rollback: the elapsed time between a degradation trigger firing and the line running on a known-good inspection state. That number is only knowable by rehearsal. A rollback drill on a planned maintenance window — trigger fired deliberately, owner executes, timestamps recorded — converts an assumption into a figure the plant can plan around, and reliably exposes the small breakages that make an untested path slow: expired registry credentials, a station without disk headroom for the previous image, a threshold stored somewhere the bundle does not cover.
What to log is determined by a single requirement: the regression must be diagnosable afterwards, not merely reversed. That means the trigger that fired and its value, the version identifiers of both the outgoing and incoming bundles, the timestamps for trigger, decision, execution and verification, the fallback mode in force, and — critically — a retained sample of the frames the suspect version was judging when the trigger fired. Without those frames, the post-mortem has a symptom and no evidence, and the same regression comes back with the next release.
The rollback path and its rehearsal records are inspection reliability artefacts in their own right, which is why we bring them into the same review as the rest of a station’s computer vision engineering evidence, and why a reliability audit of a deployed model treats rollback rehearsal frequency as a checkpoint rather than a nicety. Teams that want that audit run against a live line can see how we scope that kind of hardening engagement.
Track three numbers alongside time-to-rollback: rehearsal frequency, the count of rollbacks executed without escalation, and the production-line accuracy delta measured before and after each revert. The third one is the honest check — if reverting does not move line accuracy, the model was not the problem.
When rollback is the wrong answer
Rollback reverses a model change. It does nothing about a process change. If the trigger fired because the packaging supplier changed substrate gloss, a new board revision entered the mix, or a lamp array was replaced with a different colour temperature, the previous model version faces the same unrepresented condition and will fail the same way — sometimes worse, because it was validated on an even narrower envelope. In that situation the correct actions are a fallback inspection mode plus retraining or recalibration, and possibly a controlled stoppage; rolling back only burns the window.
The other case where rollback is wrong is a safety-relevant escape. If the failure mode is undetected defects reaching a downstream assembly step, the decision is to stop or to escalate inspection coverage first and resolve the model second. Deciding which of these three responses applies — revert, retrain, stop — is a judgement the rollback design should pre-frame, not one it should replace. What is the shortest rollback you have actually timed on a running line, rather than estimated?
Frequently Asked Questions
What exactly gets rolled back — the model weights, the preprocessing, the thresholds, or all three as one pinned version?
All three, as one unit, together with the compiled inference engine and the calibration record they were validated against. Reverting weights while leaving current preprocessing or thresholds in place creates a configuration that was never validated as a whole. Camera acquisition settings are recorded with the version but reverted deliberately, since the line may have changed them for good reasons.
What degradation triggers should authorise a rollback, and who on shift is allowed to execute it?
Written triggers — false-reject rate crossing a control limit for a defined number of consecutive subgroups, score-distribution divergence beyond a set threshold, or acquisition statistics leaving the validated range — should authorise the revert without further approval. Execution authority belongs to a named role on shift, typically the line quality engineer or the inspection-station technician, not to whoever built the model.
What fallback inspection mode keeps the line moving while the model is out?
Three options, chosen in advance: the previous pinned model version (fastest, full coverage, valid only if the part mix has not changed), a rules-based classical CV check (gross defect classes only, but independent of the model), or manual inspection at reduced rate. The choice is contextual, and the accepted throughput and coverage for each must be agreed with operations before a rollback happens.
How do we test a rollback before we need it, and how often should it be rehearsed?
Fire a trigger deliberately during a planned maintenance window, have the named owner execute the revert, and record the timestamps — that produces a real time-to-rollback figure instead of an estimate. Rehearsals typically expose expired registry credentials, stations without disk headroom for the previous image, or thresholds stored outside the pinned bundle. Rehearsal frequency itself is worth tracking as a reliability metric.
When is rollback the wrong answer, and the line should stop or the model should be retrained instead?
When the trigger fired because of a process change rather than a model change — new substrate, new board revision, replaced lighting — the previous version faces the same unrepresented condition and will fail similarly. There the answer is a fallback mode plus retraining or recalibration. For a safety-relevant defect escape, the correct first move is to stop or raise inspection coverage, and resolve the model afterwards.
Building confidence in your rollback mechanism
Version pinning, shadow-mode validation, and automated health checks form the minimum viable safety net for any deployment. Everything else is detail.
