Pilot lighting is a fixed condition. Line lighting is a slowly moving one, and that difference is why an inspection model that cleared its acceptance threshold in a validation cell starts rejecting good parts four months later without anyone changing a line of code. Lamps lose output across their service life. A replacement fixture arrives with a different colour temperature than the one it replaced. Shift changes alter the ambient contribution through skylights and bay doors. Enclosure glass accumulates dust and a film of coolant mist. None of that shows up on a maintenance ticket, and none of it is visible to an operator walking the line.
The failure is not that lighting changes. It is that nobody is measuring it. When the only instrumented signal is the model’s output, a lighting shift and a model regression look identical — and a team that cannot tell them apart will retrain a model that was never the problem.
What does lighting drift mean in practice for a CV inspection model?
A defect-detection model learns a mapping from pixel statistics to a verdict. Illumination is baked into those pixel statistics: contrast between a solder fillet and its pad, the specular highlight that separates a scratch from a printed line, the absolute brightness that determines whether a low-contrast defect sits above or below the sensor’s noise floor. Change the illuminant and you change the input distribution the model was fitted on, without touching the model at all.
Two properties make this a distinct failure class rather than a variant of ordinary drift. First, it is monotonic over long horizons — lamp output falls gradually, so the degradation is a slope rather than a step, and slopes hide well inside weekly accuracy averages. Second, it is periodic over short horizons — the same station can present measurably different images on day shift and night shift because of ambient contribution, which means a single daily accuracy figure averages away the very signal you need.
An instrumented pipeline flags an illumination shift as a drift signal days before defect-escape rates move; an uninstrumented one learns about it from a scrap batch or a customer complaint, and cannot then attribute the failure to the model, the camera or the lamp. That attribution gap is the expensive part. Retraining on drifted images will paper over a maintenance problem and quietly re-baseline the line to a degraded fixture.
What to log: the illumination telemetry channel
The signals worth logging are cheap, come from the image or the acquisition layer, and require no labels. This is the operational point: label-dependent monitoring arrives days late, while input-side telemetry is available on the frame.
| Signal | Source | What it catches | Sampling |
|---|---|---|---|
| Exposure time / gain (if auto) | Camera SDK — Basler pylon, Spinnaker, GenICam | Slow lamp decay; the camera compensating silently | Every frame, aggregated per shift |
| White-balance coefficients | Camera or ISP | Replacement fixture with a different colour temperature | Per shift |
| Reference-target patch readings | Fixed grey/colour patch inside the field of view | Absolute luminance and colour shift, independent of part mix | Every frame where the patch is visible |
| Histogram statistics (mean, p5, p95, clipping fraction) | OpenCV on the acquired frame | Contrast collapse; highlight clipping from a new specular path | Every frame, percentiles per shift |
| Illuminance at the station | Fixed lux sensor or metadata from the light controller | Ambient contribution through skylights and bay doors | Continuous |
| Focus / sharpness proxy | Variance of Laplacian on a fixed ROI | Separates optics and dust-on-glass from illumination | Per shift |
The reference target does most of the work. A part-independent patch fixed in the field of view — a matte grey chip on the fixture, outside the inspection ROI but inside the frame — gives you a measurement that does not move when the product mix moves. Without it, every image statistic is confounded by what happens to be on the conveyor.
We treat the patch reading, not the accuracy dashboard, as the primary illumination channel. It is the only one of these signals that stays interpretable when the line switches from a dark substrate to a light one on the next order.
What belongs in a commissioning baseline
Drift is a distance from something. If commissioning did not record that something, the first alert has no threshold to compare against and the team ends up negotiating from memory about “how it looked at install”.
A usable baseline, captured at commissioning and re-captured after any authorised fixture change:
- Lamp make, model, drive current, install date, and expected service hours per fixture — drift thresholds are meaningless without a lamp-life clock to hang them on.
- Reference-patch luminance and colour readings, sampled across a full 24-hour cycle so the ambient envelope is recorded, not just one convenient afternoon.
- Exposure and gain settings the model was validated under, pinned alongside the model version.
- Histogram percentiles for a fixed set of golden-sample parts, per shift.
- Photographs of the enclosure and fixture geometry, so a later “nothing changed” claim is checkable.
- The cleaning interval for enclosure glass and its last completed date.
The pinning matters as much as the measurement. Preprocessing and calibration travel with the model version, which is also the precondition for a clean revert — the mechanics of that are covered in our rollback design for production CV inspection models.
Setting thresholds that fire before the defect rate moves
Two-band thresholds work better than a single alarm line, because illumination drift is slow and the correct response differs by magnitude.
- Advisory band — reference-patch luminance or exposure has moved beyond the commissioning band’s normal shift-to-shift variation, but the inspection ROI histogram is still inside its validated envelope. Response: raise a maintenance inspection, log it, keep running.
- Action band — the ROI histogram percentiles or clipping fraction have left the validated envelope, or the patch has moved past the point where the model’s score distribution measurably shifts. Response: authorise a maintenance action, and hold the model version until the illumination is restored or the baseline is formally re-cut.
Derive both bands from the line’s own observed variation across a commissioning period that includes every shift pattern, not from a round number. A threshold set from a single day’s data will fire every night shift and be muted within a week. The threshold and re-baselining discipline itself is not specific to lighting; we apply the same structure across reliability audits, and it is where the illumination bands should be reviewed rather than invented per station.
Maintenance action, re-baseline, or retrain?
The choice the telemetry exists to make, and getting it wrong is how a lighting problem becomes a permanently degraded model.
| Evidence pattern | Correct response |
|---|---|
| Patch luminance falling monotonically, colour stable, fixture near end of rated hours | Maintenance: replace lamp, restore to baseline. No model change. |
| Step change in white balance coincident with a fixture replacement | Maintenance: match colour temperature to spec, or re-commission the baseline and re-validate the model against the new illuminant. |
| Patch stable, ROI contrast falling | Not illumination — check enclosure glass, optics, focus, coolant film. |
| Patch stable and ROI stable, score distribution shifted | Not illumination — process, part-mix or material change; hand to process engineering. |
| Illumination restored to baseline, accuracy still below pilot | Genuine model issue; now retraining is the right instrument. |
| Line has permanently adopted a different, acceptable illuminant | Re-baseline and re-validate. Retrain only if re-validation fails. |
Retraining is the last row, not the first response. A model retrained on images from a decaying lamp encodes the decay, and the next lamp replacement then reads as a regression.n.
Measuring whether the telemetry paid for itself
Two numbers carry the argument. The first is time-to-detect on illumination drift, expressed in shifts rather than weeks. The second is the production-line accuracy delta against pilot accuracy, tracked across lamp life and shift patterns rather than as one figure. Alongside them, count re-baselining events that were resolved by a maintenance action instead of a model retrain — that ratio tells you whether attribution is working. Each drift event caught by telemetry rather than by scrap is an avoided rework batch and an avoided unplanned rollback of the inspection model.
Lighting drift is the narrow input-side channel; the broader question of which input, output and label signals to watch together is developed in monitoring model drift on a production CV inspection line, and the conditions a pilot systematically fails to represent are the subject of why CV defect-detection pilots fail on the production line. The instrumentation work itself sits inside our computer vision engineering practice, and the monitoring harness it feeds is scoped as part of a validation engagement under services.
The uncertainty we have not resolved cleanly is how far a reference patch can be trusted on stations where the specular path to the part differs sharply from the path to the patch — polished metal and glossy substrates in particular. On those lines the patch still catches lamp decay reliably, but it under-reads changes in the highlight structure the model actually uses. If your inspection depends on specular signal, the patch is a floor, not a full baseline. What would you have to add to the field of view to make that highlight structure measurable per shift?
Frequently Asked Questions
What does lighting drift on a production line mean in practice for a CV inspection model?
Fluorescent bulbs age unevenly across a factory floor, shifting the spectral signature enough to degrade vision models trained under different lighting. It means the input distribution the model was fitted on is moving while the model stays fixed. Lamp output decays over service hours, replacement fixtures differ in colour temperature, ambient light varies by shift, and enclosure glass accumulates dust and coolant film — each changing the contrast and brightness statistics the model’s decisions depend on. The model has not degraded; the images have.
Which image-level signals should be logged as drift telemetry?
Exposure time and gain, white-balance coefficients, histogram statistics (mean, p5, p95, clipping fraction) on a fixed ROI, and readings from a part-independent reference patch fixed in the field of view. Add station illuminance and a sharpness proxy so optics and dust can be separated from illumination. All of these are label-free and available per frame, which is why they detect drift days before label-dependent accuracy metrics do.
How do you set drift thresholds that trigger before false-reject or defect-escape rates move?
Use two bands derived from the line’s own commissioning variation across every shift pattern: an advisory band when the reference patch or exposure leaves normal shift-to-shift variation but the inspection ROI is still inside its validated envelope, and an action band when the ROI histogram or clipping fraction leaves that envelope. Thresholds set from a single day’s data fire on every night shift and get muted within a week.
When is a lighting shift a maintenance action rather than a case for retraining?
Monotonic luminance decay with stable colour on a fixture near its rated hours is a maintenance action — replace the lamp and restore the baseline, with no model change. Retraining is correct only after illumination has been restored to baseline and accuracy is still short of pilot, or after a re-validation against a deliberately adopted new illuminant fails. Retraining on drifted images encodes the drift and makes the next lamp replacement read as a regression.
How do you separate illumination drift from camera, optics or genuine process change during triage?
The reference patch is the discriminator. If the patch has moved, the illuminant has moved. If the patch is stable but ROI contrast is falling, look at enclosure glass, focus and optics. If both patch and ROI statistics are stable and only the model’s score distribution has shifted, the cause is process, part mix or material — not the imaging chain.
What belongs in a commissioning lighting baseline?
Fixture make, model, drive current, install date and expected service hours; reference-patch luminance and colour sampled across a full 24-hour cycle; the exposure and gain settings the model was validated under, pinned to the model version; per-shift histogram percentiles for golden-sample parts; photographs of enclosure and fixture geometry; and the glass-cleaning interval with its last completed date.
Monitoring and mitigation strategies
Continuous luminance logging paired with scheduled recalibration windows will catch most drift before false-reject rates climb. Everything else is detail.
