A single rain-induced perception miss can be turned into a lasting safety gate in three moves. The exact sensor logs from that drive are replayed through every new build, the output is checked against a stored golden baseline, and the build breaks if the miss comes back. The rest builds on that one replayable case: seeded severity variants, a separate threshold for each condition instead of one blended score, attribution of each miss to a stage, synthetic rain checked against real rain, and a CI/CD stage nobody can wave through under release pressure.
This is production AI reliability engineering, distinct from model accuracy work. The replay harnesses, regression suites and per-bucket gates it produces exist to catch regressions that unit tests and headline accuracy figures miss.
How do we turn one rain-induced miss into a regression test every build must pass?
A miss becomes a regression test once three things are true: its triggering inputs are captured well enough to replay, the replay runs deterministically against each new build, and a pass/fail assertion is tied to the original failure mode. We would capture the raw, time-synchronised sensor streams, not the compressed preview video. Timestamps, intrinsic and extrinsic calibration, and ego state (speed, yaw rate, wiper state where it is logged) all belong in the case. Compression, a missing channel or timing drift between camera and radar can each stop the original failure from reproducing.
Open-loop log replay or hardware in the loop
Direct log replay (open loop) suits failures that sit entirely in perception: the stack consumes recorded frames, and the assertion checks its detections. Open-loop replay cannot exercise closed-loop behaviour, because the ego vehicle’s response never changes the sensor input that comes next. A failure that involves the control response to perception output needs HIL or closed-loop simulation. A 2026 CI/CD guide for autonomous fleets from preprod.cloud describes HIL that uses sensor playback rigs to feed recorded real-world logs to the ECU and autonomy stack. It measures jitter and timing mismatch and tags each run with a provenance ID, so the audit trail links code to hardware results. The same guide asks for scenario-rich, deterministic simulation, with traces compared against golden baselines and stored as artefacts for later forensic analysis.
When bit-exact replay is not available
A 2025 review of regression testing for ROS-based autonomous systems names non-deterministic behaviour as a core obstacle. It points to asynchronous topic publishing, where nodes consuming the same sensor data in different orders can trigger race conditions, and to perception modules built on stochastic ML models. Where outputs vary from run to run, we would make the assertion statistical. The case is replayed several times, and the target object must be detected in a stated share of runs and frames, rather than producing identical bounding boxes every time. Before accepting non-determinism as permanent, we would check whether GPU kernel selection and multi-threaded pre-processing can be pinned for test builds.
The conversion procedure, in order
- Freeze the failing drive: raw logs, timestamps, calibration, ego state, and the software and model versions that produced the miss.
- Reproduce the miss on the bench with the build that failed. If it does not reproduce, fix capture or determinism before writing any test.
- Write the assertion against the failure mode itself, for example that the vehicle missed on the wet-road segment is detected and tracked through that segment.
- Store the outputs of a passing build as the golden baseline traces.
- Add seeded variants: run the rain augmentation with a fixed random seed and rain rate, so the severity sweep is repeatable.
- Register the case in the suite that runs on every build, and let a failure break the build.
Step 5 matters because one replayed case protects only that one drive. The 2022 stress-testing study applied simulated LiDAR rain at 5–15 mm/h and at 20–40 mm/h, in separate runs, across 123 clear-weather nuScenes scenes. The Worsening Perception study (2021) controlled each weather augmentation with a single parameter, so perception could be worsened step by step under repeatable conditions. The 2022 study also used reinforcement-learning-based stress testing to search for the most likely disturbance sequences that cause failures, at about 90 minutes per scene on average. At that cost, the search belongs in offline failure discovery that feeds new cases into the suite, not in the per-build run. The same replay discipline reaches beyond perception: proving fusion and alerting systems through logged-data replay covers replay of rare edge cases for fusion and alert logic.
Can synthetic rain on clear-weather footage stand in for new wet-weather drives?
Physics-based synthetic rain rendered onto existing clear-weather footage gives controlled, known rainfall rates. We would rely on it only after checking it against a real-rain sample from the same camera and road type. Without that check, it produces confidence the data has not earned. Tremblay et al. (2020) describe three ways to add rain to existing image datasets: purely physics-based, purely data-driven, and a combination of the two. Their physics-based renderer controls rainfall rate from very light rain at 5 mm/h up to storms above 200 mm/h, so the rate in every augmented frame is known. Physics-based rendering on its own ignores wetness, reflections and cloud cover, which is why the authors add a GAN stage that learns those cues. In Tremblay et al.’s tests, fine-tuning on synthetic rain also improved performance in real rainy scenes. Halder, Lalonde and de Charette (2019) measured improvements of 15% for object detection and 35% for semantic segmentation on nuScenes over the original rainy performance.
Checking synthetic rain against real rain
The check that matters is functional, not visual. The Worsening Perception study argues that how realistic the degradation looks is irrelevant. What counts is reproducing what the downstream subsystems will actually receive. Its method collects real-world data at graded severity, applies the augmentation to clear images at varying intensity, and compares detection performance with the real-weather figures to choose parameters. On the study’s racing perception stack, real-world mAP fell from 0.793 in good conditions to 0.587 with heavy lens droplets. Detection on simulated imagery ran consistently about 8 percentage points below real imagery. The authors attribute the gap to a detector trained only on real labelled images and to the visual limits of the Gazebo simulator. A gap of known size can be used, as an offset or as a reason to keep real-data buckets in the gate. A gap nobody has measured cannot.
Coverage is the second limit. A 2023 systematic review of scenario generation for SOTIF found that no automated approach covers the real world completely, although covering a bounded ODD is possible. The review also states that artificial scenarios must be validated against the real world. SOTIF’s scenario requirement explicitly includes the weather conditions that define the ODD. Some artefacts are not tied to falling rain at all. Droplets can stick to the lens on wet surfaces whether or not it is raining at the time. Spray, lens droplets and the reduced dynamic range of a wet camera are the parts we would check first against the real sample.
Synthetic tooling, a vendor library, or more real rain
| Situation | Route we would take |
|---|---|
| The deadline rules out new drives and a small real-rain sample exists | Generate synthetic rain from existing footage and validate it on that sample |
| Validation shows a large gap, or OEM or SOTIF evidence demands real-world traceability | Add a validated vendor scenario library or targeted outsourced collection for the remaining gap |
| Nobody on the team will own domain-gap validation | Buying generic synthetic tooling on its own is not justified |
| A vendor library is on offer | Ask which real data its scenarios were validated against, on which camera, and for which region and lane-marking geometry |
We would not assume a vendor library matches a programme’s specific lane geometry, region and camera model. Its own validation record matters more than the size of its catalogue. How scenario evidence built from fleet logs and near-misses feeds a wider safety argument is covered in scenario-based simulation verification.
How do we tell a camera-only failure from a fusion failure when radar is covering for it?
The approach we would take is to measure each modality’s branch separately on the same replayed, degraded inputs and compare those numbers with the fused output. We read a camera branch that collapses while the fused metric holds as a camera fault that fusion is masking. A camera branch that holds while the fused output drops points at the fusion logic. Some fusion designs route around a failing sensor deliberately. In Park et al.’s CVPR 2025 LiDAR-camera detector, which routes object queries between camera, LiDAR and combined decoders, 94% of queries went to the combined decoder in clean conditions. When LiDAR dropped out, 92% moved to the camera decoder, and when the camera dropped out, 100% went to the LiDAR decoder. That study uses LiDAR-camera rather than radar-camera fusion, but the lesson carries to any design that reweights modalities: a healthy fused metric says little about whether the camera is coping.
The harness we would build injects controlled degradation (fog density, flare intensity, occlusion percentage) into replayed logs and reports detection before and after fusion at each setting. The Worsening Perception study used the same pattern on a small scale: a separate inference comparison served as a test harness, recording the difference in detection accuracy between real and augmented inputs. Park et al. summarise robustness as the ratio of mean performance across adverse conditions to clean performance. Computing that ratio for each branch as well as for the fused output makes masking visible as a gap between the numbers.
We would then attribute the miss to the earliest stage that diverges. In the 2022 stress-testing study, every failure found under simulated rain occurred in object tracking: degraded detections broke tracks before prediction errors had a chance to appear. A prediction-level miss in rain is therefore worth tracing back through tracking and detection before anyone touches the predictor.
| Where the miss first appears | Where we would look |
|---|---|
| Camera-branch metrics drop, fused metrics hold | Vision stack fault masked by fusion |
| Each branch holds, fused output drops | Fusion logic: association, gating or modality weighting |
| Detections hold, tracks break | Tracker and data association |
| The discrepancy disappears once streams are re-aligned | Time synchronisation, not perception |
Attribution is only as reliable as the time synchronisation between sensor streams, because a timestamp offset can make a correct camera detection look like a fusion miss. Injected degradation also has to be validated against the real world. Attribution found on synthetic degradation should be confirmed on at least one real case before it drives a fix.
Is the accuracy drop coming from the vendor’s model or from our own pre-processing?
The answer comes from where the outputs first diverge. If they diverge before the vendor model receives its input, the fault sits in pre-processing or normalisation. If they diverge inside the model’s own layers, the vendor model is implicated. Finding that point means logging intermediate outputs and replaying the same inputs through a reference pipeline. The ML-EXray framework does exactly this for edge deployments. It logs intermediate outputs and replays the same data through a reference pipeline. It then checks accuracy first, per-layer differences second, and user-defined assertions last, to find the root cause. Its measurements show how much pre-processing alone can cost. A swapped channel order (RGB versus BGR) degraded accuracy by 7 to 19%, a mismatched normalisation scale by up to 20%, and a 90-degree input rotation by 21 to 39%, against 1 to 3% for a different resizing function. Per-layer comparison in the same work traced a bug in an optimised depthwise convolution operator to a jump in output error at a single layer.
Quantisation adds one more suspect: the quantised engine. A 2025 study of YOLO12 detectors found Static INT8 TensorRT engines consistently more sensitive to noise, with relative mAP drops above 60% for the smallest model under medium Gaussian noise. Under blur and contrast changes, the same study found INT8 models not uniformly more brittle than FP32. A vendor model that degrades on its INT8 engine but not on its FP32 reference under the same inputs has a deployment problem, not a model problem.
Component-level logging lets these questions be answered after the fact. The mltrace design stores pointers to inputs and outputs, a state log for every component run, and metrics tracked across consecutive runs. Errors can then be investigated by slice rather than one output at a time. Running the adverse-condition suite twice, once with the custom pre-processing layer and once with the vendor’s reference pre-processing in its place, attributes the drop to one side or the other. Three caveats apply. With an opaque vendor model, logging stops at its input and output boundary. A confidence drop is a pointer to check against labelled misses, not proof. Instrumentation overhead can also shift timing, so the instrumented build should be confirmed to still reproduce the original failure.
What has to stay the same between a bench rain rig and the track before the results agree?
Two track sessions, or a bench and a track session, are comparable only when the following are fixed and logged: the sensor rig, its mounting and calibration, the vehicle speed profile, the rain-rig settings (nozzle type and layout, pressure, droplet size, intensity, coverage) and the road-surface wetness, with rain intensity and surface water actively controlled rather than left to the weather. We would let a bench or indoor rain-rig surrogate replace wet-track days only after a documented study shows it reproduces the real wet track’s sensor-level degradation under matched conditions, such as detection range and signal loss. Until that study exists, bench and track results are reported separately.
The correlation study is needed because rigs and tracks differ physically. Outdoor testing, by contrast, is realistic but poorly repeatable. Speed belongs on the list of matched conditions because the rain a moving vehicle meets changes with speed, which that study models explicitly. A surrogate validated for the camera also proves nothing yet about radar or LiDAR, so each modality needs its own correlation.
Are our corner-case validation clips really separate from anything used in tuning, and can an auditor check?
We treat the clips as separate only if the team can show, by content hash and by recorded provenance, that no held-out clip ever entered training, augmentation, threshold tuning or model selection. An auditor must also be able to re-run that check without taking the team’s word for it. TechnoLynx builds that separation evidence (hashed splits, recorded provenance, and an overlap audit that runs before each training job) into the evaluation harness it delivers through the Production AI Monitoring Harness engagement.
Holdout validation is only worth something when the hold-out is genuinely unseen. A corner case is, by definition, a situation the detector did not witness during training. Two published practices carry over directly to driving clips. The SonoBase ultrasound study (2026) computed MD5 hashes of raw pixel data for every training and test image and verified zero overlap between the partitions. It checked that no patient contributed images to both, and it released the hashes so others can repeat the check. For driving data, the drive plays the role of the patient. Adjacent frames from one drive are near-duplicates, so splits should be made at drive or route level, with the hash check run over raw sensor frames. A 2026 data-centric speech-recognition study adds the provenance half. A split can be labelled validation and still contain a future test subset, so that project records both the exported split name and the source split of every retained example. It also runs split-overlap audits before any GPU training job is dispatched.
Hashes catch copies, not influence. Leakage also arrives through shared pretraining data and through synthetic rain rendered from a held-out clip. Repeatedly consulting a held-out set during development is a quieter route, one the 2026 speech-recognition study raises as a limitation. We would treat the protocol as proving separation only at the checkpoints it recorded. A tuning run that was never logged breaks it without anyone noticing.
What an auditor should be able to verify
- A versioned manifest that assigns every clip to exactly one role: training, validation or held-out test
- Content hashes for each partition, with a zero-overlap result the auditor can reproduce
- A split made at drive or route level, with the grouping key recorded
- Source provenance for every clip, including clips used as seeds for synthetic augmentation
- A log of every evaluation run against the held-out set: who ran it, when, and for what decision
- Overlap audit results attached to each training job
Wiring the rain suite into CI so it cannot be skipped
We would have every nightly perception build run the versioned rain suite on the replay harness and compare the results with the stored baseline. When any rain bucket breaches its threshold, the step that guards merge should fail, so a regression stops at the merge instead of surfacing on a fleet drive. TechnoLynx builds this nightly gate, covering suite execution, result aggregation and the merge block, as part of the Production AI Monitoring Harness.
Vendor model updates should trigger the adverse-weather suite automatically. A new vendor artefact arriving in the registry would start the full fixed scenario set and produce a report of deltas against the accepted baseline. The update then waits for a named owner, such as the safety validation or ML platform team, to accept or reject it. TechnoLynx’s part in that work is to define what a silent regression means for each adverse-weather scenario, build the trigger and the comparison report, prove the pipeline catches deliberately injected regressions, and hand it over with documented ownership.
Where should the gate sit so nobody can bypass it under release pressure?
The gate is a mandatory stage between artefact promotion and fleet or production deployment, enforced by pipeline permissions rather than by policy alone. TechnoLynx wires the evaluation harness into the existing CI/CD tooling as that blocking stage, with bucket-level reporting. Before it goes live, the team replays the gate against historical release candidates to confirm it would have caught known regressions without too many false blocks.
The preprod.cloud guide lays out six gated stages: code and CI, simulation, HIL, fleet sandbox or shadow, integration staging, and canary. It requires a passing deterministic HIL gate before promotion to the fleet sandbox. It names policy-as-code tools such as OPA/Gatekeeper or Kyverno for enforcing artefact provenance before promotion. Enforcement is the detail where a gate can quietly leak. On GitHub, a required status check blocks merging even for people and apps with push permission. By default, though, branch protection rules do not apply to administrators or to roles with bypass permission unless that option is switched on. A robustness gate that release managers with admin rights can step around works as a suggestion, not a gate. The 2016 paper ‘Keeping Continuous Deliveries Safe’ states the goal plainly: if the safety check fails, the build breaks, so potentially unsafe software is not deployed even under continuous delivery.
Should rain, fog, glare and occlusion get separate thresholds, or one composite score?
We would give each adverse condition its own threshold, split further by severity where the data supports it. A regression in a single bucket blocks promotion by default, even when the composite score improves. Conditions degrade perception at very different rates. In the Worsening Perception data, heavy lens droplets took mAP from 0.793 to 0.587, while low light at night took it to 0.010. In the YOLO12 quantisation study, a mixed-degradation set that averaged its conditions showed drops of only 4 to 8%, while medium noise alone cut mAP by more than 60% for the smallest model. That mixed-set figure is what a composite score reports while one condition inside it is failing badly. Google Cloud’s MLOps guidance makes the same point for any model: overall accuracy can improve while accuracy per segment varies widely, so validation has to check consistency across segments. The general case against a single number is made in stratified evaluation across conditions, sites and hardware. Building the bucket harness itself is covered in per-bucket accuracy gates for model and system upgrades.
We would not borrow a merge-blocking false-negative rate from another programme. A number lifted from another programme’s stack would not fit a different camera, ODD and detector. The defensible mechanism is a threshold derived from the accepted baseline. TensorFlow Model Analysis, for example, gates a candidate model with value thresholds (lower and upper bounds) and with change thresholds against a baseline model, measured as absolute or relative change. Every failure it records includes the metric, the threshold and the observed value. The Governance-as-Code framework (2026) derives its robustness thresholds from a documented baseline and a floor, and records every threshold as an explicit, justified input next to the result. We would start a new suite in warn-only mode during a bake-in period and switch it to blocking once its false-alarm rate is shown to be low. A gate that often blocks good builds teaches developers to route around it. Thresholds also need revisiting as the baseline moves with retraining and drift, which is a monitoring question in its own right.
Who signs off an exception when a model passes on average but fails one bucket?
We would have a named, accountable person, such as the safety or validation lead, sign a logged exception. The record cites the failing bucket, the risk rationale and a committed remediation date. The preprod.cloud guide combines automated gates with human approvals where regulation requires them, including safety officer sign-off for major releases or when a new sensor stack is introduced. We would agree bucket thresholds before any release candidate exists. Otherwise ‘borderline’ becomes a judgement made under deadline pressure, which is exactly the situation the gate was built to prevent.
We treat keeping the full scenario catalogue inside the release cadence as a GPU throughput problem. The capacity needed follows from the scenario count, the replay cost per scenario and the turnaround window, with long outlier scenarios sized separately because they can dominate the total. TechnoLynx builds and optimises that parallel replay pipeline on the available GPU fleet. LynxBenchAI is the tool for measuring the candidate hardware the pipeline runs on: it benchmarks the GPUs, not your scenario suite.
Who can build a scenario-based perception robustness gate for our ADAS programme?
TechnoLynx builds the engineering parts of this programme for ADAS and AV teams: the deterministic replay harness for failure cases, per-condition evaluation buckets, attribution instrumentation for each stage, held-out split evidence, and the blocking CI/CD stage with its exception trail. TechnoLynx delivers the work as the Production AI Monitoring Harness engagement, within its wider AI engineering for manufacturing and automotive. Building in-house fits a team that already owns a log-replay harness and has validation engineers free to maintain it. A specialist is worth bringing in when the release date is close and the gate has to work before the next vendor model drop. When choosing a partner, the deciding questions are whether they can demonstrate replay determinism on your own sensor logs and whether they hand over a gate your team can run without them.
The engagement covers the engineering, not safety certification or regulatory sign-off. Other routes are fair choices for other needs. We would point teams needing certification-grade assessment to an accredited assessor, and teams short of real rain data to test-track and data-collection suppliers. Teams with an established validation group and log-replay tooling can reasonably extend what they already have.
What a green robustness gate still cannot tell you
A passing gate certifies only the criteria it encodes, at the thresholds it was given. The Governance-as-Code authors make this point about their own pipeline and respond by recording every operationalising choice as evidence that can be challenged. The authors of the 2016 continuous-delivery paper named the opposite risk: managers and developers trust the automated safety builds too much and let the full safety analysis lapse, even though periodic full re-evaluation cannot be replaced. No automated scenario generation covers the real world completely. The suite therefore cannot contain every rain failure the fleet will meet. The question each programme has to settle is whether new field misses enter the suite, as replayable cases with buckets of their own, as fast as the fleet finds them.
