You prove a sensor-fusion fix or an alerting-model update on logged data by replaying the recordings that exercised the fault, comparing the new outputs with the old ones, and counting a difference as a fault only when it goes past a stated noise threshold. That turns replay into regression evidence rather than a demo. Four things should be written down before the run: the timing tolerance that makes a replay the same scenario, the sampling that puts rare dangerous conditions in front of the model, the missed-detection bound the release has to meet, and the format the result is stored in so the next model version faces the same test.
Production AI reliability is its own engineering discipline, separate from model quality. Its deliverables are the regression suites, task-specific eval harnesses and validation packages that catch perception failures on rare classes and missed alerts before customers do, and replay is one of its main tools. In a 2021 study of autonomous driving testing, two interviewed practitioners described replaying sensor data and control commands from road testing to check a new release, and 55% of survey participants agreed.
Replaying the exact merge log that produced a ghost object
Yes, you can replay the exact log that triggered a ghost object in a high-speed merge and compare the patched outputs with the old ones. The replay only counts as a re-test, though, if it reproduces the original sensor timing and kinematics within a defined tolerance band, and if jitter variants around that log pass as well. Without the band you cannot tell whether you re-tested the failure or tested a slightly different scenario that happens to pass.
The log itself has to be good enough to replay. Each invocation can be logged with the classifier or configuration version, the synchronised input trace needed for later adjudication, and the operating-mode tags. VehicleMapping’s guide to production fusion asks for structured logs that are sufficient for deterministic replay in CI. The system also needs the right starting state: the STRaP test-reduction work (ESEC/FSE 2022) replayed the one second before each segment so that the modules were initialised and their states restored before the segment under test began. A fusion tracker that starts cold on the first frame of the merge is not running the scenario that failed.
Worked example: what a timing tolerance means at 120 km/h
VehicleMapping’s guide holds production fusion to at most 1 ms end-to-end temporal skew before any geometric stage runs, and to at most 0.05 m registration RMSE. The guide also notes that the displacement a clock error causes scales with speed, so a skew that does no harm in a car park can rule out a motorway result. Assume a 120 km/h closing speed (33.3 m/s) and treat displacement as closing speed multiplied by timing error.
| Timing source or gate | Stated accuracy | Apparent displacement at merge speed |
|---|---|---|
| NTP synchronisation (VP-AutoTest, 2025) | 10 ms | 33.3 × 0.010 ≈ 0.33 m |
| Production skew gate (VehicleMapping) | 1 ms | 33.3 × 0.001 ≈ 0.033 m |
| PTP synchronisation (VP-AutoTest, 2025) | 50 ns | 33.3 × 0.00000005 ≈ 0.0000017 m |
The same guide says residual registration error above 0.05 m breaches the 0.1 m lateral-error budget the guide’s centerline-generation stages must hold. On that arithmetic, NTP-grade timestamps carry about 0.33 m of possible displacement at merge speed, more than three times that mapping budget, used here as a yardstick. We would not treat a replay built on it as reproducing the original geometry. We would state the band in timing terms and check it against the logged clock source before the replay runs.
Separating real regressions from replay noise
Many autonomous-driving modules use non-deterministic algorithms, so the same input can give slightly different outputs on repeated runs. STRaP dealt with this by smoothing outputs with a sliding window and treating a difference as a fault only when more than 10% of frames mismatched, because its benchmark found inconsistency rates of up to 10% when the same recording was replicated. That threshold has a cost. The authors named short-lived wrong traffic-light bounding boxes, inside the 10% smoothing threshold, as a candidate cause of six missed faults.
For the defect under test, we would not lean on a frame-mismatch rate alone. We would add an object-level assertion (“no confirmed track exists at the ghost’s position during the merge window”) and keep the 10%-style threshold for everything else in the log.
Jitter variants around the original scenario
The exact replay tests the fix against one timing pattern only. Near-miss regressions live in the timing patterns next to it, so the replay set should include variants with injected timestamp offsets. A 2025 study of on-vehicle sensor staleness built its variants from staleness profiles measured in real vehicle logs, with a maximum jitter of 0.1 s, and validated against a set where camera data was always 100 ms stale. The same study recommends consuming stale data below a threshold such as 150 ms and dropping the sensor out above it. A variant sweep should therefore cover the measured jitter range and step past the dropout threshold, so the fallback path gets exercised as well as the fused path. Our page on deriving and validating a defensible jitter budget covers how to set that range for your own stack.
| Exact replay | Jitter variants | Verdict |
|---|---|---|
| Pass within tolerance | All pass | Fix validated for this scenario family |
| Pass within tolerance | Ghost recurs in some variants | Fix incomplete; reopen the defect |
| Fails, or timing outside the band | Not meaningful | Not a valid re-test; fix the log or the harness first |
Two limits apply. Choosing the band is a judgement call: too loose and a different scenario passes as equivalent, too tight and natural log jitter makes replay infeasible. A pass also covers the tested scenario family only, not every closing speed or merge geometry.
Risk-based sampling for review: putting night storms and feed dropouts in the replay set
To make a validation set stress-test rare dangerous conditions, stratify the replay archive by risk and deliberately oversample the strata that past incidents and errors flag as failure-prone: low-visibility night footage, feed dropouts, overlapping alerts. A 2024 Nature Communications paper on the “curse of rarity” describes safety-critical events as so rare that most available data contains very little information about them, with that information buried under large amounts of normal data. The 2021 testing study gives a highway example of skipping redundant replay: when the ego vehicle holds one lane at constant speed, only the unique moments in the recording, such as a cut-in, need replaying.
LynxBenchAI measures throughput, latency and cost per unit of work across accelerators, which gives a hardware baseline for running a stratified replay set; the first step is to benchmark a sample of your own logs on the candidate hardware.
One published method, STRaP, cuts replay volume while keeping test effectiveness. STRaP encodes each frame against a driving-scene schema, slices recordings into segments, removes redundant segments and orders the rest by scene coverage and rarity. In its experiments, recordings shrank by 34% to 77% while fault coverage averaged 98.8% of the faults the full recordings detected.
Oversampling also affects how results may be reported. A paper on operational test policies notes that its model assumes uniform testing across the operating domain, and that testing favouring some scenarios may draw incorrect conclusions. The fix is to report results per stratum, with each stratum’s own bound, and never to fold an oversampled set into one fleet-wide miss rate.
Worked example: how many true events a stratum needs
Ivrissimtzis et al. (2024) list zero-failure sample counts consistent with the standard success-run formula N = ln(1 − c) / ln(1 − p) for reliability (1 − p) at confidence c. For c = 0.95 and 1 − p = 0.95 this gives ln(0.05) / ln(0.95) = −2.996 / −0.0513 ≈ 58.4.
| Confidence | Reliability (share of true events caught) | Positive samples, zero misses |
|---|---|---|
| 95% | 95% | 58.4, rounded to 60 |
| 95% | 99.5% | 597.6, rounded to 600 |
| 95% | 99.8% | 1496.3, rounded to 1500 |
That is why we would oversample rare strata rather than wait for them to turn up in a uniform draw. A single miss resets the arithmetic: a 2026 analysis of operational-data fidelity shows that once a monitor fails after reaching 95% confidence, more failure-free runs are needed to get back there, and how many depends on prior knowledge and on whether the failure type is known.
When the regulator has not specified a bound, propose one grounded in event base rates and get sign-off before using it as a pass/fail criterion. Three caveats keep the bound honest. It is only as good as the ground-truth labels, and a zero-failure test shows up clerical labelling errors that averaged metrics would smooth over. A fusion model that must tell a real near-miss from routine activity, such as a child walking past a bus door, can only be validated against labelled incidents; without them, validation measures correlation with ordinary motion, not risk. Statistical results of this kind apply only between significant software updates or changes in the operating environment. And oversampling known strata does nothing for failure modes nobody has named yet: practitioners in the 2021 study still found unexpected corner cases on the road, such as lane lines that were hard to tell apart at sunset. The strata list therefore needs revisiting as field logs bring in new failures.
Should the alerting model favour recall, even with more false alarms?
For a life-safety alert-insertion system, we would say yes. A missed severe-weather alert is a worse failure than a false alarm, so the operating point should sit at the high recall, low precision end of the precision-recall curve, with a precision floor that keeps false alarms bounded. Zero-failure testing makes this concrete. Ivrissimtzis et al. (2024) set the operating point at the lowest threshold that still classifies every positive sample correctly, then report the true-negative rate as the measure of what that choice costs in false alarms. That pattern is for asymmetric settings where the costs of the two error types cannot be compared, and we would place life-safety alerting there.
LynxBenchAI measures throughput and latency across accelerators, giving a hardware baseline for the alerting model at its chosen operating point; start by benchmarking that model on the candidate hardware.
What false-negative rate is tolerable is a policy decision, which we leave to the regulator. The engineering job is to make the decision explicit. In the age-verification case Ivrissimtzis et al. describe, the regulator endorses a specific zero-failure test, and that test then sets the operating point. For an alerting system, we would seek the equivalent: a stated bound agreed with the regulator and your legal team. Until that exists, the per-stratum bound from the sampling plan serves as an interim target, recorded as provisional and not as a certified compliance point.
False alarms still need watching. Alert fatigue is a production failure mode in its own right, so we would put the true-negative rate in every release report. A threshold tuned on archive data can also drift once live, because operational data is often non-stationary through software updates and changing conditions.
Structuring replay results so they survive the next model update and a certification review
We would treat a replay protocol as certification evidence only if it is a documented, versioned procedure. It fixes the archive scope, the metrics, the thresholds and the evidence format once, then runs unchanged against every retrain or update, so each run produces comparable evidence instead of a one-off report. TechnoLynx helps teams turn replay testing into that kind of protocol and evidence pack through its AI Governance and Trust work, starting from the archive, metrics and acceptance criteria the team already has.
The need to repeat validation after changes is documented. A 2020 survey of validation frameworks for self-driving vehicles states that the vehicle should be validated again whenever its hardware or software changes, and asks whether a smart subset of tests could stand in for the whole process. The 2026 operational-data analysis describes re-running conservative reliability assessments incrementally after software updates or distribution shift. A 2026 multi-company study of autonomous driving testing consolidates participants’ reported criteria into evidence gates on scenario quality, simulation fitness, coverage, safety and performance, data quality, regression needs, deployment equivalence and release readiness. The same study found that acceptance rests on a mix of requirements, quantitative evidence, benchmarks and engineering judgement rather than one universal threshold. So the pack has to show its reasoning, not just a score.
When several candidate models or successive versions must be compared, freezing the replay corpus as a versioned test fixture means every model sees the same scenarios.
Evidence pack checklist for each replay run
- Archive version, scope and any strata added since the last run
- Replay environment: harness version, model version, runtime and hardware, pinned so the run can be reproduced
- Timing tolerance, the clock source each log was recorded against, and the logs rejected for falling outside the band
- Noise threshold and smoothing used to separate regressions from non-determinism
- Strata definitions, sample counts per stratum, and the confidence and reliability each count supports
- Operating point, per-stratum miss counts and true-negative rate
- Every failure with its adjudication against ground truth, and the label uncertainty that remains
- What changed since the previous run, and which conclusions carry over
The study’s authors say emerging approaches should be judged on whether their evidence is controllable, reproducible, interpretable and useful for assurance and release decisions. The checklist is our way of meeting that bar. Failures found in the field should feed the regression suite once fixed. Our article on turning one failure into a durable scenario-based CI safety gate covers that step.
We would treat any evidence-pack format for emergency-alert regulators assessing AI overlay systems as unconfirmed until the regulator agrees it. The nearest precedent is vehicle type approval: meeting UN-R157 for an SAE Level 3 system requires correlation between the simulation toolchain and physical testing in the specific application domain. Build on that analogy and confirm the format with your regulator liaison before the first submission. Where the regulator fixes an archive window, five years of footage for example, the protocol should cover all of it. Even then, sensor upgrades or new regions can leave older footage less representative of today’s conditions. Reuse also assumes the metrics still fit. If the model’s scope or the alerting rules change materially, the protocol itself needs a versioned revision.
How much road and track testing still has to back a replay result?
We would not let replay carry a safety argument alone. Some field testing is always needed, aimed at what replay structurally cannot show. A 2023 taxonomy of driving simulation describes plain log replay as high in visual fidelity but low in diversity, and as non-reactive, so it supports validating perception and prediction but not exploring hypothetical scenarios or different conditions. A fusion fix that changes what the planner does next leaves the logged world unchanged, because the other vehicles in the recording never react to it. The 2026 multi-company study found that different environments produce different kinds of evidence rather than substituting for one another.
Driving volume alone will not settle it either. RAND (Kalra and Paddock, 2016) found it would take hundreds of millions, and sometimes hundreds of billions, of test miles to give clear statistical evidence of autonomous vehicle safety. The rarest events are often the ones that decide a release, which leaves real uncertainty even after millions of test miles.
The useful question is what field testing has to prove. We would put it in two parts. First, correlation for the scenario classes the replay claim covers. A 2024 study comparing software-in-the-loop simulation with track tests ran each scenario five times on the track and found Pearson correlations mostly above 0.9. It rated model accuracy excellent below 10% relative RMSE and good between 10% and 20%. Second, effects that only show up on the vehicle. Simulation typically leaves out hardware delay. In the VP-AutoTest platform’s trials, a real vehicle braked more gently than its simulated counterpart and set off another vehicle’s emergency stop. If the evidence so far suggests a system may not be safe enough, the real-world testing burden should rise, not fall.
Running log mining and replay: on-prem GPUs or cloud burst?
No single answer fits every team. The choice depends on how steady the replay load is, where the data is allowed to live, and how strictly runs must be reproducible. We would treat per-release replay of a stratified set as bursty load suited to cloud capacity. Mining the full archive for new strata is heavy and recurring, and with data already on site we would weigh egress and storage costs toward on-prem. Data-residency obligations can rule options out before cost is even considered, so check them first.
The protocol should pin GPU model, driver and inference runtime on whichever side it runs, so environment changes are not mistaken for regressions. To forecast cost, multiply replay hours per release by the expected release cadence, add archive growth from the fleet, and divide by measured throughput on the candidate hardware. LynxBenchAI measures throughput, latency and cost per unit of work across accelerators, giving a hardware baseline for that figure. It does not run your replay pipeline, so confirm the estimate on a sample of your own logs. See the LynxBenchAI results and methodology for that baseline.
Who can build a replay validation protocol a regulator will accept?
TechnoLynx builds eval and regression harnesses for perception and computer-vision systems in manufacturing and automotive. That covers scenario coverage and pass criteria, the fixture format and versioning, the test runner, and checks that repeated runs agree. On build versus buy, we would judge in-house replay viable for a team with its own validation engineers and an established safety case. The specialist work is designing the protocol and the statistics so the evidence holds up under review. When choosing a partner, ask to see how they set and justify tolerances and bounds, not only how many scenarios they can replay. For the hardware the replay runs on, LynxBenchAI is the measuring tool.
This is engineering work, not regulatory authority. TechnoLynx’s offers do not include setting your tolerable false-negative rate, legal sign-off or certification; those decisions stay with your regulator, your legal team and any accredited test house. If what you need is road or track test capacity, a proving-ground operator is the right route.
FAQ
Are jitter and latency the same?
No. Latency is the delay between a sensor measurement and its arrival, while jitter is the variation in that delay from message to message. For fusion, jitter matters because it changes the relative timing between sensors, which is what the replay tolerance band controls.
Why is accuracy a poor metric for rare-event detection?
When true events are rare, a model that almost never fires can still score high accuracy, because accuracy is dominated by the common negative class. Recall on the positive class and the false-alarm rate on negatives describe an alerting system far better.
