Deriving a Defensible Jitter Budget for Multi-Sensor Fusion

A defensible jitter budget for multi-sensor fusion is derived, not asserted. You get there in four steps. Replay recorded logs with controlled, injected timestamp jitter. Find the level at which tracking error or ghost-track splitting crosses a threshold set by the safety goal. Confirm the damage comes from timing, not from gating logic or sensor noise. Only then choose the synchronisation fix, and the field monitor that keeps jitter inside the number.

A rule of thumb fails here for a structural reason. Fusion accuracy is limited either by intrinsic sensor noise or by the quality of time synchronisation, and which limit binds depends on how fast the platform and its targets move. In a Kalman filter sensor fusion stack, the standard filter models measurement noise but still assumes each measurement was taken exactly at its reported timestamp.

What jitter budget belongs in the fusion requirements and the safety case?

The number that belongs in the requirements is the largest timestamp error at which the fused output still meets the accuracy the safety goal needs. It depends on the platform’s dynamics and each sensor’s precision, so there is no universal microsecond figure to copy.

TechnoLynx starts by working with the fusion team to frame that budget question per sensor pair and per safety goal. It then builds the jitter sweep and its tracking-error and ghost-track metrics around the existing fusion algorithm and highway logs, and hands over the documented budget with the traceability artefacts the safety case needs.

TechnoLynx delivers this budget derivation and jitter sweep through LynxBenchAI.

The Syncline bound gives the first estimate

The Syncline model breaks estimation error into three parts: sync-induced error, sensor-induced error and systematic error from calibration, mounting and misalignment. It approximates sync-induced error as linear velocity times synchronisation error, plus distance to the object times angular velocity times synchronisation error. The worst-case form uses the expected maximum linear and angular velocities: δ*sync(τ) = vmax·τ + d·ωmax·τ.

The critical synchronisation error, τcrit, is where the worst-case sync-induced error equals the worst-case sensor-induced error. Below τcrit the system is sensor-bound and better synchronisation buys little. Above it the system is sync-bound, and timing limits accuracy. The sensor with the lowest τcrit sets the synchronisation requirement for the whole fusion.

The spread is large. In the Syncline paper’s critical-error table for a large surface vessel, an MRU5 attitude sensor has a τcrit of 0.746 ms, while a uBlox F9P PVT GNSS receiver has 1039.230 ms. The difference comes from precision: the low-accuracy GNSS fix barely notices timing error on a slow platform, while the high-precision attitude sensor is sensitive even there.

One caution applies. The Syncline model was checked against simulation for georeferencing and mapping, not road-traffic tracking. That makes it the right tool for a first estimate and the wrong one for a final budget.

Worked example: a first per-pair estimate from the bound

Assuming 30 m/s, 50 m to the object and a 0.1 rad/s yaw rate, sync-induced error per millisecond of jitter: 30 × 0.001 + 50 × 0.1 × 0.001 = 0.030 + 0.005 = 0.035 m.
τcrit for a lidar-derived track with an assumed 0.05 m error: 0.05 m ÷ 35 m/s ≈ 1.4 ms.
For a camera-derived position with an assumed 0.5 m error on the same geometry, τcrit ≈ 0.5 ÷ 35 ≈ 14 ms.

Under the same motion, the two sensors’ τcrit values differ by an order of magnitude.

Per-pair budgets, measured thresholds and what the assessor sees

Pairs also differ in ways the bound does not capture. In nuScenes, the lidar runs at 20 Hz and the camera at 12 Hz. At 40 km/h, the longest gap between paired samples corresponds to 7.3 m of travel. A study of timing attacks on camera-lidar fusion found detection depended mainly on lidar inputs and tracking mainly on camera inputs. A single global budget set by the tightest pair is safe but can over-constrain looser pairs. A global budget set any looser leaves the tightest pair exposed.

Published sweeps give a feel for the curve but not your number. A paper in the ION’s NAVIGATION journal swept jitter standard deviations of 1, 2, 5 and 10 ms in a lunar descent simulation, and concluded that jitter under 2 ms could likely be ignored for that scenario. That cut-off comes from one lunar-descent terrain-camera scenario, not highway driving.

Define the acceptability line before the sweep, not after. A 2026 arXiv study on driving models set its threshold as the perturbation level at which mean trajectory error first exceeds 115% of the clean baseline. It then paired that threshold with a high-severity failure rate per scenario, because threshold and severity did not move together. A fusion jitter budget benefits from the same two columns: the jitter at which tracking error or ghost-split rate crosses the line, and how bad the failures are past it.

For the safety case, ISO 21448:2022 (SOTIF) covers intended functionalities whose situational awareness comes from complex sensors and processing algorithms. It gives guidance on design, verification and validation, and operation-phase activities. The traceable chain should run like this: safety goal, then required position or velocity accuracy, then per-pair τcrit estimate, then sweep evidence, then the budget, then the runtime monitor threshold. The sweep is only as representative as its logs, so low-speed and adverse-weather segments may need their own analysis.

How do you replay recorded sensor logs with injected timestamp jitter?

The method that isolates jitter-induced tracking degradation is a paired replay. Run the same recorded log twice through the same fusion stack, once untouched and once with parameterised timestamp perturbation. The same random seed and the same filter tuning apply to both runs, and you compare the difference in fused-track accuracy.

The DejaVu testbed shows the pattern. Camera and lidar nodes replayed the KITTI Tracking Dataset under a single PTP domain, and the fusion node paired messages with ROS 2’s ApproximateTimeSynchronizer within a configurable tolerance. The lidar node had built-in temporal offset injection, so misalignment could be introduced without modifying the fusion node. Delays ran from zero to five frames, applied to one modality or to both independently. Tracking was scored with HOTA, detection accuracy, association accuracy, MOTA and identity switches.

A three-configuration design from the NAVIGATION study is worth copying: a synchronous baseline, a run with static latency only, and a run with jitter. It separates “the measurement is late” from “the measurement time is uncertain”, which call for different fixes.

Two things break the isolation. If the process or measurement noise tuning of an EKF or UKF changes between runs, it can hide jitter sensitivity or inflate it. And other noise sources must stay stationary across the paired segments. For the wider practice of validating fusion and alert systems through replay of rare edge cases, including how to pick and curate the logs, see the companion piece.

Is ghost-track splitting caused by the jitter itself, or by gating and association reacting to it?

Attribute ghost-track splits to timing only when the split rate tracks injected jitter magnitude on paired replays and disappears on the jitter-free baseline. If splits persist without jitter, or show no correlation with it, the fault sits in association, calibration or detection logic instead.

The mechanism runs through the innovation covariance. When timing noise is not modelled, the actual measurement error is larger than the filter expects, so the state covariance underestimates the error and the estimate jumps. Timing errors also introduce nonlinear error terms, even into a linear system. Our reading is that in a gated tracker this pushes more true detections outside the validation gate: the track coasts or drops, and the next detection seeds a new track. That is a ghost split caused by timing but executed by the gate.

The gate also distorts the diagnostics you would use to check this. Only measurements with a normalised innovation squared (NIS) below a threshold reach the update. As a result, post-gate NIS follows a truncated chi-square distribution, and its mean is no longer the nominal value. A 2025 arXiv analysis showed that nearest-neighbour selection between just two in-gate candidates cuts the expected selected NIS to about 0.912, less than half the nominal two-dimensional reference. A well-matched tracker can therefore show post-gate NIS below the nominal reference.

Timestamp-based pairing applies its own tolerance window. In the DejaVu testbed, the synchroniser formed pairs whose reported timestamp difference stayed within tolerance while the actual misalignment was large. More generally, synchronisation error reaches local association, filtering and track fusion separately, so attribution has to say which of the three is failing.

Diagnostic checklist for ghost-track attribution

Ghost splits reproduce on the jitter-free baseline replay of the same log: not a jitter problem; look at association, calibration or the detector.
Split rate rises with injected jitter while seed, log and tuning stay fixed: filter timing sensitivity.
Split rate responds to gate threshold changes but not to jitter magnitude: gating and association logic.
Post-gate NIS compared against a gate-aware corrected reference, not the nominal chi-square mean.
NIS consistency checked online on real data, which needs no ground truth.
NEES evaluated separately wherever ground truth exists, because a corrected NIS does not guarantee state-space consistency.
Synchroniser pairing logged with both reported and true offsets, so pairing errors are not misread as filter errors.

The filter change for this test is small. Adding timing uncertainty to the innovation covariance removed the large jumps and brought the error back inside the uncertainty band in Arne Baeyens’ 2023 simulation. If adding that term lowers your split rate while the gate stays unchanged, the attribution to timing is strong. Calibration drift can coexist with jitter and confound the result, so check extrinsics before you conclude.

Is the false alarm coming from sensor noise, timing misalignment or the fusion algorithm’s interpolation?

A fusion false alarm, such as an edge spike that triggers false scrap calls on an inspection line, can be pinned to one of three causes. Replay the same logged data and compare three staged outputs: raw sensor only, time-aligned but unfused, and fully fused.

TechnoLynx frames the spike as a signal-chain fault and builds the staged-output logging and replay harness on the target hardware. It reproduces the spike on demand to show which stage introduces it, then hands the harness over so the line team can re-run it whenever a new spike appears.

The decision logic follows from where the spike first shows up:

In the raw stream alone: sensor noise at the physical edge, which calls for a hardware or placement fix.
Only after time alignment: a calibration or synchronisation issue, which is tunable.
Only after fusion, with clean and aligned inputs: an interpolation or sampling-rate mismatch, which needs redesign rather than recalibration.

This mirrors the Syncline decomposition into sensor-induced, sync-induced and systematic error. Known root causes of camera-lidar misalignment include different operating frequencies, trigger mechanisms, timestamp delays, and intrinsic or extrinsic noise.

To tell calibration from a sampling-rate problem quickly, check whether the error scales with relative motion. Sync-induced error grows with velocity and angular rate. A spatial calibration bias is an affine offset, while temporal misalignment comes from the independent motion of the platform and the objects and cannot be represented by a single affine transform. In the TimeAlign study (arXiv, 2024), a spatial-alignment method failed to recover detection accuracy under a single lidar frame of lag. For cameras, measured offsets have also tracked exposure time, made up of fixed triggering and communication delays plus half the exposure, so an exposure change can move the offset.

Spikes can have several causes at once, for example noise made worse by marginal sync, and offline replay may miss line vibration or electrical noise. Treat a single-cause verdict as provisional until the fix removes the spike on the live line.

Hardware trigger, PTP or interpolation: which fix stays inside the latency budget?

Hardware triggering, where physical access allows it, is the fix most likely to bring jitter back inside budget. PTP comes next where triggering is infeasible and the network can deliver the precision. Software interpolation or extra filtering is acceptable only if its measured latency cost still leaves margin under the real-time limit.

TechnoLynx turns the jitter problem and the latency ceiling into a bounded design space and evaluates trigger, PTP and interpolation against the existing fusion architecture. It integrates the chosen method on the target hardware and measures end-to-end latency and throughput under production-like load before rollout.

Approach	What it fixes	Depends on	Watch for
Hardware trigger	Simultaneous acquisition, with timestamps taken close to the physical measurement	Trigger wiring and sensor support, such as Ouster’s opto-isolated sync input	Rolling-shutter and scanning sensors still need per-point time models
PTP (IEEE 1588 / gPTP)	A common time base across sensors, which Ouster lidars support directly	Infrastructure quality: clock drift was the largest error source in NXP’s gPTP analysis	The network may be outside your team’s control
Timestamp correction or online calibration	Estimated offsets applied before fusion	Sufficient, non-constant motion to estimate the offset	Small spurious corrections degrade the filter, so apply them above a threshold
Covariance inflation for timing noise	Unmodelled jitter in the filter update	A known jitter distribution	Trades tracking responsiveness and precision for consistency
Interpolation between samples	Aligns streams to a common instant	Locally linear motion	Fails for agile manoeuvres and larger jitter

Two rules keep the fix honest. First, budget any added false-alarm filtering and the sync fix together against one latency limit. Second, the throughput contract is re-validated after installation, never assumed from a vendor specification. When the residual timing error turns out to come from the network rather than the sensors, tracing and taming highway-speed PTP jitter in sensor fusion covers that diagnosis.

Can the budget test be re-run for every firmware or algorithm change?

A repeatable sweep replays a fixed log set with injected jitter across a defined grid of magnitudes, distributions and scenarios, scores tracking error against the agreed threshold, and runs unchanged on every new firmware or tracker version. That way the budget is re-checked rather than re-derived.

TechnoLynx first defines the tracking-error metrics, the jitter injection model and the target scenarios with the customer. It then builds and validates the replay and injection tooling against the fusion pipeline on the target hardware, and packages the sweep to run automatically on each release.

Procedure: building the release-gate sweep

Freeze a versioned log set weighted to high speeds and turn rates, since sync-induced error scales with linear velocity and with angular rate times distance.
Measure precision demand per scenario rather than assuming that complex scenes are fragile. In the 2026 driving-model study, lane-keeping and vehicle-following degraded earlier than intersections.
Sweep magnitude on a grid around the Syncline estimate, for example the 1, 2, 5 and 10 ms levels used in the NAVIGATION study.
Sweep distribution as well as magnitude. In the DejaVu study, a constant five-frame delay cut MOTA by 1.4%, while random delays cut it by 60.5%.
Include step and compression patterns. Stepping the grandmaster clock back by 0.20 s every 0.22 s pushed alignment metrics past a 0.10 s synchroniser tolerance that held in the benign phase.
Delay modalities jointly as well as singly. In the DejaVu study, a one-frame delay cut car detection mAP by 7.4% for camera and 65.5% for lidar, and by 89.2% when both were delayed.
Record the threshold and the severity past it for each pair and scenario, and compare against the stored budget as a pass/fail gate.

Runtime monitoring closes the loop. The DejaVu authors list statistical monitoring of monotonicity, jitter and freshness counters before fusion as a detection measure. Setting that monitor’s alarm level from the documented budget ties the field trigger back to the safety goal. The method has a limit: sensitivity is specific to the fusion architecture, so a substantial redesign means re-deriving the budget, not just re-running the gate.

Take a lidar firmware update that leaves measurable residual jitter. That figure is acceptable or not only relative to the documented per-pair budget. If a hardware trigger or timestamp correction brings it inside budget without destabilising the fusion covariance, absorb it in software. If not, treat it as a timing regression to take back to the vendor, citing the timing specification. In the TimeAlign paper, GraphBEV’s spatial alignment did not recover detection accuracy under a one-frame lidar lag. Whether to roll back meanwhile is an operational decision about cost and fleet disruption, and timing analysis alone does not settle it. Where perception inference eats the latency headroom a sync fix needs, the Inference Cost-Cut Pack is the engagement aimed at that side.

Telling a real near-miss from routine traffic in labelled incidents

Validating a fusion model against labelled incidents only works if “near-miss” is defined in ground-truth terms before the run. Otherwise severity labels slide into proxies. The 2026 driving-model study was explicit about this: without collision, rule-violation or near-miss ground truth, its high-severity band was a proxy for downstream severity and not a claim about realised outcomes. Where ground truth exists, state-space consistency (NEES) should be checked separately from innovation consistency.

Production AI reliability is the discipline that builds task-specific evaluation harnesses, regression suites and release-readiness checks, and it is distinct from model accuracy. The definition of a near-miss itself is a decision for the safety team, and the fusion data cannot settle it.

Who can derive and validate a jitter budget for our fusion stack?

TechnoLynx derives and validates jitter budgets for multi-sensor fusion stacks. The work covers framing the budget per sensor pair and safety goal, building the jitter-injection replay and sensitivity sweep around the existing tracker and logs, separating timing from gating and noise, and integrating the synchronisation fix on the target hardware. Our sector work is described under AI engineering for manufacturing and automotive.

This work produces engineering evidence for the safety case. It covers the engineering part, not safety certification or regulatory sign-off. Sign-off belongs with your functional-safety team and an independent assessor.

FAQ

What is the difference between latency and jitter in sensor fusion?

Latency is a delay between capture and use. When it is static and known, a filter can account for it directly, for example by treating it as a filter state. Jitter is uncertainty in when the measurement was actually sampled.

What are the limitations of the Kalman filter for sensor fusion?

The standard Kalman filter assumes each measurement was taken exactly at its reported time. Timing errors also introduce nonlinear terms, so even a linear system with sampling-time uncertainty gets a suboptimal linear update.