If PTP time sync jitter spikes during highway merges and the tracker starts producing ghost tracks, there are four layers to check. The candidates are queuing on the network and boundary-clock path, sensor firmware timestamping, host driver scheduling, and physical stress on the oscillators. The displacement a residual clock offset causes scales with speed, so the same clock error that does no harm in a car park rules the data out on a motorway. Treat the link between merges and offset spikes as a hypothesis and test it one layer at a time.
The work has three parts: find the layer, prove the fix on the drive that exposed the fault, and put monitoring in place so the next firmware release cannot bring the fault back unnoticed.
Why PTP time sync jitter spikes at highway speed and not in stationary tests
Competing traffic causes queuing delay, vibration shifts oscillator frequency, and thermal expansion causes drift. Any of those can push the PTP offset around.
Network and boundary-clock path. Packet delay variation (PDV) across a network varies with the number of hops, link speeds, utilisation rates and how the switches behave. Much of that variation comes from queuing delay caused by competing traffic, and queuing can change quickly. PTP assumes the forward and reverse delays are similar, so queuing in one direction shows up directly as a skewed offset estimate. The NVIDIA (Mellanox) PTP design guide notes that switches without hardware timestamping or PTP-specific handling degrade time transfer when PTP packets are queued or links carry asymmetric load, and it recommends marking PTP messages with DSCP 46 so they go into the expedited-forwarding queue. PTP-aware boundary clocks split one long path into shorter segments and reduce the impact of PDV. Port-level hardware timestamping also removes the PDV between the port and the control processor.
Physical effects on oscillators. A typical quartz crystal has an acceleration sensitivity below 2 ppb per g. The response is roughly linear up to around 50 g and depends on the direction of the force. Vibration on a mobile platform is random and spreads from below 10 Hz to 10 kHz or higher. Microwave Journal shows a stable ovenised oscillator whose phase noise got more than 30 dB worse under random vibration of 0.05 g²/Hz. Thermal expansion, process variation and ageing all cause drift as well. For a typical crystal at under 2 ppb/g, a sustained 1 g adds roughly 2 ns of error per second. That is why we doubt vibration alone explains large excursions. Our concern is rather that vibration upsets the servo’s frequency estimate. In an NXP presentation to IEEE SA, clock drift (the change in a crystal’s ppm over one second) is named as the largest source of error in gPTP’s neighbour rate ratio.
Sensor firmware. An Ouster lidar in TIME_FROM_PTP_1588 mode keeps a monotonic counter that jumps forward or slows down when a new 1588 sync event arrives. Autoware’s documentation attributes scan-to-scan timestamp jitter to the mechanical design of lidars. Autoware’s documentation gives an example of 0 to 8 ms of noise between scans at 10 Hz.
Host driver stack. The linuxptp ptp4l manual says “timed out while polling for tx timestamp” is likely caused by a driver bug, and names a starved timestamp-processing kworker as one possible cause. Software timestamping on arrival under a non-real-time OS can be off by hundreds of milliseconds, against nanoseconds for hardware timestamping. GPIO timestamping is only deterministic if its interrupt always has top priority, and common FTDI UART-USB adapters add up to 16 ms of buffering delay.
Layered elimination: which layer correlates with the spike
| Suspect layer | What it correlates with | First check | Fix direction |
|---|---|---|---|
| Switch queuing / network load | Mean path delay jitters with network load | ptp4l path-delay mean and deviation; switch path-delay threshold logs |
Hardware timestamping, PTP-aware switching, DSCP 46 marking |
| Boundary-clock path | Port state changes; offset-from-master variation | Port state change logs, offset-from-master variation | Boundary clocks splitting long paths into shorter segments |
| Sensor firmware | Sensor timestamps step or slew while the host clock stays steady | Sensor timestamp mode; scan-to-scan interval spread | Timestamp mode and offset configuration |
| Host driver | Tx-timestamp timeouts | ptp4l log messages |
Kworker priority, driver patch |
The design guide sets out a troubleshooting order. Its twelve steps run from interface status and domain number through message rates, counters, announce data and port state changes to path delay and offset variations and multiple upstream ports. Mismatched sync or delay rates between devices can leave a port cycling through states on announce timeouts. Per-type counters also show whether delay requests and responses run at 1:1.
The cause may be several layers at once, for example vibration and network load acting together, and then it needs more than one mitigation. Bench and HIL rigs reproduce highway conditions imperfectly, so field data still has to confirm the diagnosis. A grandmaster redundancy audit improves availability, but it does not find the source of the jitter.
How much cross-sensor skew can a fusion tracker tolerate before ghost tracks appear?
VehicleMapping describes production fusion stacks that hold end-to-end temporal skew to 1 ms or less before any geometric stage runs. It also argues that temporal misalignment is the main source of registration artefacts in high-velocity mapping. The tolerance for your own tracker has to be measured. Inject known offsets into logged sequences, score the tracker against ground truth, and read the tolerance off the resulting sensitivity curve; no single figure from elsewhere will give it to you.
The tolerance moves with speed, because clock-error displacement scales with speed. The Syncline model (arXiv, 2022) shows that sync-induced error depends on both the platform’s dynamics and the size of the synchronisation error. Slow platforms are limited by sensor noise and fast ones by synchronisation. The error has a translation term and an orientation term, and the orientation term is the heading change during the offset multiplied by the distance to the object. The sensor with the tightest critical sync accuracy sets the requirement for the whole fusion stack. Our reading is that merges may be harsh because they add yaw and lateral motion, which brings the distance-scaled orientation term into play just as relative velocities peak.
Measured sensitivities are severe. In the DejaVu injected-delay attack experiments on multimodal perception (arXiv, 2025), a one-frame lidar delay cut MVXNet’s car detection mAP from 84.1 to 9.7, and a three-frame camera delay cut car MOTA by 73%. Identity switches rose sharply as camera delay grew. The jitter-versus-offset distinction showed up clearly: a constant five-frame delay cost only 1.4% MOTA, while random delays cost 60.5%. A simulation study in the MDPI journal Drones likewise found that association and filtering performance got worse as synchronisation error grew.
Those delays are whole frames, far larger than gPTP’s ±500 ns accuracy over seven hops. In our view, ghost tracks point to millisecond-scale excursions or mis-paired frames, not to nanosecond PTP noise. We suspect a servo step, a timestamp timeout or a failover drift rather than steady-state jitter.
Worked example: what residual skew costs at motorway speed
Assumptions: ego speed of 96.9 km/h (the top of the KAIST speed range in the Ultra-Fusion evaluation), which is 26.9 m/s; a static target; and the translation term only.
| Skew | Displacement at 26.9 m/s | Displacement at 7.0 m/s (25.2 km/h) |
|---|---|---|
| 400 µs | 26.9 × 0.0004 ≈ 11 mm | ≈ 2.8 mm |
| 1 ms | ≈ 27 mm | ≈ 7 mm |
| 1.5 ms (BMCA failover drift example) | ≈ 40 mm | ≈ 10.5 mm |
Compare those figures with the 0.05 m registration RMSE that VehicleMapping uses to gate fusion output. A 1.5 ms failover drift uses most of that budget at motorway speed before any rotation is counted.
You can automate the regression test by running the injection-and-scoring pipeline on every build and flagging any run where ID-switch rate or position error crosses the measured threshold. Results depend on the model and the scenario. Do not extrapolate beyond the range you tested, and re-derive the threshold whenever the fusion algorithm or the sensor suite changes. Turning the curve into a jitter budget that engineering and safety can both sign off is a separate piece of work.
Proving the sync fix on the exact merge that exposed it
To prove the fix, replay the logged highway scenario that exposed the bug through both the old and the fixed pipeline, and compare the artefact metric on the same data. A different drive or a synthetic scene cannot show that a fix holds under the relative velocities that triggered the fault. VehicleMapping describes the loop that makes this possible. Fused output is compared continuously against ground-truth trajectories, structured telemetry records registration residuals and temporal-skew metrics rather than pass/fail flags, and logs follow schemas detailed enough to reconstruct fusion state after the event and replay it in regression tests. Ultra-Fusion’s simulated sequences use the same idea for perturbation studies: they replay the same scene and motion while changing timing offsets. TIER IV’s Driving Log Replayer evaluates perception output per replayed rosbag and averages results across a dataset.
- Cut the failing merge from the log with margin either side, together with the PTP statistics recorded at the time.
- Check replay fidelity first: replayed timestamps must match the recording.
- Run the pre-fix pipeline and record ghost-object count, ID switches and position error against ground truth.
- Run the fixed pipeline on identical data and record the same metrics.
- Apply pass/fail criteria: offset excursion within bound and artefact metrics within the measured tolerance.
- Inject offsets around the bound to show how much margin the fix leaves.
- Add the scenario to CI as a permanent regression case.
You can replay the logged merge against the patched setup instead of booking more highway slots, provided the failure is in the recording and the patch only changes timestamp handling. For per-message cross-sensor skew, a starting threshold is VehicleMapping’s assertion of 1 ms or less. Set the offset-excursion bound from your measured tolerance. A fix that lives in the network, such as a switch or grandmaster change, cannot be exercised by replaying recorded timestamps. It needs HIL replay through the patched network or new road miles.
Passing one logged merge does not show the fix holds at other speeds or in other conditions. The wider method for edge cases is covered in proving fusion and alerting systems through logged-data replay.
Monitoring that catches the desync while PTP still reports healthy
Watch two things: PTP health, and the actual skew between lidar, camera and radar timestamps measured against the fusion budget. Desync can arise after the clock layer, in driver buffering or sensor timestamping, where a PTP-only monitor never sees it.
TechnoLynx builds this kind of cross-sensor latency budget monitor. TechnoLynx starts by deriving the fusion algorithm’s skew tolerance, then instruments the timestamp streams on the target hardware and wires the thresholds into continuous alerting.
On the PTP side, ptp4l prints summary statistics at a configurable interval: offset RMS, maximum absolute offset, frequency offset mean and standard deviation, and path delay mean and standard deviation. A monitoring client can subscribe to per-sync timing data through a UNIX domain socket. Servo stability is signalled by the move from SERVO_LOCKED to SERVO_LOCKED_STABLE, but the offset threshold for that move is disabled by default. The NVIDIA guide describes Mellanox Onyx thresholds that log every mean-path-delay or offset calculation crossing a user-set bound. EBU guidance adds message rates, lost messages and state changes to the list of things to watch.
For direct skew, Autoware’s documentation shows how to check per-lidar header timestamps with ros2 topic echo and confirm concatenation status on /diagnostics. Proposed detection defences include statistical monitoring of monotonicity, jitter and freshness before fusion, plus kinematic cross-checks against IMU, odometry and CAN signals. VehicleMapping adds an IMU-versus-GNSS drift gate of 0.05 m or less alongside skew telemetry. VehicleMapping keeps this skew telemetry in the same validation loop described for replay.
The guard against a future firmware change has two parts. The replay suite runs on every firmware build, and the skew monitor keeps running in production, because a threshold checked only at validation time catches nothing afterwards. Telling timing-driven track drift apart from ordinary sensor noise is a separate statistical question. Fleet-level rollout of these checks belongs to building and proving a fleet-wide time-synchronisation regime.
Can grandmaster or boundary-clock redundancy hold the line while the root cause is open?
Yes, as an interim measure, but only if failover testing shows that excursions during a merge stay inside the tracking tolerance. Where they do not, add a defined degraded mode for fusion.
gPTP lets slaves synchronise to more than one master. In a 2022 automotive Ethernet simulation, the network switched to the time domain served by the secondary clock when the primary failed. Relying on the BMCA alone is slow. In a 2020 IEEE automotive presentation, the BMCA took 2.5 to 3 seconds per link on the bench, a five-level clock tree needed about 15 seconds to fail over, and a 100 ppm local clock drifted 1.5 ms in that time. Keeping primary and backup grandmasters synchronised on separate domains avoids the time jump, provided each endpoint handles redundant domains, for example through AUTOSAR’s Synchronized Time-base Manager. Boundary clocks with holdover keep devices synchronised through outages. Holdover lasts from seconds with a TCXO to days with a rubidium oscillator.
Failover can cause problems of its own. EBU notes that when time suddenly arrives from the other end of the network, cascaded boundary-clock control loops can settle with transient noise. SEL notes that slow spanning-tree reconfiguration can switch devices between grandmasters several times. With three PTP domains, an end station can detect and identify one faulty domain; with two, it can detect a fault but not tell which domain has it. Where redundancy cannot bound the excursion in time, delay-aware fusion with confidence scores can fall back to slowing down, increasing spacing or lowering autonomy. None of this fixes the underlying sensitivity.
The same containment logic applies when a safety-critical rail-crossing system shares a grandmaster with an unrelated ADAS test corridor. Give the rail domain its own grandmaster, made redundant so it is not a new single point of failure, unless an audit proves the shared boundary clocks contain faults. Boundary clocks reduce propagation risk, but they do not remove it when the physical network is shared. A documented, repeatable path-and-hop audit for safety review is its own procedure.
What to log on the demonstrator so the next spike names its own cause
Attribution needs either logging at the suspected boundary or an offline replay against a known-good reference.
TechnoLynx does this attribution work under Production AI Reliability. The first step is to instrument the vehicle’s timing boundaries and build a replay harness against the recorded logs.
At minimum, log the following on the same time axis: ptp4l statistics on every node, switch threshold and port-state logs, raw sensor timestamps next to host receive times, driver timeout messages, IMU and temperature, vehicle speed and lane-change events, and the firmware version on every ECU. With that data you can replay the merge offline and find the first hop whose offset departs from its upstream neighbour. That hop is where the fault first shows. Logging added after an incident cannot capture the original event. Keeping calibration and sync events in the same trail is also what later lets a field complaint be traced back to a specific calibration event.
Once the faulty hop is fixed, show that ghost objects are gone by running the exact recorded merge before and after the PTP fix and comparing occupancy-grid metrics against the pre-fix baseline. For an external audience, report the result as an error bound, not a single accuracy figure. TechnoLynx can build that pre-and-post comparison into CI so every later sync or fusion change is checked against the same scenarios.
Who can root-cause PTP jitter in our fusion stack and keep it from coming back?
TechnoLynx diagnoses timing faults in sensor-fusion stacks and builds the replay regressions and skew monitors that stop them coming back. On build versus buy: a team with linuxptp experience, timing-aware network engineers and good logging can work through the triage above itself. A specialist earns their cost when the fault crosses network, driver and fusion code and nobody owns the whole path. When choosing a partner, ask whether you will own the replay suite and the monitors once the work ends. TechnoLynx runs the engagement through Production AI Reliability, as part of its AI engineering for manufacturing and automotive.
The equipment vendor’s support team is the right route for device-level faults, and dedicated PTP test equipment is the right tool for conformance testing.
FAQ
What is sensor fusion in autonomous vehicles?
Sensor fusion combines camera, lidar and other sensor streams into one picture of the scene, and it depends on all the fused data referring to the same moment. Time synchronisation lets that data be timestamped precisely, which improves fusion precision and reduces the computation needed.
