Reliable PTP clock drift detection means not reading every movement in offset-from-master as a clock fault. Packet delay variation (PDV) and software timestamping both put variable error into the offset calculation, so offset moves even when the clock is fine. In practice: log offset-from-master, mean path delay and port state, discard delay-inflated samples using minimum-delay (“lucky packet”) selection and an outlier filter, and only then run a change detector such as CUSUM or Page-Hinkley on what is left, with explicit handling for gaps after sync loss or port changes.

TechnoLynx offers to frame, instrument and operate this kind of drift detection for PTP or gPTP (IEEE 802.1AS) timing. The work covers framing what counts as drift for your signal, choosing the test, building the logging pipeline on the target compute, validating thresholds against known-good and known-fault runs, and operating the alerting, including buffering and reconciliation where links drop. The first step is scoping which signal you actually need to watch, as part of production AI reliability work.

Is it drift, or is it packet delay variation?

The offset a PTP slave reports comes from four timestamps. Nokia’s documentation gives mean_path_delay = [(t4 − t1) − (t3 − t2)] / 2 and offset_from_master = (t2 − t1) − mean_path_delay. That arithmetic assumes the Sync and Delay_Req packets take equally long. When their delays differ, the calculated offset carries an error of half the difference, and when the two delays change relative to each other over time, the slave clock shows jitter or wander in response.

Renesas lists the usual causes of PDV inside a switch as processing, statistical multiplexing, buffering, port contention and dropped packets, and across a network as route diversity and misordered packets. Not all of this noise looks the same. In telecom usage, phase noise at 10 Hz and above is jitter and phase noise below 10 Hz is wander. Wander is the trouble, because the PDV processes behind it shift their distributions and means over hours or a whole day, so attenuating them takes a filter with extremely low bandwidth.

Meta’s engineers note that a clock’s offset from its time source keeps changing through constant correction, oscillator drift and plain temperature variation. Servo design limits how much PDV-driven wander the slave clock shows. In Renesas’s tests, a slave with a PDV filter in its servo loop peaked at an MTIE of 0.7 µs, while one with a 0.1 Hz low-pass filter peaked at 41.9 µs and failed the G.8261 wander limits. If the servo passes PDV through as slow wander, a change detector placed after it will flag that wander as drift, and the detector will be working correctly while giving the wrong answer.

Signatures to tell the causes apart

What you see	Likely cause	What sets it apart
Fast scatter in offset	Jitter from software timestamping	Software timestamping adds tens to hundreds of µs of variable delay
Slow offset excursions over hours	PDV-driven wander	PDV distributions and means shift over hours or a whole day
Path delay explodes or goes negative, offset jumps by orders of magnitude	Disabled or missing transparent clock	Path delay itself is wrong; Meta saw offset move from ±100 ns to −400 µs
Occasional offset spikes	Timestamp displacement in the NIC	Occasional outliers, which Meta discarded with an offset outlier filter
Slow offset trend	Oscillator drift or servo failing to track	Offset changes with drift and temperature; a mistuned servo raises no alarm
Stable offset with constant bias	One-step/two-step mismatch	Harder to detect than a clock that fails to lock

The minimum-delay rule that separates offset from PDV

A delay above the tracked minimum cannot be told apart from PDV, so it should not drive a correction. In most networks the minimum path delay stays fairly constant, so it can be found by keeping the last N meanPathDelay measurements and taking their minimum, Min_meanPathDelay(n) = min(meanPathDelay[n+1−N : n]). TI’s algorithm then compares each Sync measurement against that baseline. If the master-to-slave delay is below the minimum, there must be a time offset of at least the difference, and a correction is made. If it is above the minimum, there is no way to tell offset from PDV, so no correction is made.

Nokia describes the same idea as “lucky” or “fastest” packets: some fraction of packets cross the network with minimal queuing, so running a selection filter across all 1588 packets to find them removes much of the network delay variation. Qualifying packets is a trade-off. Too strict, and there is not enough data to track frequency changes; too loose, and the rate estimate takes on excess variation. The method also has a failure mode: if no true minimum-delay messages arrive, the algorithm cannot tell offset error from PDV and may make wrong corrections. TI notes that above 80% traffic load its algorithm still needed refinement because far fewer minimum-delay packets got through.

Two points matter for drift detection. Network reconfiguration can step the minimum path delay, though TI notes such steps are usually infrequent. The detector should also be fed offsets computed from qualified packets only, with the qualification rate logged alongside.

What to log so a PTP drift alert can be trusted

NVIDIA’s PTP design guide says that under normal operation both the mean path delay and the slave’s offset from master should be stable and vary only within a small range of nanoseconds, and that a max/min offset threshold can be set so every PTP calculation outside it is logged. The same guide logs PTP port state changes so transient and permanent changes affecting stability can be identified. The TSN-FlexTest testbed team re-measures path delay periodically because it fluctuates, and checks the ptp4l log for errors or excessive clock deviation before trusting a measurement run.

A minimum log set, per slave port:

• offset-from-master per Sync
• mean path delay per exchange, plus the tracked minimum
• port state transitions with timestamps
• time of receipt of the last Sync message, which Meta’s tooling reports alongside offset and path delay
• servo frequency adjustment, since the ptp4l servo steers the hardware clock through adjfreq
• threshold-crossing events for offset
• an independent check, such as monitoring the NIC’s PPS-out, which Meta uses as a fail-safe alongside a ±50 ns offset window between its Time Card and NIC

Path delay logged next to offset is what exposed the transparent-clock failure in Meta’s logs. Meta’s logs showed path delay exploding, sometimes going negative, and the offset moving from ±100 ns to −400 µs; worse, that offset was itself inaccurate because the path delay calculation was wrong.

Which change detector fits offset-from-master: CUSUM or Page-Hinkley?

The two are related: Page-Hinkley implements the CUSUM control chart for detecting changes in the mean. CUSUM charts are designed to detect small, persistent changes in a process. The summary of Basseville and Nikiforov’s text states that in most applications the main problem is detecting small changes, and that small sensor faults in a navigation system can grow through integration into serious position errors.

CUSUM: four parameters, set from a known-good period

A CUSUM chart takes four parameters: the expected mean μ, the expected standard deviation σ, the slack k (the size of shift to detect) and the control limit H, with μ and σ taken from a normally distributed period in statistical control. In the Measurement Lab implementation those values come from a training segment of normal behaviour and are then applied to the whole series, producing upper and lower sums whose violations mark shifts in either direction. The CUSUM-based CAD method is designed to detect sustained changes, not single anomalous points. For PTP, we would take μ and σ from a period with confirmed hardware timestamping, transparent clocks in place and a stable path delay, and run the chart two-sided, since drift has no preferred sign.

Page-Hinkley: a fading-memory CUSUM

River’s Page-Hinkley detector implements the CUSUM control chart, signals change detections without warning zones, and supports a two-sided test for increases and decreases in the mean. Instead of a sliding window, Page-Hinkley folds each observation into a running mean and uses a forgetting factor to weight recent values more heavily. A mode parameter selects “up”, “down” or “both”. The forgetting factor is the useful lever for PTP: it lets the reference mean follow slow, legitimate change (temperature-driven offset that the servo is tracking) while still catching a break in trend.

Filter first, then calibrate on null data

A 2026 study found that error-stream detectors (ADWIN, DDM, EDDM, Page-Hinkley) attribute every alarm to real drift, and on pure covariate streams they alarm even when the underlying mechanism has not changed. For its matched comparison, the same study calibrated detector sensitivity on stationary null streams to zero observed false alarms. A calibration-CUSUM paper used Monte Carlo draws to characterise the null distribution of the statistic, and showed the chart signalling only at its prescribed false-alarm rate while in control but quickly once calibration was lost. For PTP, the null data is recorded offset from a known-good period: replay it through the detector and raise H (or the Page-Hinkley threshold) until false alarms fall to a rate your on-call team can live with.

We would also run a second detector on mean path delay. A shift there points at the network.

Failures the stack offset will not show you

Several gPTP failures produce a stable bias or degradation that raises no diagnostic error. Promwad describes a one-step/two-step mismatch in which the stack keeps running and correcting, but applies corrections to the wrong timestamp values, producing “a stable but systematically offset clock”. A switch without transparent-clock residence-time correction degrades accuracy to the level of its queuing jitter, typically microseconds to hundreds of microseconds, with no explicit error. A PHY driver that does not timestamp Pdelay_Req and Pdelay_Resp breaks per-link delay measurement even though the datasheet lists gPTP support. Servo bandwidth mismatched to the oscillator either fails to track thermal wander or amplifies delay noise into jitter, and neither raises an alarm in standard gPTP implementations.

Meta reached a similar conclusion for data centres: without transparent clocks, large switch buffers can delay packets by up to a couple of milliseconds, causing hundreds of microseconds of path-delay error and asymmetry with “no simple way to detect this”. Its fleet also hit a NIC whose 128-packet timestamp buffer let non-PTP packets displace legitimate timestamps, which Meta handled with an offset outlier filter that discards occasional outliers and holds the mean frequency during a micro-holdover.

The point that matters most for sensor systems: Promwad notes that a system can show sub-microsecond gPTP offset in the stack log while delivering tens of microseconds of timestamp scatter at the sensor interface, caused by interrupt and DMA latency between PHY capture and application timestamping, and that catching this takes a hardware reference measurement against a common external trigger.

How connectivity gaps and holdover distort PTP drift detection

During a gap, follower clocks lose their reference and diverge at a rate set by their local oscillator. Promwad gives 20 to 50 ppm as typical for a 25 MHz crystal oscillator, which is 20 to 50 µs of error per second, enough to corrupt sensor-fusion timestamps within seconds of grandmaster loss. The same happens to every follower during a Best Master Clock election without hot standby. A Netdev 0x1A talk proposes bounding this explicitly: drift uncertainty U_drift = age × D_max, where age is time since the last sync event and D_max a configured frequency-error bound, so 100 ppm over 1 s gives 100 µs. Age is computed from the PHC time of the last sync ingress.

That gives a clean rule for the detector:

1. Tag every sample with sync state (locked, holdover, re-acquiring) and age since last sync.
2. Do not feed holdover samples into the drift detector as if they were locked; track them against the U_drift envelope instead.
3. When the link returns, treat the first offsets as the accumulated holdover error the servo is about to remove, not as a drift event; exclude a settling window and reset the CUSUM sums or Page-Hinkley reference.
4. Raise a holdover alert only if the observed error on reconnect exceeds the envelope the configured D_max predicted.

Buffered on-board data needs the same tags. Store samples with the local clock reading, sync state and a monotonic sequence number, then reconcile on reconnect by sequence number rather than wall-clock time, so a drift window is neither counted twice nor lost when the clock steps. A CUSUM or Page-Hinkley check holds a few running values per series, so it can run on-board during an outage and sync its state and alarms when the link returns. Treat this reconciliation step as an engineering choice and validate it against recorded outages.

Can a system flag timestamp disagreement before it corrupts a fused track?

Yes, if each sensor’s timestamp carries its uncertainty and its clock domain. The Syncline model defines synchronisation error between sensors relative to a reference timestamp, for example μ_sync = t_GNSS − t_LiDAR, with each sensor stamping measurements on its own clock. The same work notes that FPGA hardware timestamping can reach nanosecond precision while software “on arrival” timestamping under a non-real-time OS can be off by hundreds of milliseconds. Domain boundaries matter too: Promwad puts AUTOSAR Time Sync over CAN at 50 to 500 µs between CAN and Ethernet ECUs, so an IMU on a CAN segment carries tens to hundreds of microseconds of systematic uncertainty.

The Netdev uncertainty model sums estimated offset, the estimate’s own uncertainty, age × D_max and quantisation into a per-event bound. Its ordering rule says A precedes B only if t_A + U_A < t_B − U_B; otherwise the order is ambiguous. A fusion stage can use that directly: when two measurements’ intervals overlap by more than the motion model tolerates, flag the pair before association rather than after the track has diverged.

Separating timing drift from a genuine fusion error then comes down to correlation. The spatial error a time offset causes is the relative velocity multiplied by the offset. Timing-induced residuals therefore grow with relative velocity. Logging per-sensor attribution (clock domain, sync state, age, uncertainty) next to fusion residuals makes that argument checkable by a safety reviewer or classification society, whose acceptance criteria remain theirs. How PTP jitter shows up in sensor fusion at speed is traced in taming highway-speed PTP jitter in sensor fusion; the fusion architecture itself and grandmaster redundancy are separate design questions, though a single grandmaster without hot standby is the failure that makes every follower diverge at once.

Matching the detector to the signal

Signal under suspicion	Method	Watch out for
PTP/gPTP offset-from-master	Minimum-delay selection and outlier filter, then two-sided CUSUM or Page-Hinkley	Constant bias is hard to detect; needs external reference
Mean path delay	Monitor against its stable band	Step changes after reconfiguration
Timestamp disagreement in fused tracks	Per-sensor uncertainty intervals and attribution logging	CAN and software-timestamped sensors
A rate (fraud, escalation, reject) with varying volume	p-chart with per-period limits	Too few events per period
Input or model-error distribution	Drift detectors calibrated on null data	Error-stream detectors cannot tell covariate shift from real change
Any signal over intermittent links	Gap-aware state tags, settling window, reset on reconnect	Double-counting buffered windows

The same discipline outside timing

The method transfers: filter the known noise, model normal from a known-good period, then test for sustained change.

Rate metrics with varying sample sizes

For a monthly missed-fraud rate or a weekly escalation rate, a p-chart sets limits per period: UCL = p̄ + 3√(p̄(1 − p̄)/n), LCL = p̄ − 3√(p̄(1 − p̄)/n), floored at zero, which gives the chart its stair-step limits as n varies. The same guide recommends 20 to 25 periods to establish limits and samples big enough to expect at least 1 to 5 events each. Whether a move from 4.2% to 5.1% over three weeks is significant therefore depends entirely on weekly volume, because the limits narrow with the square root of n. A CUSUM on the same rate is designed to detect small, persistent rises. The escalation threshold should follow the cost of a missed shift, not the p-value alone.

Input population drift

Distributional tests such as PSI or Kolmogorov-Smirnov compare a recent window against the training window. The 2026 drift study is a useful warning: input-space detectors can report only covariate (“virtual”) drift and miss pure concept change, while error-stream detectors call everything real drift, so the type of drift decides whether retraining is worth its label cost.

Telemetry gaps and degraded inputs

Link loss masking sensor or model drift is the same problem as PTP holdover: tag samples with connectivity state and reset on reconnect, as described above. Where input quality itself degrades (wider bid-ask spreads making quotes noisier, for instance), choose the input series that matches the real decision variable before testing for drift, because a mid, bid or liquidity-adjusted series can each bias the conclusion.

A PTP clock drift detection procedure, step by step

1. Confirm hardware timestamping on every PTP message type, including Pdelay messages, and transparent-clock support on every bridge.
2. Measure bias against an external hardware reference at the sensor interface.
3. Log offset, mean path delay, port state and last-Sync time per sample.
4. Apply minimum-delay packet selection and an outlier filter before running the change detector.
5. Fit μ and σ from a known-good period and set k and H.
6. Calibrate thresholds by replaying null data until the false-alarm rate is acceptable.
7. Run two-sided CUSUM or Page-Hinkley on filtered offset.
8. Track holdover against age × D_max and reset the detector after an alarm.
9. Re-validate after any switch, NIC or driver change.

Step 2 is the one no amount of statistics replaces. A well-calibrated CUSUM signals only at its prescribed false-alarm rate while in control, and rapidly once the process shifts. It cannot reveal a stable bias that was there from the start.

FAQ

What is a CUSUM chart?

A CUSUM (cumulative sum) chart is a statistical control chart that tracks the variation of a process and can detect small shifts in its mean. It was introduced by Page in 1954 and is designed to detect small, persistent changes optimally.

What are the different types of control charts for counts and rates?

For attribute data, a p-chart tracks proportion defective with varying sample sizes, an np-chart tracks the count of defectives with a fixed sample size, a c-chart tracks defects per unit with a fixed unit size, and a u-chart tracks defects per unit with varying unit size. For continuous measurements, X-bar charts apply instead.