Correlating SCADA, AGC and forecast timelines into one record you can trust is a clock problem first and a data problem second. Power-system event sequencing is commonly specified at 1 ms, and records from geographically dispersed sources can only be aligned once they share a recognised time standard such as UTC. The work has three parts: calibrate each system’s offset against a shared reference, link related records with trace-context-style identifiers so ordering does not rest on timestamps alone, and prove the merge with injected offsets and a hand-built ground truth before anyone uses it for root cause.
Correlating SCADA, AGC and forecast timelines starts with the accuracy each source can claim
A merged incident timeline is only as precise as its least precise source, so the first job is to write down what each clock can honestly claim. The requirements and the delivered accuracy differ by orders of magnitude, and the gap is where silent mis-ordering comes from.
| Requirement or source | Timing figure | Stated by |
|---|---|---|
| Event sequencing across the power system | 1 ms is common | MacLaren, AltaLink |
| SER and SOE device time tags | 1 ms typical specification | SEL, “The Perfect Time” |
| NERC proposal cited by SEL | one quarter cycle, about 4 ms at 60 Hz | SEL, citing NERC |
| Disturbance monitoring clocks under PRC-002-4 | ±2 ms | NERC technical rationale, September 2022 |
| Monitoring equipment clocks under PRC-028-1 | ±100 µs, with the accuracy of applying the stamp not mandated | NERC technical rationale, August 2023 |
| Status change time-tagged when an RTU is polled | wrong by up to the time between scans, sometimes several seconds | SEL, “The Perfect Time” |
| Time passed down through SCADA master, RTU and IED tiers | a few to several ms of latency per level | SEL, “The Perfect Time” |
| A 100 ppm RTU clock holding 1 ms | must be resynchronised more often than every 10 s | MacLaren, AltaLink |
Two readings follow from the table. Sources that time-tag at the RTU against a disciplined clock can be interleaved by timestamp at millisecond level, while sources tagged at poll time can only be placed within their scan interval, and the merge has to carry that interval as an uncertainty rather than pretend it away. WECC’s guideline makes the same point for SCADA master event recording, which inherits inaccurate time tags from the polling interval.
The second reading concerns scope. Inside a single master station, an absolute time error still preserves the order of events and the interval between them, because every device is set to the same wrong time. Across several control areas that protection disappears: records from several stations only line up within a few milliseconds when a common source of synchronised time feeds all of them. A three-area merge is therefore an absolute-alignment problem, not a relative one.
Forecasts sit on a different timescale entirely. Economic dispatch produces base points from forecast load over a 15-minute (or 5-minute) interval, and AGC absorbs every deviation inside that interval whatever its cause, forecast error included. Millisecond alignment of the forecast feed matters little; what matters is knowing which forecast version was in effect at the moment of the trip.
How do we reconcile clock differences between the EMS, AGC and forecast logging systems?
Offsets are reconciled by measuring each clock against a reference you trust, modelling how that offset moves over time, and refusing to merge a source whose clock cannot be shown to be trustworthy.
The standard measurement takes the time difference to a reference server and corrects it by half the round-trip delay, which assumes the network delay is symmetrical in both directions. Where the logs themselves are the only evidence, the equivalent is a time anchor: a record that carries both the local system time and a timestamp from an external source considered correct. When the two agree, an examiner can say the clock was likely correct at that point.
WECC recommends using routine investigations of transmission line faults to confirm the accuracy of time synchronisation at connected stations. AltaLink applied a shared-event comparison in a lab test of a PTP scheme against IRIG-B, triggering one event simultaneously on two SCADA I/O modules synchronised by different schemes and comparing the recorded timestamps.
A single calibration point does not settle the matter. A check at one moment identifies the skew at that moment and says nothing about how the clock behaved before it. Linear offset interpolation assumes each clock drifts at a constant rate, and because that is only an approximation, ordering violations can still appear, especially when the offset measurements are far apart. Becker and colleagues frame the test as the clock condition: a message must not appear to be received before it was sent, and to avoid violations the timestamp error should ideally be smaller than half the message latency. Their remedy is retroactive correction of timestamps based on logical clocks rather than interpolation alone. For SCADA-to-AGC exchanges the same check applies: an AGC setpoint that appears before the ACE sample that caused it means the offset model is wrong.
When the reference clock itself is wrong
Calibration assumes the reference is sound, and one SCADA/EMS incident shows how that assumption fails. A newly installed GPS clock had missed the April 2019 week-number rollover, which happens every 1,024 weeks, and was reporting a date in 2004. Forcing the SCADA/EMS servers to synchronise to that clock caused a loss of monitoring, control and ICCP connections to neighbouring entities.
Time-quality flags do not always catch a failed source either. NIST’s SP 1500-08 records that after a GPS loss, the time-quality bits in IRIG-B data and in PTP implementations were not properly updated to show the loss of the primary source or degraded accuracy. Redundant time sources are one of the means utilities use to verify that the time is correct. A merge pipeline should therefore not rely on a quality flag alone, and should cross-check every source against an independent reference before applying its offset.
Can W3C Trace Context-style propagation work on SCADA and AGC data?
Yes, at the correlation layer rather than in the field protocols, and what it adds is causal linkage that does not depend on any clock. The W3C Trace Context specification defines a standard way to propagate context information that uniquely identifies individual requests as they pass between services. Its traceparent field describes the request’s position in its trace graph in a fixed-length format, and tracestate carries optional vendor-specific name/value pairs. The trace-id is a 16-byte identifier for the whole distributed trace, and the parent-id is an 8-byte identifier for the request as the caller knows it.
The specification is defined for HTTP, though its authors note it is relevant to other protocols through extensions. The practical design assigns an incident trace-id at ingestion, records each SCADA alarm, AGC action and forecast snapshot as a span or event under it, and uses links to connect records that belong together without a strict parent-child relationship. OpenTelemetry’s tracing API requires that a span can record links to other spans, from the same trace or a different one. Span and link design in general is covered in our article on distributed tracing.
Why bother when timestamps exist? Lamport showed that the “happened before” relation can be defined without physical clocks, and that it gives only a partial ordering of events. A link between an AGC command and the plant response it produced is a happened-before fact, and it stays true even when the two systems’ clocks disagree. Research on digital forensic “hyper timelines” makes the same case for implicit timing information, integrating sequence numbers and positional information alongside timestamps from clocks of different granularity. Timestamps give the merge its scale; identifiers and sequence numbers stop it from reversing cause and effect.
Is there a tool that can auto-merge these feeds instead of reconciling them by hand?
The mechanisms for an automated merge exist in adjacent tooling: tracing back ends, stream processors and forensic timeline generators. Jaeger’s clock-skew adjuster changes span start times and log timestamps when a child span appears to be off relative to its parent because of clock skew between servers. Apache Flink’s watermark alignment groups sources and sets a maximum drift from the current minimal watermark, pausing a source that runs too far ahead until the others catch up. Digital forensics has timeline generators such as log2timeline/Plaso, which are limited by the quality of the data they extract.
On the input side, PRC-028-1 specifies a simple CSV format for SER data so that data from one entity can be combined with data from others into a detailed sequence-of-events timeline.
Automating the merge is worth it. The FERC/NERC inquiry into the September 2011 Arizona-Southern California outages found that entities’ time-stamping methods varied and not all were synchronised to NIST, which made validating event timing time-consuming and difficult. Researchers describing manual timeline analysis call it time-consuming, labour-intensive and prone to errors.
If the forecast system keeps only its current values, or overwrites past runs, the merged timeline cannot show what AGC was being asked to follow when the unit tripped. Point-in-time retention of every issued forecast, keyed by issue time and validity window, is a precondition, not a feature to add later.
How do we test that the merge does not silently mis-align events across sources?
We recommend testing a timeline-merging pipeline by injecting synthetic events with known time offsets, checking that it recovers them, and benchmarking it periodically against a hand-reconstructed ground-truth incident timeline.
TechnoLynx frames the correctness and latency requirements for a merge pipeline across heterogeneous EMS, AGC and forecast sources, builds the offset-injection harness and timing instrumentation on the target pipeline, and validates offset recovery and live-event latency against an agreed budget.
LynxBenchAI publishes measured results for candidate AI hardware, so if the correlation job’s compute host is still being chosen, the first step is to check those results against the latency budget the merge has to meet; it measures the hardware and does not run your pipeline.
Known offsets can be produced at several layers. For containerised components, Chaos Mesh’s TimeChaos simulates clock skew inside a single container without affecting the whole node, by redirecting clock_gettime in the vDSO to a function that applies an injected shift such as two hours backward. For time-distribution paths, the ACTS loop-back technique measures round-trip path delay and sets a clock with less than 10 ms uncertainty, which is a usable reference only for checks coarser than 10 ms. For field devices, triggering the same event on two modules under different synchronisation schemes and comparing their timestamps gives a direct check.
A procedure for offset-injection tests
- Generate synthetic event sets that mimic real structure, including groups of related events of varying size, the way parametrised event-log generators produce ground-truth signatures.
- Apply known offsets per source, placing injections away from the start and end of each series and logging the random seed and every injection choice for reproducibility.
- Compute expected orderings only from the injected data, never from the pipeline’s own metadata, so the check cannot pass by leaking its answer.
- Mark cases where the evidence is too weak as uncertain rather than forcing a pass or fail.
- Score offset recovery error per source and count any reversed cause-and-effect pairs, then run the suite as a regression test on every parser or offset-model change.
The limits are real. Synthetic offsets will not reproduce every drift pattern or timestamp irregularity that heterogeneous field systems produce, and passing the suite says nothing about event patterns it never contained. TemporalBench’s authors state openly that their event injection is for controlled evaluation, not a faithful simulation of real-world causes.
Benchmarking against a hand-reconstructed incident timeline
A ground-truth timeline gives a check that is independent of the injected data. The 2011 inquiry shows what building one involves: the sequence-of-events team resolved discrepancies between sources, built an events database of all known events and times, and verified each record against SCADA and PMU data. The result was a precise sequence of more than 100 events in under 11 minutes, which became the foundation for root-cause analysis and simulation. A published methodology for evaluating LLM-based timeline analysis requires a ground-truth dataset to assess the outcome.
We would compare the automated merge against such a timeline on ordering agreement, per-event time residual and which events it missed or duplicated, and treat every disagreement as a defect until it is explained. Reconstructing a timeline by hand is expensive, so this works as periodic, high-confidence validation on a small sample of past incidents, while injection tests carry continuous regression.
What latency budget should the correlation job meet during a live frequency event?
No source we found sets a latency budget for a timeline-correlation job, so the budget has to be derived from the decision it serves. AGC is a very short-cycle control algorithm whose measurements must refresh within 2 seconds, and secondary control is activated within 30 seconds and reaches full load within 5 minutes. A merged view meant to inform an operator during secondary activation must arrive well inside that 30-second window. One meant to keep pace with AGC itself must finish each update within the 2-second measurement cycle, including parsing, offset correction and any watermark wait for a lagging source. Set the budget from the tightest downstream decision, then measure the pipeline against it under replayed live-event load, not under quiet-hour volumes.
Who can build and validate a merged timeline across our three control areas?
For energy and utilities teams, TechnoLynx frames correctness and latency requirements, defines the performance budget, builds an injection and ground-truth test harness, and keeps regression tests running on the deployed pipeline. A specialist is worth bringing in when the merge has to be trusted for root cause on a deadline. When choosing a partner, ask how they validate offset recovery and how they handle sources whose clocks cannot be trusted.
A utility that only needs a one-off reconstruction of a single past event may be better served by its own event-analysis team, following the NERC guidance that a report’s sequence of events should record times as accurately as possible and state the status of AGC and other critical subsystems.
Publishing the uncertainty with the timeline
The most useful output of a correlation pipeline is a stated uncertainty for each event’s position, alongside its timestamp. A polled RTU alarm placed within its scan interval, an AGC action ordered by a causal link, and a forecast value tied to a retained snapshot are three different kinds of evidence. A root-cause argument built on the merged timeline should be able to say which kind it is leaning on. NIST’s smart-grid timing work makes the same demand of control: utilities increasingly need to know the quality of every time signal before they act on the data. The open question for any given grid is which of its sources can meet that bar today, and which need recalibrating before the next trip is analysed.
