An incident investigation can show which forecast value was in use at the moment of a trip by replaying an append-only, versioned record of the forecast, because such a log of changes lets the system’s state at any past point be rebuilt. Today’s historian value cannot do that job, because a revised forecast may have replaced the original.
TechnoLynx helps operators prove which forecast version was live at a trip, starting by mapping which feeds and audit duties need point-in-time integrity, then building append-only snapshot storage and a time-synchronised pipeline, as part of its AI engineering for energy and utilities work.
Why today’s historian value is not evidence of what the system knew
In a store that overwrites on rewrite, the value returned today need not match what was stored at trip time. At least one widely used time-series store treats a later write to the same key as a replacement. InfluxDB identifies a point by its measurement, tag set and timestamp, and when a new point arrives with the same three, it keeps the new field value for any matching field key. A corrected forecast written against the same timestamp and field therefore overwrites the value operators saw. InfluxData’s own workarounds (adding a unique tag, or nudging the timestamp) keep both values but change the schema of the points and can complicate queries.
The broader point comes from event-sourcing: a store that only persists the latest state cannot provide the evidence for when and why something happened, while an append-only log of changes lets any past state be rebuilt.
OSIsoft’s PI Data Archive stores forecasts in dedicated “future” points, kept separate from the historic points that hold actuals, so future data is never overwritten by historic data, and a point cannot be switched between the two types after creation. That separation keeps forecasts from being overwritten by actuals. The PI documentation cited here does not address a forecast reissued for the same future hour, so whether your installation keeps the earlier issue is something to test, not assume.
Valid time, record time and the question an investigator is actually asking
A bitemporal database has two axes of time. Valid time is when a fact is true in the real world; transaction time is when the fact was recorded in the database. A bitemporal model carries both, and that is what makes two different questions answerable: “where did John live in 1992?” uses valid time, while “in 1992, where did the database believe John lived?” is a rollback query on transaction time, and the two can return different answers if the data was altered later.
An incident investigation asks the rollback question. It does not care what the best forecast for 14:00 is now; it cares what the system believed about 14:00 at the instant the protection operated. Martin Fowler’s explanation of bitemporal history shows that the model also covers future-dated facts: a change announced on May 5 for May 12 is recorded with a record time of May 5 and an actual time of May 12, so bitemporality applies to the future as well as the past.
The Phorecaster365 reference architecture for pharmaceutical sales forecasting (2026) states the same separation in forecasting terms, as event time versus knowledge time. A correction posted after a forecast cutoff may relate to an earlier date, but it cannot be treated as if it were known at that cutoff, and a replay must reconstruct the earlier information state rather than join everything to the latest data. In that design, source corrections create a new data version instead of rewriting the evidence tied to an earlier forecast, and a human adjustment creates a new version while the original model forecast is retained.
Mechanisms that already implement as-of storage
Several existing tools implement as-of storage:
- SQL Server system-versioned temporal tables: each update moves the previous row into a history table and closes it with a ValidTo set to the transaction begin time in UTC. A
FOR SYSTEM_TIME AS OFquery then reconstructs the data as it was at any past moment, with the time interpreted as UTC. - dbt snapshots: Type 2 slowly changing dimensions create new records instead of overwriting, and (from dbt 1.9) carry
dbt_valid_fromanddbt_valid_tocolumns so you can find the version valid at a given point in time. - Feast point-in-time joins: by default only the event timestamp is constrained, so a value backfilled or corrected after the query timestamp can still be returned. Passing
filter_by_created_timestamp=Trueadds a created-timestamp condition so each row sees only values already available at its own timestamp.
A query that looks point-in-time on event time can still leak a later correction unless it also filters on knowledge time. Any forecast store an investigation relies on should be checked for exactly this behaviour.
How does time synchronisation bind a forecast version to the trip timestamp?
Binding works when the forecast version’s record time, the SCADA sequence-of-events record and the operator log are all stamped against the same reference, so “the version recorded before the trip” means the same thing in every feed. Where versions are linked to snapshots by timestamps from a single clock, point-in-time snapshots can be created without the clock-synchrony problem; across several servers, a snapshot can only be guaranteed consistent within the known upper bound on clock drift between them.
For the protection side, NERC PRC-002-4 (September 2022 draft) requires Transmission Owners and Generator Owners to synchronise SER, FR and DDR data to UTC with device clock accuracy within ±2 milliseconds, and to hold evidence of that synchronisation. That requirement covers SER, FR and DDR data, not the forecast store or the evidence pipeline.
How loose is too loose? A 2026 study of distributed inference pipelines gives a useful rule: timestamp-based ordering is safe only while the maximum clock error between stages is smaller than the minimum time separation between causally related events. In that pipeline, no ordering violations appeared up to 3 ms of injected skew, and clear violations appeared by 5 ms. The figure belongs to that system and will differ in yours, but the rule transfers: if a forecast revision landed a few milliseconds before the trip and the clocks disagree by more than that, the timestamps cannot tell you which came first. For scale, the PNNL time synchronisation report (2017) notes NTP over the public internet usually holds a clock within ten milliseconds of the reference, and around one millisecond with a time server on the local intranet.
Clock failures that make an as-of answer wrong
A NERC lesson learned (2025) describes a newly installed GPS clock that missed the April 2019 week-number rollover, which happens every 1,024 weeks, and so reported a date in 2004; forcing SCADA/EMS servers onto it caused loss of monitoring and control. The same document notes that many audit trails rely on precise time synchronisation and that discrepancies can be exploited to obscure activity. PNNL reports that ISO New England found none of its online streaming PMUs applied the 2015 leap second properly, producing erroneous timestamps and bouncing phase angles.
The practical consequence is that clock health must itself be part of the evidence. The inference-pipeline study proposes a binary “causality health” signal that declares when timestamps can no longer be trusted, so outputs can be annotated or automation gated. Recording something equivalent alongside each snapshot lets an investigator say not only which version was live, but how confident the timeline is.
What makes a forecast snapshot immutable enough to stand up in an investigation?
A snapshot holds up when nobody, including administrators, can alter or delete it during its retention period without that change being detectable, and when its time of existence can be shown independently of the system that stored it. Append-only storage is the base layer: in a transaction-time table records are never deleted, and updates only set an end time on the superseded entry.
Tamper evidence is the next layer. One 2026 proposal for model audit trails stores each event in an append-only JSON-lines log with an RFC3339 timestamp and a SHA-256 hash computed over the event payload plus the previous entry’s hash, so modifying or deleting an entry breaks the chain. Crosby and Wallach (USENIX Security 2009) go further with a history tree: each appended event produces a version-indexed commitment that is signed and published, and incremental proofs show that two commitments make consistent claims about the shared past. Publishing those commitments outside the system lets a third party verify that an entry was submitted, and in what order, without the logging system’s cooperation.
Storage-level retention locks add a third layer. Cohasset’s 2025 assessment of Amazon S3 describes versioned buckets with Object Lock where every attempt to modify a version’s content is rejected, overwriting creates a new version, and version identifiers and creation timestamps are system-generated and immutable. It also records that S3 system clocks synchronise to an external time source and cannot be adjusted by users or administrators. The regulatory test Cohasset cites is a useful bar for forecast evidence too: the system must be able to access and reproduce the original record if it was modified or deleted.
Finally, a trusted time-stamp proves when a snapshot existed. Under RFC 3161, a Time Stamp Authority issues a token as evidence that a datum existed before a particular time, with a hash imprint of the data, a unique serial number and a generation time in UTC. Hashing each issued forecast and stamping it at issue gives an investigator proof of existence that does not depend on the historian’s own clock.
One more element is easy to miss. Phorecaster365 notes that a model file alone is not sufficient to reproduce an earlier service: rollback must preserve the feature definition, calibration state, context schema and hierarchy mapping with it. The snapshot that matters is the published value plus enough context to show where it came from.
How is point-in-time forecast versioning different from point-in-time recovery backups?
Point-in-time recovery restores a database to its state at a past time, while forecast versioning keeps each issue, correction and adjustment as its own version. In PostgreSQL, continuous archiving replays WAL records from a base backup and stops at a recovery target given by date and time, named restore point or transaction ID. After recovery, PostgreSQL creates a new timeline so new WAL does not overwrite the old history.
| Question | Point-in-time recovery (e.g. PostgreSQL WAL replay) | Per-forecast version snapshots |
|---|---|---|
| Purpose | Restore the database to a chosen recovery target | Make a published value reconstructible |
| Unit of history | The whole database state at a target | Each forecast issue, correction and adjustment as its own version |
| How you query the past | Replay WAL from a base backup to a recovery target | An as-of query, such as FOR SYSTEM_TIME AS OF |
The distinction matters in practice. Restoring to trip time works only for times since the base backup was taken, and yields a single consistent snapshot of the database as it was then.
A checklist for a forecast snapshot audit trail
Whether an existing historian or pipeline already meets these needs is a property to measure on its current implementation, not to assume. These are the checks we would run:
- Write behaviour: reissue a forecast for the same future timestamp and confirm the first value is still retrievable.
- Two time axes: confirm every forecast record carries both the time it applies to and the time it was recorded or became available.
- As-of retrieval: replay a known revision event and check the query returns the pre-revision value for a timestamp before the revision, filtering on knowledge time, not only event time.
- Clock alignment: measure offset between the forecast store, SCADA SER devices and operator-log hosts against the ±2 ms UTC bar PRC-002-4 sets for disturbance data.
- Clock health records: confirm clock faults (falseticker alarms, rollover, leap seconds) are logged with the snapshots.
- Tamper evidence: verify a hash chain or signed commitment exists and that breaking one entry is detected.
- Retention: check that snapshot retention covers the audit window; PRC-002-4, for example, requires Transmission Owners to retain evidence for Requirement R1 for five calendar years, and an investigation can extend retention.
- Retrieval time: confirm evidence can be assembled within the turnaround the investigation or regulator expects.
If the first check fails, nothing downstream can fix it: point-in-time proof is impossible until versioned storage is in place, and building an evidence pipeline on top of an overwriting store only produces a confident-looking timeline of revised values. If versioned snapshots already exist, the work moves to the evidence-assembly pipeline and its validation.
Binding the forecast version to SCADA and operator logs
The forecast version is one strand of the evidence. Keying the SCADA alarms, operator actions and forecast-model logs by stable identifiers, and stamping them against a common time source, lets a single timeline be filtered and ordered around the event. Building the full time-ordered reconstruction across control areas is a separate task, as is consolidating multi-algorithm evidence into an auditable regulatory record.
Two limits are worth keeping in view. Retention policies can quietly purge the very snapshots an investigation needs, so the retention window has to be set from the audit and regulatory horizon, not from storage cost. And all of this proves which forecast was in use, not whether that forecast was any good: forecast accuracy is a separate question, and a perfect audit trail of a poor forecast is still a poor forecast. What the audit trail does settle, if it was built before the trip, is the argument over what the system knew, and that argument is far harder to settle afterwards.
