In a data leakage machine learning audit of a remaining-useful-life (RUL) model, the test is simple to state. A feature leaks if any input it aggregates is timestamped after the prediction it feeds, or reached your systems after that prediction was made. To find it, audit feature windows against prediction timestamps. To size it, re-run the same evaluation with point-in-time-correct features and compare the metrics. To prevent it, have the feature store build training data with as-of (point-in-time) joins. Only once the leak is closed does it make sense to ask whether the new model can replace the fixed maintenance schedule. We recommend proving it in three steps: a shadow run, a backtest against known failures and calibrated prediction intervals.

Is this leakage or just overfitting?

Both leakage and overfitting give you a model that does worse on new data than its evaluation promised, but they show up in different places. Overfitting usually appears on the validation set as a model that generalises poorly. Leakage inflates training, validation and test scores together, so the drop only appears once the model sees truly unseen data, often in production. Leakage occurs when information not available at prediction time is used to build the model, giving overly optimistic estimates, for example from cross-validation.

RUL pipelines are exposed to this by their very structure. Labels measure time to a future failure, so every feature window over the sensor history must stop at the prediction time. Any record corrected or backfilled after the fact can also be pulled into a window where it was never actually available. Proxy features are the other trap. Epic’s sepsis model used antibiotic prescriptions as a feature, but antibiotics are usually prescribed after a sepsis diagnosis, so the feature stood in for the outcome and inflated performance. A “work order opened” flag plays the same role in a maintenance dataset. For AI engineering for manufacturing and automotive teams joining sensor histories to maintenance records, that join is the first place to audit.

The problem is widespread. Kapoor and Narayanan (2023) found leakage in 17 scientific fields that use machine learning and proposed a taxonomy of eight leakage types. A 2022 static-analysis study of more than 100,000 public notebooks found leakage in nearly 30% of them.

How to detect data leakage in machine learning RUL pipelines, and how much it inflates accuracy

We recommend confirming leakage in an RUL model with a two-step forensic audit. First, check each feature’s computation window against the prediction timestamp in the code. Then recompute the same model’s evaluation with point-in-time-correct features and measure the difference. The recomputation is what tells you whether a full rebuild is worth it.

A leakage audit procedure for an RUL feature pipeline

1. List every feature with the timestamp its window ends on. Flag any window that reaches past the prediction cutoff, and any input that has no availability timestamp at all.
2. Read the operations that work across many rows first. In the 2022 notebook study, transformations that process one row at a time were safe, and reduce-like transformations across many rows were the suspicious ones. Scaling, mean and standard deviation, and PCA were the most common sources of preprocessing leakage.
3. Rebuild each flagged feature on a representative sample using only point-in-time inputs. Keep the model, splits and hyperparameters fixed so that the feature change is the only difference.
4. Score the leaky and corrected pipelines on the same repeated resampling design and compare their performance. In the bioLeak package, BCa intervals need at least 10 paired repeats, and 20 or more is the recommended threshold.
5. If the gap is negligible, fix the features and move on. If it is material, plan the full rebuild and re-evaluation before anyone quotes the old numbers again.

How big can the gap get? In a 2026 preprint describing bioLeak, a leakage package built for biomedical machine learning, an intentionally leaky pipeline scored a cross-validated AUC of 0.92, while the guarded version scored 0.54 on the same folds. In its case study, 20 paired repeats put the inflation at 0.181 AUC, with a 95% interval of 0.174 to 0.187. In a separate LSTM forecasting study, 2-way and 3-way splits typically kept the RMSE gain from leakage below 5%.

One result in that literature matters a lot for RUL teams. In the bioLeak simulations, look-ahead leakage produced a mean AUC inflation of only 0.028, against 0.244 for normalisation fitted on the full data. With no real signal present, look-ahead leakage was largely undetectable. Timestamp leakage can be small and quiet in aggregate metrics, so the code audit has to find it.

Manual review is not enough on its own. In the 2022 notebook study, 7 of 100 hand labels had to be corrected after a second expert looked at them. The automated tool itself reached 97.6% precision but only 67.8% recall, so it misses some leaks. In that study, the static analysis handled most notebooks within a few seconds. Treat a clean result as a first pass rather than a clean bill of health.

The sample has limits. Leakage confined to particular assets or features can sit outside the subset you sampled. The recomputed drop is an estimate until it is confirmed on the full dataset. And an event-time-only audit will miss values backfilled or corrected after the prediction time. The bioLeak authors are explicit that their audit does not prove leakage is absent: if a relevant grouping variable is omitted, no audit can reconstruct it.

Should our feature store only answer as-of queries?

Yes, wherever you can. A point-in-time-valid feature uses only inputs available at or before the prediction cutoff. As-of queries return only values knowable at each timestamp, a guarantee built into the store instead of left to script review. Mechanically, a point-in-time join needs an append-only history of timestamped feature values and a timestamp on every label. For each label it takes the latest feature row for the same entity at or before the label’s time.

Event time is not enough. A sensor record stamped before the cutoff may only have landed in the warehouse after it, so you should push the cutoff back by a realistic availability delay. That delay differs by feature: a streaming counter may be available within seconds, while a nightly batch aggregate is a day stale by design. Even a correct join on event time reproduces what was true, not what the serving path could read. That is how training-serving skew survives a correct join, and the fix is to join on availability time or train on logged serving features.

Existing feature stores already provide point-in-time joins. By default, Feast’s point-in-time joins only constrain the event timestamp, so a value backfilled after the entity timestamp can still come back. Setting filter_by_created_timestamp=True adds the created_timestamp <= entity_timestamp condition that keeps backfilled values out of training data. Databricks applies point-in-time logic only when you declare timeseries_columns or timestamp_keys. Without them, it matches only rows with an exact time match. Hopsworks builds training data with an AS OF LEFT JOIN against the label table. For preprocessing, scikit-learn pipelines make sure transforms are fitted on the training subset only.

If migrating the feature store is not feasible in the short term, use a strict cutoff parameter on every rolling-window query and enforce it with automated tests. The CAPIRE framework runs a temporal validation check in continuous integration, verifying that no feature uses data after the cutoff. Script-level enforcement is still prone to human error at the edges: timezones, and inclusive versus exclusive boundaries.

Point-in-time checklist before any RUL model reaches a launch decision

• Every label row carries an explicit event timestamp in UTC.
• Retrieval uses an as-of join, never a join to the latest snapshot.
• Each feature declares a data-availability delay, and the join respects it.
• A created-timestamp filter keeps backfilled values out of training data.
• Online and offline paths use one feature definition, not two implementations.
• A backfill of a single known entity reproduces the value it had at the time.
• Sequence windows are generated after the train-test split, with a buffer between folds where they overlap.
• Window boundaries are tested for timezone handling and inclusive versus exclusive cutoffs.

The migration itself can also introduce bugs, so re-run the leakage audit after it lands.

Running the new RUL model in shadow mode beside the fixed schedule

We recommend retiring a fixed maintenance schedule only after running the new model alongside it. The RUL model scores live inputs in parallel and nobody acts on its output. The schedule keeps running until the model’s detections and RUL accuracy have been checked against what actually happened.

TechnoLynx’s Production AI Reliability engagement builds this release gate for RUL models. It starts by agreeing with maintenance and risk owners what “safe to rely on” means (tolerable misses, false-negative limits, criticality tiers), then designs the shadow protocol, the fallback trigger logic and the calibration checks, and instruments the parallel run. Shadow results and calibration evidence are release-readiness artefacts, which is a different question from whether the model is accurate.

A shadow run without explicit go/no-go thresholds tends to stall in analysis paralysis. Portainer recommends measurable acceptance ranges and at least a week of baseline data before the shadow starts. For low-volume or highly variable workloads, it says the comparison takes weeks rather than days. In RUL work the binding limit is how many failure events the trial window sees, not request volume, so the trial length should be set by the failure modes you need to observe. Promotion is a governance moment as much as a technical one: someone named signs off, the change is traceable, and rollback is one action away. Offline proxy metrics also need to correlate with the operational outcome they stand in for.

For an RUL model, the criteria should cover missed failures compared with what the fixed schedule caught historically, the lead time of correct flags, and interval coverage. The fallback (reverting to the fixed interval, or a conservative action) has to be tested before cutover, not designed after the first miss. The asymmetry is documented in the prognostics literature. NASA’s prognostics metrics allow asymmetric error bounds because an early prediction is preferred over a late one, and the RULSurv authors note that overestimating RUL is often worse than underestimating it. Shadow validation is only as good as the length and representativeness of the trial, and a model can look accurate there yet miss a rare failure mode the trial never saw.

How much predicted uncertainty is acceptable before the planning horizon shrinks?

We recommend setting the acceptable level by asset criticality, as a policy choice with a named owner. Sort assets into criticality tiers and give each tier a pre-agreed maximum predicted uncertainty. When an asset’s live, calibrated uncertainty crosses its tier’s limit, shorten the shutdown-planning horizon or call an unscheduled shutdown, with sign-off from that tier’s risk owner. NASA’s prognostics metrics link acceptance to the system’s risk-absorbing capacity. A prediction counts as inside its error bounds only if enough of its probability mass falls within them (the β-criterion). The cost of mitigation influences the bound width, and the time corrective action takes influences the time window. Predictions made too late for any corrective action are of little practical use.

For a sense of where such thresholds can land, Mitici et al. (2023) found that the optimal replacement time for turbofan engines was close to the lower bound of the 99% confidence interval of the RUL estimate. In that study, the approach cut cost by 53% against time-based maintenance. That is one study’s result on one benchmark, not a target for your fleet. NASA’s own framing of decision risk asks “how soon is too soon and how late is too late?”. If the uncertainty estimates themselves are miscalibrated, no threshold choice will make the policy reliable, which is why calibration has to be proven first.

How do we check a 90% interval really covers 90% of failures?

Count. On held-out cases, the true failure time should fall inside the 90% interval about 90% of the time. Don’t assume it does: Kuleshov et al. (2018) note that a 90% credible interval from a Bayesian model generally does not contain the true outcome 90% of the time. Coverage on its own is not enough either, because you can always raise it by widening the intervals.

Check	What it tells you	Trap to avoid
Coverage (PICP) at 90%	Share of true failure times inside the interval	Inflated by wide intervals; pair it with width
CWC	Coverage and interval width together at a fixed level	Single confidence level only
ENCE	Binned match of predicted variance to observed error, no hyperparameters	Unstable below about 500-1000 samples
D-calibration	Whether survival curves are calibrated across 10% slices, tested with a χ² test	Small samples give unstable binned results

A 2025 benchmark of regression calibration metrics found that they frequently disagree, which lets a team cherry-pick a flattering one. Its authors recommend reporting several metrics. Aggregate scores can hide poor calibration in subgroups, so in RUL work check calibration per asset class. When coverage fails, post-hoc recalibration can fix intervals on any regression model given enough i.i.d. data. It degrades when data is insufficient or the distribution shifts. Conformal prediction guarantees, under mild assumptions, that the true RUL falls in the predicted interval at a prespecified rate. Conformal variants built for censored survival data give either marginal or locally valid intervals. With censored failure data, simple plug-in prediction bounds are not asymptotically correct.

Who can validate our RUL model before we retire the fixed schedule?

TechnoLynx validates RUL model transitions for maintenance and reliability teams through its Production AI Reliability engagement. It defines with your risk owners what “safe to rely on” means, designs and instruments the shadow comparison against the fixed schedule, builds the uncertainty-calibration pipeline and tier-based thresholds, and supports cutover with the fallback kept live. Keeping test data away from the people who build the model removes whole classes of leakage. When choosing a partner, ask for the go/no-go protocol in writing and check that the evaluation harness is handed over to you, not kept.

Google’s ML Test Score is a rubric of 28 actionable tests for scoring production readiness. If the parallel run also forces a choice of inference hardware, LynxBenchAI is the tool for measuring that hardware; it does not run or score your RUL model.

Backtesting an RUL model when there are only a few failures to hold out

Replay the new probabilistic model and the old deterministic one over the sensor history before each known early failure, and compare when each would first have flagged elevated risk. If the new model flags earlier on every case, by a margin that matters for planning, the committee has evidence it can weigh. Fix the flag rule before the replay (for example, the probability of failure within the planning window crossing the tier threshold), and build the replay features point-in-time. Otherwise the backtest just re-runs the leak. NASA’s prognostic horizon metric gives a precise way to express the result: the time between the first prediction that meets the accuracy criterion and end of life. Errors near end of life matter more: a 10-hour error is far less significant at 1,000 hours of predicted RUL than at 25.

Two failures is a very small sample, and a good backtest result does not show general improvement. The worse risk is contamination. If those same failures informed feature choices or tuning, they are not a test set: any data-driven choice has to be validated on data it never touched. A backtest also validates detection timing only, not calibration of the full RUL distribution. The RULSurv authors recommend holding out several units to evaluate a predictive model, and re-evaluating whenever it moves to a new application.

If the current model ignores censored units, measure what that costs before the committee meets. Rebuild the model with censoring-aware survival methods on the whole fleet and replay both models against the same historical decision dates. On bearings with 25% censoring, the RULSurv study’s censoring-aware models reached a mean absolute error of 12.6 to 14.7 minutes, against 18.5 minutes for a LASSO baseline that ignores censoring.

What uncertainty indicator should accompany each RUL number shown to planners?

Give planners a calibrated interval and a separate flag that distinguishes the two kinds of uncertainty. Aleatoric uncertainty describes variance in the failure process itself and tells planners how much risk they are taking. Epistemic uncertainty, when large, means the input differs from the training data and the prediction should not be trusted. In an ensemble study on NASA’s CMAPSS data, a model trained on one engine subset showed much higher epistemic uncertainty on another subset it had never seen, which signalled that retraining was needed. Low uncertainty is not automatically good news either, since it can mean an overconfident model.

No test can fully validate behaviour in an operating regime that has never occurred.

Re-auditing for leakage late in delivery without derailing the schedule

Start with a representative sample of assets or trackers, not the full fleet, and have the fast retrain path ready before the result comes in. A static pass analysed most notebooks within a few seconds in one study, so it is a cheap first filter before any recomputation. Leakage confined to assets outside the sample can still escape, so if the sample is clean, schedule a full-fleet check after delivery rather than skipping it.

If leakage is confirmed, we recommend retraining with the existing training pipeline and the point-in-time-correct features swapped in. Keep hyperparameter selection inside a nested cross-validation loop so that the retrain does not trade one leak for another. That assumes feature correctness is the only defect. If the audit turns up other pipeline faults, a fuller rebuild is the honest answer. Tell stakeholders about a possible slip conditionally: say up front that a sample audit is running and what each outcome implies, then give a bounded estimate based on the observed retrain time. Don’t announce a slip before the result, and don’t stay silent until the full audit ends.

Once the model is cut over, the validation questions turn into monitoring questions, and segment-aware drift monitoring and retraining triggers take over from the shadow run. The order of work still holds: a flawed pipeline can produce accurate and replicable results, and neither makes them valid. Calibration evidence built on leaked features would be the clearest case of that.

FAQ

What is the most common cause of data leakage?

In the 2022 study of public GitHub notebooks, repeatedly evaluating on the same test data was the most frequent type, at 18.5% of notebooks, ahead of preprocessing leakage at 12.3%. Preprocessing leakage most often came from scaling, mean and standard deviation computed over the whole dataset, and PCA.

Is 99% accuracy a sign of overfitting?

Not on its own. Inflated scores on both training and test sets are a classic leakage symptom, and you can investigate them by checking target correlations and looking for features with unusually large weights.