Time series cross-validation exists because ordinary shuffled k-fold gives misleading scores on time-ordered data. Shuffling puts later observations in the training folds and earlier ones in the test fold, so the model is scored on samples that sit right next to what it learned from, and the score comes out inflated. The honest way to check whether a model will hold up on a future period it has never seen has two parts. During development, use forward-only folds: an expanding or rolling window, a gap where dependence is strong, and whole-period folds (leave-one-period-out). Then, before you deploy, run one final held-out later period that is big enough for a drop in AUC to be told apart from noise.
Why shuffled k-fold misleads on time-ordered data
Observations close together in time are correlated (autocorrelation). KFold and ShuffleSplit assume independent, identically distributed samples. On time series data, the scikit-learn user guide says, that assumption creates “unreasonable correlation between training and testing instances” and gives poor estimates of generalisation error. The same guide gives a concrete case: news articles ordered by publication time. Shuffle them and the model is “tested on samples that are artificially similar (close in time) to training samples”, which overfits it and inflates the validation score. Kapoor and Narayanan list k-fold cross-validation on temporal data as a form of temporal leakage. They define temporal leakage as a test set holding data from before the training set, so that the model is built with information “from the future”.
The damage goes beyond a slightly optimistic number. Kapoor and Narayanan re-examined civil-war prediction papers that claimed complex models beat logistic regression. Once they corrected the leakage, the complex models did no better than the baseline in every case but one, and in that one the AUC gap fell from 0.14 to 0.01. In those papers, leakage produced the model advantage the studies claimed.
The tscv documentation shows the mechanism with a toy series. The true rule is y at time t equals x at t−1. A naive split over the first five points instead supports the rule y = 1 − x, because the test point can be inferred from its neighbour in the training set. The model learned a rule that holds only because test and training points were adjacent.
When plain k-fold is still defensible
Bergmeir, Hyndman and Koo (2018) showed that for a purely autoregressive model whose errors are uncorrelated, standard K-fold CV compared favourably with out-of-sample evaluation. Empirical work points the same way under stationarity. In one study of echo state networks, k-fold CV gave “very good and stable results” on the longer, more stationary Sunspots and Electricity series, while the shorter, drifting series did better with forward-facing schemes.
Our reading is that k-fold is a reasonable choice when the features are lags, the residuals are close to white noise and the process is stable. In fraud data, temporal patterns shift as fraud tactics evolve. In that setting the forward-only schemes below are the safer default.
How sklearn TimeSeriesSplit works, and expanding versus rolling windows
TimeSeriesSplit in scikit-learn returns the first k folds as the training set and fold k+1 as the test set. Each training set is therefore a superset of the one before it: an expanding window. Any surplus samples go into the first training partition, which is always used. The signature is TimeSeriesSplit(n_splits=5, *, max_train_size=None, test_size=None, gap=0). By default test_size is n_samples // (n_splits + 1), and max_train_size caps the size of any single training set. With that cap set, training stops growing once it reaches the limit and the window slides forward at a fixed length.
The splitter has one requirement that is easy to miss. The docs state that samples must be equally spaced if metrics are to be comparable across folds, because only then does each test set cover the same time duration. Irregular data breaks this: equal sample counts no longer mean equal time spans. In that case, split on period labels rather than on row counts, as the grouped approach later in this article does.
Choosing between expanding and rolling windows depends on how the process changes. The echo state network study describes its walk-forward-style scheme as keeping the training length fixed and training “locally” in time. That rests on a weaker stationarity assumption than training on all history. The same authors draw a useful line between a process that “wanders” back and forth, where wide coverage of history helps, and one that “drifts” in one direction, where the data just before the test interval is most valuable. They found that forward-facing schemes picked good forward-predicting models when the process was evolving, but which k-step variant came out best was inconsistent, and choosing the wrong one could hurt results noticeably.
Our judgement is to use an expanding window when older regimes are likely to recur and a rolling window when old behaviour is now misleading. Running both is informative, since the echo state authors suggest comparing the schemes’ effectiveness to judge non-stationarity.
Why put a gap between training and test folds
A gap removes the samples just before each test fold from training, so the observations most strongly correlated with the test fold cannot leak into the model. In TimeSeriesSplit the gap parameter sets the number of samples excluded from the end of each training set. The tscv documentation calls the gap a Chinese wall: the thicker it is, the weaker the train-test dependence, “given that temporal dependence decays with distance”. Racine’s hv-blocked cross-validation removes the observations adjacent to each test block for exactly this reason.
A gap also has a second, more practical use. Fraud labels arrive late, because chargebacks and investigations take time. The fraud detection handbook therefore separates training and validation sets with a delay period matching the one in production, and sets the validation length equal to the test length. In that case the gap mirrors a real operational constraint.
The echo state network authors list three objections. A gap costs training and validation data. On non-stationary data it does not remove the dependence anyway. And on non-stationary data, when you predict what directly follows, the most recent data is likely the most valuable. Their experiments gave “mixed results… leading to no clear recommendation for or against”. Our judgement is to use a gap when labels mature late or when features are built over trailing windows that overlap the boundary. Otherwise, size it to the dependence you can actually measure rather than adding one by reflex.
Grouping folds by whole periods: leave-one-period-out and GroupTimeSeriesSplit
Fold by whole periods, not by rows, when each period shares context that the model will not have for a new period. The scikit-learn guide states the general principle for grouped data: every sample in the validation fold must come from groups that do not appear at all in the paired training fold. LeaveOneGroupOut holds out one group per split, and the guide notes that the groups “could be the year of collection of the samples”, which turns it into cross-validation against time-based splits.
In the echo state network study’s k-fold setup, training data coming after the validation fold “should not necessarily be considered a problem”. Kapoor and Narayanan’s rule for claims about the future is stricter: test data should always carry a later timestamp than training data. Use leave-one-period-out (in both directions) as a robustness check across regimes, and keep a forward-only test for the deployment number.
For forward-only grouped folds, the mlxtend GroupTimeSeriesSplit lets you group data before splitting, whereas TimeSeriesSplit works only at the record level. It supports both rolling and expanding windows through window_type, a gap_size that skips whole groups between training and test, and a shift_size step. Groups must be consecutive. It can also run a single holdout with n_splits=1, which TimeSeriesSplit cannot.
A worked study shows the pattern. A dropout-prediction study held out each of 15 entry cohorts in turn. It merged cohorts too small for stable estimates with their neighbours to make 16 folds, and framed the design as mimicking deployment on a new cohort. It reported low variance of ROC-AUC across folds, with no cohort behaving differently from the rest. Consistent folds like these show the findings are not driven by one cohort’s anomalies. One unusual fold is just as informative, because it points to a regime the model has not learned.
Leave-one-crisis-out in risk modelling is the same idea with stressed periods as the groups. Whether a backtest built on two known crises is itself overfitted to them is covered in our piece on backtesting risk engines against historical crises.
One caveat comes from ecology. Roberts et al. (2017) warn that blocks following environmental gradients can hold out whole regions of predictor space, so some temporal folds test extrapolation rather than interpolation. The concern is that unwanted blocking of predictor space then overestimates interpolation error. If one period contains predictor conditions no other period has seen, its fold measures extrapolation. Report it separately.
Out-of-time validation versus out-of-sample validation for fraud, AML and credit
Out-of-sample validation means testing on data not used for training. Out-of-time validation adds one condition: the test data comes from a later period than the training data. In temporal modelling the sequential train, validate, test split is often simply called out-of-sample validation, which is why the two terms get mixed up. In fraud work, out-of-time performance varied dramatically by architecture.
A regulatory governance framework paper for AI fraud detection in U.S. banking trained on the earliest 75% of IEEE-CIS transactions (442,905) and tested on the latest 25% (147,635), keeping preprocessing identical. XGBoost reached a temporal AUC of 0.9004 with ΔAUC of −0.0017, while an LSTM fell to 0.8579 with ΔAUC of −0.0626. The rankings held at 60/40 and 70/30 splits too. The lesson is that the out-of-time drop depends on architecture: the LSTM led on random-split AUC yet trailed XGBoost out of time. The paper’s 0.85 minimum out-of-time AUC is its own proposed default. It was derived from its LSTM result, and the paper says institutions should recalibrate it with their own out-of-time experiments.
The fraud detection handbook adds two practical points. Prequential validation, which repeats the hold-out by shifting training and validation blocks back in time, gives confidence intervals and was the handbook’s most robust option for model selection. It also found that AUC ROC picked a depth of 50 on validation, where the best test depths were 3 to 6. In the handbook, average precision tracked test performance more closely than AUC ROC.
For banking organisations, the Federal Reserve’s SR 26-2 guidance on model risk management says validation “can reveal performance deterioration over time and inform judgments about acceptable performance ranges”. It adds that meaningful deviations lead banks to consider whether adjustment, recalibration or redevelopment is warranted. An out-of-time test provides evidence for that judgement. How we approach model engineering in this sector is on our AI engineering for finance page.
How large the held-out period must be before a performance drop means something
The held-out period needs enough positive events that the AUC confidence interval is narrower than the drop you care about. That number has to be calculated from your own event rate and effect size. It cannot be fixed in advance. A 2019 study of machine learning on electronic health records gives a reference point using the Bamber formula: the maximum achievable 95% interval is ±0.036 with 500 cases and 500 controls, and ±0.011 with 5,000 of each. Ten times the data narrows the interval by about 0.036 / 0.011 ≈ 3.3, close to √10 ≈ 3.2, which matches the usual square-root scaling of standard errors.
LynxBenchAI measures the AI hardware that runs these validation folds; it does not evaluate your model.
Now compare that with the fraud results. The 0.0017 XGBoost drop is smaller than both of those reference intervals. The 0.0626 LSTM drop is larger than both. (Strictly, testing whether two AUCs on the same data differ calls for a paired test, so treat this comparison as a first screen.) The same EHR study required at least 30 cases and 30 controls before computing an interval at all, because below that the AUC may not be normally distributed. In its prospective year, most diagnosis models had fewer than 10 positives, and the intervals widened accordingly.
Small evaluation sets are noisier than fold-to-fold spread suggests. Varoquaux (2018) showed neuroimaging cross-validation error bars of around ±10% at 100 samples, and found that the standard error across folds “strongly underestimates them”. To judge whether a held-out drop is real, compare the AUCs with DeLong’s test or a bootstrap. Bootstrap percentile intervals at the 2.5th and 97.5th percentiles are a standard alternative. Be aware, though, that plain DeLong and apparent-bootstrap intervals can undercover, especially at low events-per-variable.
What about seasonal problems such as injury prediction, where each season is a natural fold and positive events are rare? Our judgement is that within-season folds will overstate performance, because they share rosters, rules and conditions with training. Both the drop and the minimum event count should be estimated for your own data rather than borrowed. If a single future season holds too few events for a usable interval, pool several future seasons, wait for more data, or report the interval rather than a point estimate.
Combining leave-one-period-out folds with a final held-out period
Leave-one-period-out is an extra layer of validation. It does not replace the final forward-chaining test, because the two answer different questions. Period folds show whether performance is stable across historical regimes. The final held-out period shows how the finished model, with all its tuning, does on a later period it never saw. The scikit-learn guide makes the same structural point in general form: a test set should still be held out for final evaluation even when you cross-validate, because tuning against a set leaks knowledge of it into the model.
Validation procedure, in order
- Fix the evaluation period first: hold out the most recent complete period(s) and touch nothing in them until the end.
- Build every preprocessing step inside the fold: imputation, scaling, encoding and feature selection are fitted on training data only, because imputing training and test together inflates “out-of-sample” accuracy.
- Cut features at a hard observation boundary, so nothing after the prediction time enters the inputs; an implausibly perfect score is a reason to audit for post-outcome variables, as one cohort study found when ROC-AUC of 1.00 traced back to two leaked administrative flags.
- Develop with forward-only folds (
TimeSeriesSplitorGroupTimeSeriesSplit), adding a gap that matches the production delay. - Run leave-one-period-out as a stress test and read the spread across folds, not only the mean.
- Calculate the minimum number of held-out events from the effect size you need to detect, the positive rate and the confidence you want, before you look at the result.
- Score the final model once on the held-out period, report the AUC with a DeLong or bootstrap interval, and treat that figure as the deployment-readiness number.
Reading the outcome
| What you see | What it usually means | Next step |
|---|---|---|
| Held-out AUC within the interval of the forward-fold estimate | No detectable temporal degradation at this sample size | Deploy with monitoring; the interval width is your real precision |
| Held-out AUC clearly below, all period folds stable | The newest period differs from history | Diagnose shift before retraining |
| One period fold far below the others | A regime the model extrapolates into | Report separately; check predictor-space coverage |
| Too few held-out events for a 30/30 interval | The test cannot support a conclusion | Pool periods or wait; report the interval, not a point |
A significant drop tells you that something changed, not what changed. Covariate shift, a new segment entering the data and genuine overfitting can all produce the same number. Separating “drift from a new merchant category” from “the model is broken” needs its own diagnostics on feature distributions and per-segment performance. The framework paper’s SHAP rank-correlation monitoring is one proposal for a leading indicator of concept drift. Its threshold is a calibrated default, which institutions are told to set for themselves.
The question this procedure cannot settle is whether the next period will resemble any period you have. Leave-one-period-out shows how widely the model’s performance has varied across regimes so far. Whether the planners using its output can tell a reliable prediction from an extrapolation beyond the training data is a separate design problem, and it deserves its own answer.
Who can measure the hardware that runs these validation folds?
LynxBenchAI measures the AI hardware that runs these validation folds; it does not evaluate your model, so the validation design above stays with your own team.
FAQ
What is hold-out validation?
Hold-out validation sets part of the historical labelled data aside from training and uses it to estimate performance on future data. In time-ordered problems, the held-out part should be the most recent block, separated from training by any delay that matches production.
What is an out-of-sample prediction?
An out-of-sample prediction is one made on data that was not used to train the model, which is the performance a test set is meant to estimate.
Is an AUC of 0.5 good?
No. The C-statistic, which equals the ROC AUC, ranges from 0.5 to 1.0, with larger values meaning better discrimination. A value of 0.5 is what random ranking produces.
