To test whether a crisis model works on a crisis it was not tuned to, judge it on data and scenarios outside its fit. We would structure out of sample testing for a stress model in three layers. First, hold out data in time order rather than in random folds. Second, generate reverse and synthetic scenarios that go beyond the two crises you already know. Third, check every generated scenario for both severity and plausibility before it counts as evidence. A model that does well on the replays and collapses on everything else has been overfitted to two shapes. It has not learned how crises behave.
Have we tuned the model to replay 2008 and 2020, and would it fail on a different shock?
A stress model adjusted until it reproduced 2008 and 2020 outcomes runs the risk of overfitting, keeping variables whose explanatory power is spurious outside the estimation sample. The general mechanism is well documented: a model fitted and scored on its training data gives an optimistically biased in-sample estimate, and only an out-of-sample estimate tells you how it will do on new data. Kupiec (2017) noted that stress test model specifications are usually chosen on in-sample fit (high adjusted R², low residual error, weak variables dropped), and that this carries a constant risk of overfitting and of keeping variables whose explanatory power is spurious.
The 2008 natural experiment
Direct evidence comes from Kupiec’s 2017 study of alternative stress test models, which used the 2008 crisis as a nearly ideal out-of-sample episode. He estimated models on quarterly bank regulatory data from 1993 to June 2008. He then fed them the actual macroeconomic path from September 2008 to June 2011 as the stress scenario and compared the projected bank income with what really happened. The best-fitting model within the estimation sample produced the least accurate forecast over the three-year horizon. Over that period the US banking system’s actual cumulative income before tax and extraordinary items was about $180 billion, while the complex disaggregated model with the best in-sample fit projected a cumulative loss of nearly $459 billion. The simplest specification (one equation, macroeconomic factors only) came closest, but Kupiec still judged it too inaccurate for setting regulatory capital. He repeated the out-of-sample check on an independent September 2004 to June 2007 window, which held no real stress, and the macro-only models were again the most accurate.
The lesson for anyone validating their own engine is plain. Goodness of fit on the crises you trained on is not evidence of generalisation, and it can point the wrong way.
Leave-one-crisis-out, and why it is thin
Leave-one-crisis-out applies the hold-out logic of cross-validation to crisis episodes: fit the engine without one crisis, then test it on the one you left out. It measures directly whether the engine can handle a shock it has not seen. Cross-validation exists to test a model on data it was not estimated on, so that overfitting shows up. It only gives meaningful results when the training and validation sets come from the same population, though, and financial systems are non-stationary: a later five-year period is not a fresh draw from the same population as the earlier one. Random splits are also a poor fit for time series where extrapolation is the point, so rolling, time-ordered splits are the more appropriate design.
The caveat is serious. With only two crisis episodes, leave-one-crisis-out has two folds and very little statistical power. We would read a pass as weak evidence and a collapse as informative. That is why held-out history has to be supplemented with generated scenarios, while keeping in mind that synthetic scenarios rest on assumptions a real future shock may not respect. It is not where validation stops.
The decision rule follows from this. If held-out and synthetic scenarios give performance comparable to the in-sample 2008/2020 results, treat the engine as reasonably generalising. If performance falls apart on them, treat the backtest as overfitted and revise the model or the validation scope before relying on it. The EBA’s 2018 stress testing guidelines (EBA/GL/2018/04) also ask for backtesting of stress testing models and list model performance on out-of-sample data, meaning data not used for model development.
How to generate stress scenarios that go beyond the historical record
One overview lists four ways scenarios are chosen: a historical episode, a hypothetical one, a statistical procedure targeting a given probability, or a reverse stress test seeking the least severe scenario that causes a given outcome. Historical replay, using observed past events to test current portfolios, is the most direct route.
Supervisors expect firms to go further. The Basel Committee’s stress testing principles say scenarios not based on historical events may be warranted when new vulnerabilities appear or when historical data do not contain a severe crisis episode. The EBA guidelines require scenario design to consider hypothetical, non-historical events as well as historical ones. They also ask institutions to deal with historical data underestimating tail risk, for example through severe hypothetical scenarios and by considering events beyond the usual percentile confidence levels. The Federal Reserve’s scenario design policy allows market shocks built from historical episodes, hypothetical events or a hybrid of the two, and says hypothetical events can produce risk-factor moves never seen in history.
Reverse stress testing: how the worst case is found algorithmically
Reverse stress testing starts from a pre-defined adverse outcome, such as a breach of regulatory ratios, insolvency or illiquidity, and works backwards to the scenarios that could cause it. A forward stress test asks what the loss would be under a pre-defined shock. A reverse test asks which joint movements of risk factors would produce a given loss. For a model validator, this inversion is exactly what is needed, because it surfaces failure paths nobody wrote down in advance.
Three algorithmic families do the search:
- Worst-case search over a plausibility region (fixes plausibility rather than the loss). Breuer and colleagues (IJCB) define a region of plausible scenarios by Mahalanobis radius and search it systematically for the scenario with the worst portfolio loss. Widening the radius gives more severe, less plausible worst cases. Narrowing it does the reverse. Either way, plausible scenarios that never happened are included. In their example, a hand-picked “GDP shrinks by 3 percent” scenario cut expected profit only moderately, while worst-case scenarios of the same plausibility were considerably more damaging.
- Empirical likelihood. Glasserman, Kang and Kang (OFR) estimate the most likely scenarios leading to losses above a threshold with a nonparametric empirical-likelihood estimator of the conditional mean of market factors given large losses, and add a sampling method for exploring large-loss regions.
- Conditional reconstruction from one shock. A 2026 arXiv framework rebuilds a coherent multivariate scenario from a shock imposed on one asset class, in parametric (Gaussian), semiparametric (empirical likelihood plus Gaussian or Student-t local sampling) and fully nonparametric (Mahalanobis-neighbourhood resampling) variants. The nonparametric variant avoids misspecification but draws repeatedly from the few historical points near the stress, which shrinks the simulated dispersion. The authors also warn that a global covariance averaged across regimes is unsuitable, because correlations intensify under distress.
In the ECB’s multiple-scenario reverse stress tests, a significant nonlinearity emerged between CET1 severity and macro adversity, partly because amplification mechanisms appear in the tails of the CET1 distribution. A model that scales losses linearly with shock size will miss this.
Stochastic simulation as a forward-looking capital test
Stochastic simulation replaces one adverse scenario with a large number of them and reads capital adequacy off the resulting distribution. Montesi and Papiro (LSE Systemic Risk Centre) define stress testing as assessing a bank’s fragility against all potential adverse scenarios rather than one. Their framework solves a multi-period forecasting model by Monte Carlo, with a copula-based joint distribution built from the marginals. It truncates key distributions so that draws fall within the stress range, and takes correlations from 2007-2012 data. The output is a probability of breaching a threshold such as the minimum regulatory capital ratio. Applied to 29 G-SIBs at two severity levels, it showed a sharp rise in breach probabilities between the milder and the harsher setting.
The ECB’s BEAST-based approach bootstraps historical shocks and draws model parameters from their estimated distributions. Its scenarios are therefore statistically plausible (they rest on past data) and economically plausible (they keep the model’s internal consistency). In that euro-area application, the probability of the system CET1 ratio falling below 10% was 6% at a five-year horizon, 2% at three years and 0.1% at one year.
For validation, the value of stochastic simulation is that it tests the model across a distribution of plausible futures instead of the one path it was calibrated to. Truncation choices and correlation windows are themselves assumptions to stress, though.
Can machine learning or generative models create plausible synthetic stress scenarios?
They can, within limits that matter more than the architecture. The main methods in the sources differ in what they assume and in where they break:
| Method | How it generates scenarios | Where it breaks |
|---|---|---|
| PCA / autoencoder / VAE latent perturbation | Perturb latent factors (typically ±2σ) or sample a VAE’s learned latent distribution, then decode to stressed returns | PCA misses non-linear dependence. VAE sampling gives less direct attribution and costs more compute |
| Conditional VAE on macro variables (MacroVAE) | Generates return sequences conditioned on specified macro scenarios, including inflation-growth combinations rarely or never observed | Still learned from historical returns, so it reflects whatever relationships the training data contained |
| Large-deviations extrapolation | Pushes observed samples to more extreme levels while keeping the configurations that drive large losses, reaching events not in the sample | Only as good as the baseline tail model that defines plausibility |
| Copula / NORTA | Generates synthetic scenarios that preserve marginals and correlations from scarce data. One study expanded 16 high-fidelity flood scenarios to 800 | Preserves the marginals and correlations it was given, and the cited study used flood scenarios, not financial crises |
| Causal Bayesian network with ML-guided sampling | Learns causal structure from data, then steers sampling towards loss-making branches | In simulation, false positives reached about 49% of inferred arcs, and experts still had to filter out unacceptable scenarios |
One trap is specific to validation. A 2025 arXiv PCA/AE/VAE framework is presented as reproducing historical crises, with stress sensitivity seen consistently around 2008 and 2020. Reproducing those two episodes is an in-sample check, and in-sample fit gives an optimistically biased assessment. A generator earns trust when it is evaluated out of sample, as MacroVAE’s rolling evaluation is, and when its novel scenarios survive the severity and plausibility checks below.
When should a stress scenario be narrative-based rather than statistically generated?
Choose a narrative scenario when the scenario has to be communicated clearly and held to a guaranteed severity. Choose statistical generation when you need to know how likely the scenario is and what you might be missing. The Federal Reserve prefers a recession-based severely adverse scenario to a probabilistic one, for three reasons: it guarantees a recession of a specified severity, adverse outcomes are easier to describe in absolute terms than as tail percentiles, and probabilistic uncertainty estimates are model-dependent. Narratives also matter where history is missing. The EBA expects expert judgement to drive severe but plausible scenarios for new products, business lines or acquisitions that lack historical data.
The two approaches are not exclusive. The ECB authors doubt that purely data-driven approaches offer scenarios with clear economic narratives, and propose semi-structural models. They also show how to select, from the set of sufficiently severe simulated scenarios, the subset closest to a chosen narrative, so that its probability can be evaluated. For validation we would use both: narrative scenarios to probe the shocks the business worries about, and generated ones to find the plausible shocks nobody thought to write down.
How CCAR and DFAST scenarios relate to a firm’s own scenarios
Supervisory scenarios test the system, and a firm’s own scenarios are meant to test the firm. Supervisors typically apply the same scenario to every institution so they can see how the system responds to one shock. CCAR and DFAST share the same three Federal Reserve scenarios (baseline, adverse, severely adverse) and differ in the capital actions assumed. CCAR also requires company-run projections under a bank-determined “BHC stress” scenario aimed at the firm’s own vulnerabilities. The Fed’s 2013 guidance says plainly that a BHC scenario featuring only a generic macro weakening similar to the supervisory severely adverse scenario does not meet expectations.
This bears directly on overfitting. The 2026 severely adverse scenario describes a recession comparable in magnitude and length to 2007-09: unemployment peaks at 10%, equity prices fall about 58%, and the international variables deviate from baseline by amounts similar to that crisis. An engine tuned until it reproduces 2008 and passes the supervisory scenario has, in effect, been tuned twice to the same shape. The same source notes that focusing on the bank-specific BHC stress scenario counteracts “model monoculture”, where all banks evaluate risk with the same kind of models. The Fed also observes that varying the hypothetical elements of the market shock limits trading firms’ ability to shift portfolios to minimise expected losses. Your own out-of-sample scenarios belong in that bank-specific space.
Checking that a synthetic scenario is both severe and plausible
The EBA defines severity as the degree of deterioration from baseline in macro and financial variables. It defines plausibility by consistency with current conditions, a coherent narrative and support from probability distributions and historical experience. Plausibility is not restricted to events that have happened, and Monte Carlo methods are acceptable for assessing it. The Basel principles require scenarios that are severe but plausible. Both properties are measurable. The checklist below is how we would gate a synthetic scenario before letting it count as validation evidence.
Diagnostic checklist for a generated stress scenario
- Mahalanobis plausibility. Compute the scenario’s Mahalanobis distance under the risk-factor covariance. This accounts for each variable’s unusualness and for how unusual their joint movement is. A scenario that never occurred can score as more plausible than one that did.
- Tail shape of the plausibility metric. Do not assume the distance is normally distributed. Kritzman and co-authors (MIT Sloan) found the empirical distribution skewed and fat-tailed. Under normality the Fed’s scenarios looked wildly implausible, while on empirical distributions the one-year scenarios looked marginally plausible.
- Covariance window. Choose the estimation period deliberately. The same study excluded COVID data because, in a small sample, extreme scenarios would otherwise look normal. Use local or stressed covariances where correlations shift under distress.
- Severity against historical episodes. Score the scenario against past recessions. The Fed’s severity measure gives 100 to a real GDP decline equal to the Great Recession’s 4.2% and 0 to the average mild recession’s 0.6%. Remember the stated limitation: these scores describe macro outcomes, not the stress they imply for banks.
- Worst case at equal plausibility. For a given plausibility radius, run a systematic worst-case search, which ensures no harmful scenario is missed and prevents a false sense of safety.
- Internal consistency. Confirm the risk factors move together coherently. Shocking one asset class in isolation produces internally inconsistent scenarios.
- Comparison with history and expert challenge. Compare the scenario’s outcomes with actual historical events and have the business challenge its assumptions, since any distributional assumption carries model risk.
What generalisation evidence actually looks like
An honest validation pack for a crisis model reads differently from a replay report. It shows the in-sample 2008/2020 fit next to time-ordered hold-out results. It includes reverse stress tests that name the scenarios that break the engine, and a population of synthetic scenarios that passed explicit severity and plausibility gates, with performance compared across all of them. Stable behaviour across that set is real evidence. A sharp drop outside the two known crises is evidence too, of the uncomfortable kind.
The uncertainty that remains is irreducible. The generators rest on distributional assumptions that carry model risk, and two historical episodes give little statistical power. What the methods above do is shrink the space of failures you have not looked at.
For risk engines in finance more broadly, see AI engineering for finance.
FAQ
Was 2020 worse than 2008?
By the Federal Reserve’s 2026 scenario documentation, the 2007-2009 recession was significantly deeper and longer lasting than the 2020 pandemic recession.
