To backtest a risk engine against 2008 and 2020 losses, you compare its forecasts with realised outcomes over a period that was not used to build it. You test the exceptions against thresholds of acceptability set in advance. Then you work out whether the misses come from omitted factors, other specification errors or plain chance. A high crisis exception count does not by itself mean the model is broken. In 2020, US regulators removed backtesting exceptions incurred between 6 and 27 March, finding they reflected unusual market conditions rather than deficiencies in banks’ VaR models. The real work is in telling those two cases apart, and in recording how you did it in a form a supervisor can check.
Backtesting a risk engine against 2008 and 2020 crisis losses
A crisis backtest of a risk engine has three jobs. It confirms whether realised outcomes fall inside the expected ranges or confidence intervals set around the forecasts. It checks which asset classes or loss buckets show a systematically missed tail. And it decides, for each miss, whether it is a model defect or a feature of that particular crisis.
TechnoLynx builds the pipeline that does this work for a bank’s own risk engine. That means defining the loss-distribution metrics, error bounds and asset-class segmentation with the model owners, implementing the backtest against 2008 and 2020 market and loss data on the target compute, tuning it so repeated runs across scenarios stay quick, and checking the implementation itself before it goes into validation operations. The work runs as a Generative & Agentic AI R&D engineering build, starting with framing the metrics and error bounds.
Exception counts and the Basel traffic-light zones
The Basel Committee’s backtesting framework for a 99% daily VaR counts exceptions. These are the days in the most recent twelve months, about 250 observations, on which the trading outcome was not covered by the risk measure. Under Basel MAR32, exceptions against actual and hypothetical P&L are counted separately, and the larger of the two counts is the one that applies.
The zone boundaries come from binomial probabilities. A model with true 99% coverage will produce five or fewer exceptions in 95.88% of 250-day samples, so the yellow zone begins at five. The red zone begins at ten, the point where the cumulative probability reaches 99.99%.
| Exceptions | Zone | Multiplier (Campbell’s summary) | Presumption |
|---|---|---|---|
| 0-4 | Green | 3.0 | No backtesting concern on its own |
| 5-9 | Yellow | 3.0 + 0.2 × (N − 4) | Increase presumed unless the bank shows the model is sound |
| 10+ | Red | 4.0 | Automatic presumption of a model problem |
In the yellow zone the burden of proof sits with the bank, not the supervisor. That is why a crisis backtest needs a written explanation for every exception, and a count is not enough.
Which backtest statistics apply to a 99% one-day VaR backtest?
The two core tests are Kupiec’s proportion-of-failures (POF) test and Christoffersen’s independence test, which can be combined into a conditional-coverage test. Kupiec’s likelihood-ratio statistic is asymptotically chi-square with one degree of freedom, and you reject the model when it exceeds the critical value. Direction matters. Too many violations means the model under-estimates risk, and too few means it is over-conservative.
Kupiec has a blind spot: it counts violations and ignores when they happen. Two models with the same total exceedances pass the same Kupiec test even if one of them bunches its violations into the volatile weeks.
Christoffersen’s test closes that gap. It asks whether violations cluster in time, because a model with correct unconditional coverage but clustered exceedances is still misspecified: it is failing to capture the volatility dynamics that drive tail risk. The independence statistic is asymptotically chi-square with one degree of freedom, and the conditional-coverage statistic combines it with Kupiec’s. One practitioner reference says to report rejections at the 5% level, with both tests passing for full conditional coverage.
| Test | Question it answers | Distribution under the null | What it misses |
|---|---|---|---|
| Kupiec POF | Is the violation rate right? | χ² with 1 df | Clustering and magnitude |
| Christoffersen independence | Do violations cluster? | χ² with 1 df | Rate errors on its own |
| Conditional coverage | Both at once | χ², combining the two statistics | Weaker against a model that fails only one property |
| Lopez magnitude loss | How far past VaR do the losses go? | Loss function, better for comparing models | Not a formal accept/reject test |
US supervisory model risk guidance (SR 11-7) asks for more than an exception count. It asks banks to check for clustering of exceptions, compare the loss distribution against other percentiles, and document both the choice of tests and how the results are interpreted.
What the 2008 and 2020 records show
The 2008 record shows why the independence test matters. One study of bank VaR performance used data from five large US banking institutions. They found bank VaRs highly conservative before the crisis, with exceedances that were excessive and clustered during it. GARCH-based benchmark VaRs were the most accurate in the crisis period and showed no exceedance clustering. Over the whole sample, exceedance rates for all but one bank were not significantly different from 1%, yet independence was strongly rejected for every bank. An unconditional coverage test alone would have passed all but one of them.
The size of the crisis misses was large. Three banks ran exceedance rates of 2.7 to 4% against a 1% target, and the other two ran 10% and 5%. The authors put this down to a common but very slow adjustment, taking months, to the more volatile conditions.
The 2020 record runs the other way. A Federal Reserve note on the COVID-19 crash found that systemic banks averaged 6.3 exceptions. That is close to the 6 exceptions produced by the best-performing GARCH(1,1) model on a simple proxy portfolio, which points to market conditions, not model deficiencies, as the main driver. US regulators then excluded all exceptions between 6 and 27 March 2020. The PRA took a different route and let firms offset multiplier increases from new exceptions against a matching cut in risks-not-in-VaR capital. The OCC states that it considers whether a regime shift has occurred, such as abrupt moves in interest or exchange rates, major political events or natural disasters.
The exclusion was not a blanket amnesty. Exceptions outside the window were judged to come from month-end valuation adjustments or idiosyncratic risk, and they stayed in the multiplier. That line between market-wide and firm-specific is the distinction your own crisis backtest has to draw.
Localising under-prediction by asset class and loss bucket
The crisis studies cited here report results mostly at bank level. Localising a miss is therefore a design choice you make in your own pipeline, not something you can read off the literature. Basel already applies the idea at desk level: each desk is backtested at the 97.5th and 99th percentiles, and a desk with more than 12 exceptions at 99%, or 30 at 97.5%, in twelve months falls back to the standardised approach. The Federal Reserve’s 2020 analysis found that exceptions were spread across many markets, with credit and rates linked to more exceptions and larger losses.
Two techniques do most of the work:
- Segment-level exception analysis. Run the same coverage and independence tests per asset class, desk or risk-factor family, using a segment mapping you fix before the run.
- Loss-bucket tests. Pearson’s Q test splits the unit interval into bins, for example [0, 0.01], [0.01, 0.05], [0.05, 0.10] and [0.10, 1.00], and compares observed violations in each bin. Under an accurate model the statistic is approximately chi-square with k − 1 degrees of freedom. In Campbell’s review, Pearson’s Q flagged a model that under-reported VaR by 25% in 94.2% of cases, against 79.7% for Kupiec. Multi-quantile tests do best when risk is systematically under-reported.
Segmenting has a cost. Every slice holds fewer exceptions than the whole, so power falls. With enough slices, one of them will fail by chance. The segment list and the multiple-comparison correction should be fixed before any result is seen.
Magnitude belongs here too. One normalising-flow study reports an actual 99% worst-day S&P 500 loss of −9.29% in 2008, against −3.06% from a Gaussian model. The authors call crisis-window comparisons like this suggestive rather than statistically conclusive. Treat such a gap as a pointer to where the tail shape is wrong, not as a test result.
How do you tell genuine misspecification apart from a crisis that will not repeat?
You never get certainty here. You get a reasoned attribution backed by checks a reviewer can repeat. SR 11-7 sets out the question as whether the differences stem from omitted material factors, from other specification errors such as interaction terms or linearity assumptions, or from randomness consistent with acceptable performance. The Basel framework separates a correctable model problem from an unexpected bout of high volatility that nearly all models fail to predict, which does not necessarily mean redesign. A regime shift can produce many exceptions that all share one cause.
| Observation | Leans towards | Next check |
|---|---|---|
| A simple benchmark (e.g. GARCH) misses about as often as the engine | Crisis conditions | Confirm the timing lines up with market-wide moves |
| The benchmark misses far less and does not cluster, while the engine clusters | Engine’s VaR adjusts too slowly to volatility | Compare the engine’s VaR responsiveness with the benchmark’s |
| Fixed parameters miss much more than daily-updated ones | Structural break in the data | Document the break; judge how fast the engine should adapt |
| Exceptions concentrate in one segment in both crises | Possible omitted factor | Root-cause analysis of that segment |
| Misses keep needing the same post-model adjustment | Flawed design or misspecification | Recalibration or redevelopment review |
The parameter check is concrete in the 2020 data. The GARCH coefficient on the lagged squared residual rose from 0.29 to 0.39. A GARCH model with daily updating produced 6 exceptions on the proxy portfolio, while one with fixed 2019 parameters produced 9, as many as the simpler historical simulation models.
Structural differences between the two crises also need writing down. In 2020, trading volume rose and new positions’ P&L largely offset losses on existing positions. In the GFC, liquidity and volume dried up. Backtesting uses hypothetical P&L on the prior day’s portfolio, excluding intraday new positions. So the same engine can look worse in 2020 on hypothetical P&L than the firm’s real result. The PRA also expects comparison with challenger models where practicable.
Does the stress-test engine reproduce 2008 and 2020 system-wide losses within its published band?
A stress-test engine’s claim to reproduce 2008 and 2020 aggregate losses holds only if simulated losses for both crises fall inside the band set in the methodology and every divergence is traced to a named model or data cause. If either condition fails, the methodology or model is revised. Passing is relative to the methodology. A wide band passed is weaker evidence than a narrow one.
One study shows how far off multi-year stress models can be. It used the 2008 crisis as an out-of-sample test of stress models. Actual cumulative US banking-system income before tax and extraordinary items was about $180 billion over three years. The best-fitting complex disaggregated model projected a cumulative loss of nearly $459 billion, an underestimate of $630 billion. The best in-sample fit gave the least accurate forecast, and provisions for loan and lease losses were the largest source of error by a wide margin. Philippon, Pessarossi and Camara backtested European stress tests and did not find a systematic tendency to under-predict realised losses. They also note that a realised-loss backtest gives only one observation per bank once the scenario arrives.
The two studies test different things: US income models through the 2008 crisis, and EU credit-loss sensitivities for 2014 to 2016. So they do not contradict each other. Neither supports a general claim about stress models.
Explaining a divergence after the fact invites hindsight bias. The cure is procedural. Write the attribution categories and the evidence each one needs before the comparison is run, and record any category added afterwards as such.
Showing reproducibility to supervisors and auditors without disclosing model internals
You can show reproducibility without opening the model, because some backtests need only forecasts and outcomes. The Strict and Intercept ESR regression backtests need only ES forecasts and realised returns. The authors stress that regulators receive ES forecasts but not the extra inputs that other tests require. The e-backtesting method of Wang, Wang and Ziegel is model-free and uses only losses and forecasts. A supervisor or auditor given the versioned forecast and outcome series, the test specifications and the pre-committed tolerance can rerun every statistic without seeing a line of model code.
Around those series sits the validation report. SR 11-7 expects reports to set out what was reviewed and the deficiencies across conditions, with an executive summary, a statement of model purpose, and a synopsis of results that includes major limitations and key assumptions. TechnoLynx treats such evidence as an engineering output in its own right, built so that a non-engineer on a model-risk committee can read it and sign against it. It is not regulatory sign-off.
Redaction has a real cost. An auditor who cannot see the internals can confirm the statistics but cannot fully verify why a divergence happened. That is where effective challenge comes in. Under SR 26-2 this means review by people with the expertise, independence and standing to force change. The question of who counts as that party is covered in our piece on what counts as independent model validation across industries.
What error tolerance should a crisis backtest commit to?
The guidance cited here leaves the number for a crisis-loss backtest to the institution. The tolerance is a policy choice the institution sets, approves and documents before the run. SR 11-7 asked model risk policies to set targets for model accuracy and standards for acceptable levels of discrepancy. It said outcomes that consistently fall outside the bank’s predetermined thresholds call for adjustment, recalibration or redevelopment. The PRA’s SS1/23 puts it at board level: the model risk appetite should include thresholds for acceptable model performance and tolerance for errors, and firms need clear guidelines for a maximum tolerance on performance exceptions. The ECB’s internal-models guide expects quantitative validation thresholds whose breach triggers investigation.
For daily 99% VaR, the Basel zones give an externally fixed tolerance. For crisis aggregates, scenario losses or cashflow engines, the committee has to write its own. A tolerance agreed after the results are in is not a tolerance. Framing the margin as a pre-registered non-inferiority test against a benchmark is one way to make it defensible, and our non-inferiority testing framework for backtesting risk models sets that approach out.
Committing the tolerance before the run
- Name the metric per crisis: aggregate loss, loss at chosen quantiles, or exception counts by segment.
- Set the numerical band, in basis points or currency units, and state whether it is one-sided against under-prediction.
- Fix the segment list and the multiple-comparison treatment.
- Write the attribution categories for divergences and the evidence each requires.
- Get committee approval and record the date, before any crisis output is produced.
- Define what a breach triggers: recalibration, post-model adjustment or redevelopment, each with independent review.
Has SR 11-7 been replaced by SR 26-2?
SR 26-2 sets out outcomes analysis as an element of the validation process. It compares model outputs with real-world outcomes, and material errors or persistent deviations outside the organisation’s established performance thresholds may warrant adjustment, recalibration or redevelopment. It names back-testing as one standalone form of outcomes analysis, and it lists out-of-sample and out-of-time testing among model testing activities. Both letters refer to the institution’s own thresholds. The newer one describes validation as informing judgements about acceptable performance ranges.
Total cashflow, or the timing of every waterfall step?
For engines that project cashflows through a securitisation waterfall, the committee has to say explicitly whether only the total must match or each step’s timing as well. A total-only criterion says nothing about the timing of individual waterfall steps. Treat this criterion as the committee’s call, recorded with its reasoning.
Who can build a crisis backtesting pipeline for our risk engine?
TechnoLynx builds the backtesting pipeline for a bank’s own risk or stress-test engine as part of its AI engineering for finance work. The engagement runs as Generative & Agentic AI R&D engineering. That covers framing the metrics, bounds and segmentation; implementing the backtest against 2008 and 2020 data; making repeated runs across scenarios fast on your infrastructure; and checking the implementation before it goes into validation operations.
The engagement does not set your tolerance, approve your model or stand in for second-line validation: having built the pipeline, we are not independent of it. It is not regulatory or legal advice. If your quant team already has a validated test library and only lacks capacity, building in-house is the shorter route; a specialist earns its place where the pipeline, the performance work and the evidence trail all have to be built from scratch. When choosing a partner, ask to see how they pin data and test versions, not just which tests they run. Teams that only need standard VaR tests run on existing data can use packaged tooling. MathWorks’ Risk Management Toolbox, for example, supports binomial, traffic-light, Kupiec, Christoffersen and Haas tests. Effective challenge has to come from experts with sufficient independence and standing to effect change.
What a pass against two crises can and cannot prove
A pass shows that the engine was consistent with two episodes under a tolerance you chose. It does not show that the engine is right. The tests are weak at one-year sample sizes: unconditional coverage tests have low power to detect systematic under-reporting. With an 8-violation threshold over a year, the chance of catching a model that reports the 3% VaR instead of the 1% is only 65%. Most VaR backtests also show heavy size distortions in small samples. The Christoffersen independence p-value is highly discrete, because one violation followed directly by another is rare. The study’s authors argue that test size should be screened before a test is used at all.
Expected Shortfall makes the job harder. Basel replaced VaR with ES as the standard market risk measure, and because ES is not elicitable, backtesting it is substantially harder than backtesting VaR. Then there is the sample itself. In 2008 liquidity and trading volume dried up, while in 2020 trading volume rose. Matching both says little about a third crisis built differently. History gives a warning on scale: in an Oliver Wyman comparison cited by Wharton researchers, 25% of 16 institutions lost at least 150% of their pre-crisis economic capital estimates.
So the question that matters is not whether the engine passed. It is what the firm will do when the next exceptions arrive and it has to decide quickly whether they mean a broken model or a new kind of crisis.
FAQ
What is value at risk in finance?
Value at risk (VaR) is an estimate of a specified percentile of the loss distribution for a trading position over a fixed holding period. For regulatory capital, the 99th percentile 10-day VaR is used. A backtesting exception occurs when the daily loss exceeds the VaR estimate.
What is risk tolerance in the context of model risk management?
In model risk management, risk tolerance is part of the board-approved model risk appetite. It sets thresholds for acceptable model performance and tolerance for errors, together with limits on model use and exceptions.
