Extreme events such as storms, heatwaves and record extremes are rare and contribute little to averaged skill scores. That lets a forecast score well overall and still fail on rare, high-impact events. Forecast uncertainty quantification should be tested where it matters: with threshold-weighted proper scores, calibration checked within subsets, and tail-calibration checks. Only a load or renewable forecast that passes those tests should set reserve quantiles or trading offers.
How do you stop averaged skill scores hiding forecast failure during storms or heatwaves?
Report skill separately for each weather regime and for a dedicated held-out set of extreme-weather cases, next to the global average and never in place of it. The global number summarises performance over the entire test set. The per-regime numbers tell you whether you can lean on it during the events that drive imbalance and reserve costs.
LynxBenchAI benchmarks throughput, latency, memory, energy and cost per unit of work across accelerators, informing the choice of hardware that runs these backtests. It measures the hardware, not the forecast skill.
TechnoLynx designs and implements regime-conditioned, walk-forward backtesting suites for forecasting pipelines. The first step is to agree with the operations and trading teams which regimes (storms, heatwaves, cold snaps, high-wind ramps) and which metrics matter for their risk tolerance. From there the work covers curating the held-out extreme cases, setting up stratified metric reporting on your existing training and inference stack, and checking the stratified results against the aggregate scores to confirm the protocol surfaces failures the averages missed. The suite is then left running so that every new model version is checked against the extreme-event set.
Pooled skill scores and the rare events they bury
When skill is pooled across samples whose event climatological frequency varies, scores can report fictitiously high skill. This is a form of Simpson’s paradox, and it affects threat scores, the Brier skill score and the relative operating characteristic. A 2025 GMD paper makes the mechanism explicit: an overall skill score breaks down into subset contributions, each the product of the subset’s own skill, its frequency weight and a reference weight. Small subsets contribute less to the overall score simply because they are small. In that paper’s decadal temperature example, the total RPSS was about 0.1. Underneath it, one climate regime scored 0.3 and the other regimes showed little or no improvement over the reference.
Rare events are a small subset, so the forecast’s performance on them barely moves the global number. A 2025 study of extreme precipitation forecasts found that mean CRPS was dominated by non-extreme amounts because those amounts are far more frequent. That study also cites an earlier result (Loveday et al., 2024) in which standard mean squared error showed no significant difference between meteorologists and automated guidance, while a threshold-weighted version showed the meteorologists did better on temperature extremes.
In the 2021 Pacific Northwest heatwave, beyond a week of lead time, the machine learning models tested performed up to at least three times worse than their summer 2022 baseline, against up to two times worse for ECMWF’s HRES. For Storm Ciarán, the machine learning models forecast peak winds of 25–26 m/s against 30 m/s in ERA5. The authors note that wind damage losses are often assumed to scale with the cube of normalised gust speed over a threshold, so even a small underestimate matters. A 2025 preprint on record-breaking events found that AI models underestimate both the frequency and the intensity of records, and that the error grows with the size of the record exceedance. The models behave as if their forecasts had a soft cap near the most extreme training values.
Building the extreme-event held-out set
Define “extreme” relative to local climatology, not as a fixed absolute value. ECMWF’s 2014 verification work found that a fixed 16 m/s wind threshold was exceeded mainly over sea, while a threshold relative to the model climate also gave signals over land. One study uses the 95th percentile of local climate, roughly a 20-year local return period. Another uses the 99th and 99.9th percentiles of six-hour accumulations and says plainly that the choice is somewhat subjective. The 95th percentile tends to capture mostly moderate extremes.
A high threshold targets rare events but leaves a small sample, while a low threshold gives steadier statistics but cannot separate extreme-event skill from general skill. Uncertainty in the score grows as the sample shrinks. Correlated forecast errors at multi-step lead times reduce the effective sample size further. A TechnoLynx protocol uses walk-forward backtesting with a curated held-out set of extreme cases. Per-regime scores need confidence intervals rather than a bare point value.
For scoring, weighted verification work recommends the threshold-weighted CRPS (twCRPS) or the vertically re-scaled CRPS when the interest is high-impact weather. twCRPS inserts a weight function into the CRPS integral, for example w(z) = 1(z > t) to concentrate on outcomes above a decision threshold t. It accounts for both the predicted probability of the event and its severity. The outcome-weighted variant judges severity only. Because twCRPS is a proper score applied to all cases, it avoids the selection bias that comes from conditioning on extreme observed or forecast events.
Why you cannot simply score the storms on their own
One shortcut is to compute ordinary scores only on the cases where a storm actually happened. That shortcut creates the forecaster’s dilemma. Lerch et al. (2017) put it bluntly: if evaluation proceeds conditionally on a catastrophic event having been observed, always predicting calamity becomes a worthwhile strategy. Gneiting and Ranjan (2011) showed that a proper scoring rule becomes improper when it is applied only to forecasts issued when particular outcomes occurred, and Bellier et al. (2017) noted that the same problem affects calibration checks. The record-extremes study therefore warns that an extremes-only score should not be the sole evaluation criterion.
Weighted scoring rules are the canonical way out. They emphasise the outcomes of interest while staying proper. A second safeguard is to report the overall score next to the conditional one, which is what the record-extremes study did to show that no model was hedging its extreme forecasts.
What published GraphCast evaluations do and do not certify
Independent evaluations of GraphCast reach different conclusions on different extremes. On record-breaking heat, cold and wind, HRES consistently outperformed GraphCast, Pangu-Weather and FuXi. On extreme precipitation, GraphCast-GFS achieved better point-to-point twCRPS than the high-resolution HRRR model, yet HRRR still outperformed it at short lead times, and rankings shifted with neighbourhood size. Each of these GraphCast evaluations describes one architecture, one training set and one set of verification choices. It does not certify a fine-tuned emulator, a different region, or the power-conversion step that turns weather into megawatts, so your own extreme-event held-out set is still needed.
Do your day-ahead bands contain reality as often as they claim?
A coverage backtest answers this directly. It counts how often the realised load or generation falls inside each stated prediction interval, a quantity called the prediction interval coverage probability (PICP). For a well-calibrated forecast, the observed coverage of a P% interval should equal P within the sampling uncertainty of the dataset. If reality lands inside the band more often than stated, the band is too wide. If it lands inside less often, the band is overconfident (too narrow) and has to be widened or the model retrained. Testing several interval levels at once is a much more stringent check than testing a single variance figure.
The full-distribution version of the same check uses the probability integral transform (PIT): evaluate the forecast CDF at each outcome, and for a calibrated forecast those values are uniformly distributed. Increasing, decreasing, hump-shaped and U-shaped PIT densities correspond to the typical bias and dispersion errors.
Average calibration is not enough
A band can pass on the full validation set and still be unreliable in subsets. Pernot (2022) separates average calibration from tightness, which is calibration checked within subsets of the validation set, and finds that average calibration does not guarantee reliable individual uncertainties. Running the coverage test inside groups along a relevant feature turns it into a local calibration diagnostic. In one synthetic example, the local analysis showed small uncertainties underestimated by a factor of about 2.0 and large ones overestimated by about 2.3. For wind power, the natural groups are the level of predicted generation and wind direction, since these change the shape of the predictive densities. PNNL found that wind forecast error distributions differ by forecast level: near-normal at low wind, but biased at high wind, because actual output cannot exceed installed capacity.
The tails need their own test, since standard calibration does not guarantee tail calibration. Two post-processing methods for European precipitation produced forecasts with very similar average CRPS, the logistic one roughly 1% better. Yet only the censored GEV version was calibrated in the tails, while the logistic forecasts had tails that were too light and were unreliable for extremes. Standard evaluation would have passed both. The same pattern shows up in post-processed Swiss temperature forecasts that were well calibrated overall but not conditionally calibrated on extreme heat, and, in Pernot’s chemistry study, in two models that reported an identical PICP of 0.995 despite very different behaviour.
The occurrence ratio compares observed threshold exceedances with the forecast’s predicted exceedance probability. A ratio above one means the forecast underestimates exceedances. A binomial test on that ratio and a Kolmogorov–Smirnov test on the excess PIT values check, separately, whether exceedance is forecast correctly and whether excess PIT values are uniform. Conditional PIT histograms check calibration given that a threshold was exceeded. Because they only assess severity, they should be accompanied by a standard reliability diagram for the probability of occurrence. For ensembles, the authors advise conditional checks only when a reasonable number of members, around ten, are expected to exceed the threshold.
Calibration drifts with the seasons
A one-off backtest misses calibration that changes over time. In a German weather-station study, the cumulative evidence against calibration built up in the first half of each calendar year but not the second. In another case, the full-period PIT histogram looked close to uniform while sequential testing had found strong evidence against calibration earlier in the record. Sequential e-value tests let a forecaster reject calibration as soon as the evidence crosses a threshold, without fixing the sample size in advance. That makes them a natural fit for the continuous checks described in building model risk management into the ML lifecycle.
A tightened band is only usable for decisions if it is still calibrated. The aim of a density forecast is to be as sharp as possible subject to calibration. In one net-load study, quantile regression gave the sharpest extreme intervals, but those intervals were not calibrated and so were not usable.
Which test answers which question
| Question about the forecast | Test | What a failure looks like |
|---|---|---|
| Do the bands hold on average? | PICP at several levels; PIT histogram | Coverage off nominal; non-uniform PIT |
| Do they hold in each regime? | Local coverage (LCP) within groups along a relevant feature | Group intervals not overlapping the target |
| Do they hold in the tails? | Occurrence ratio plus excess-PIT test; conditional PIT with reliability diagram | Ratio far from one; non-uniform excess PIT |
| Has calibration drifted? | Sequential e-values on PIT | Cumulative e-value rising past the rejection level |
| Which model is better on extremes? | twCRPS above a decision threshold | Higher (worse) twCRPS than the competing model above the threshold |
| Is the forecast worth acting on? | Potential economic value across cost-loss ratios, never alone, since it is hedgeable | Value at or below climatology |
From probabilistic forecasting to reserve and trading decisions
We treat two jobs here as separate analyses. One looks back and attributes reserve cost to forecast miscalibration rather than to real variability. The other looks forward and gives the trading desk a calibrated probabilistic forecast with an expected-cost number attached.
TechnoLynx’s engagement starts with the trading and operations stakeholders. Together they pin down the decision the probabilistic forecast must support and the calibration and latency limits it has to meet. The work then chooses a technique such as quantile regression or a CRPS-optimised model, implements it on the target hardware at the required refresh cadence, and validates both calibration and the expected-cost translation against historical imbalance outcomes. The final step is putting the probabilistic feed and the cost metric into the desk’s day-to-day workflow.
Putting a money figure on reserve held against forecast error
Dispatch and commitment procedures use only the mean values of load and renewable forecasts, despite their uncertainty. PNNL notes that this leaves it unclear what extra balancing effort and cost will be needed closer to real time. Reserve requirements are commonly set from quantiles of historical forecast deviations at a chosen confidence level. That points to an attribution method. Rerun the reserve rule on the error distribution of the current forecast and on a better-calibrated alternative, and price the volume difference. We treat the gap between the two as the reserve cost the better-calibrated forecast would save.
In a robust-dispatch test system, shrinking the uncertainty set from 1 to 0.8 reduced total reserve credits from around $2,300 to $1,000 and expected operating cost from $194,000 to $192,000. The same study found that the volume of reserve that actually carries value does not change monotonically with the uncertainty level. In a net-load study, calibrated conditional tail models reduced the overall reserve volume needed to manage a given risk, compared with unconditional tails. So the honest answer to “what would tightening the bands by X% save?” is to rerun the reserve rule at the narrower quantiles and price the difference. That answer only holds if the narrower bands pass the calibration tests above.
Worked example: upward reserve from calibrated quantiles
An illustrative example in a net-load forecasting study, based on current GB practice, sets reserve to meet the system operator’s risk appetite. Suppose the risk appetite is that reserve should fall short in no more than 4 hours per year. A year has 8,760 hours, so the target probability is 4 / 8,760 ≈ 0.00046. Rounded, that is the α = 0.0005 quantile of net load for upward reserve (and α = 0.9995 for downward).
If the system is scheduled on the median forecast, the upward reserve requirement for each 30-minute period is q0.5 − q0.0005. For a risk-neutral operator, the relevant probability (the ratio of marginal reserve cost to the value of lost load) typically falls between 0.01% and 0.25%.
The catch is that those quantiles sit deep in the tail. In the same study, quantiles below 1% and above 99% from quantile regression alone were not calibrated for any region, while quantile regression combined with generalised Pareto tails was consistent with calibration. The reserve figure is therefore only as good as the tail-calibration test behind q0.0005.
Other methods build the margin by convolving the load, generation-outage and wind-error densities. Another family chooses between recursive, anticipative and hybrid use of historical and forecast information, with the CAISO study finding the hybrid gave the lowest risk. PNNL’s histogram-based approach sizes capacity and ramping envelopes at chosen confidence levels.
What the trading desk needs instead of a padded point forecast
A point forecast is, in Pinson’s phrase, roughly “just the mean of whatever may happen”, and it is not the best input to many trading decisions. When balancing prices are asymmetric, the optimal wind offer is a specific quantile of the predictive distribution. Its level is π↓ / (π↓ + π↑), set by the predicted downward and upward regulation unit costs for that market period. That is the direct replacement for blanket padding: the desk offers a quantile chosen by price asymmetry instead of a mean with a fixed margin.
Which forecast the desk needs depends on the decision. One analysis found that market scheduling needs only an expected price forecast in almost all cases but often needs marginal distributions of renewable output and demand. Bidding-curve optimisation needs joint price and residual-demand distributions, and storage needs the full temporal joint distribution. Scheduling tools such as PLEXOS cannot take a probabilistic forecast directly, so it has to be converted into scenarios first.
Turning uncertainty into an expected imbalance cost per schedule works as follows. For each settlement period, combine the predictive distribution of your deviation from the contracted volume with the imbalance price that would apply in each direction, and sum the expected penalty over the schedule. Under single-price balancing this needs one more input. The price penalises participants who are out of balance in the same direction as the system and rewards those in the opposite direction. So the expected cost depends on a probabilistic forecast of system length as well as on your own power forecast. In a GB study, strategies using a probabilistic system-length forecast earned more than strategies that tried to minimise imbalance volume, while a deterministic system-length forecast did not.
Treat the imbalance-price side with care. German imbalance prices between 2018 and 2021 ranged from about −6,500 to 24,500 EUR/MWh. In that study, a naive intraday-price benchmark had the lowest CRPS, but its empirical coverage was very poor. The authors concluded that balance-responsible parties should minimise imbalance rather than chase balancing-market opportunities. Updating decisions on a dynamic probabilistic forecast gave a 5% revenue increase over point forecasts in simulation. One limit remains: how improvements in calibration or skill scores translate into decision value is still an open question, especially when forecasts are used sub-optimally.
Is it the wind, solar or load model that is wrong?
Splitting forecast error by source requires separate solar, wind and load forecasts and actuals. A generic forecasting deployment that reports one blended error cannot be decomposed after the fact without access to its intermediate outputs. The building blocks exist: system margin uncertainty can be built from separate load-error, generation and wind-error densities, and wind error behaves differently at different forecast levels.
TechnoLynx frames the decision the diagnostics must support, then checks whether the existing vendor deployment already has per-source diagnostics. Where they do not, the next step is to specify the EMS export and build a decomposed probabilistic pipeline under a Generative & Agentic AI R&D engagement.
Who can build regime-aware uncertainty testing into our load and renewable forecasts?
TechnoLynx builds regime-conditioned and extreme-event backtests, calibration validation, and the translation of forecasts into expected imbalance cost for the trading desk. On building in-house versus bringing in a specialist: a team that already runs probabilistic models and has clean forecast-actual history may only need the testing protocol and the decision wiring. A team starting from point forecasts needs the models as well, and a TechnoLynx engagement covers both. When choosing a partner, ask whether they will report per-regime and tail results that could make their own model look worse. For the sector context, see AI engineering for energy and utilities.
This work does not suit every buyer. It is engineering, not regulatory approval of a reserve methodology. It also does not suit a buyer who simply wants a licensed forecast feed. One alternative is an in-house team using open tooling such as the scores Python package, which includes twCRPS and the Diebold–Mariano test.
FAQ
Is a prediction interval the same as a confidence interval?
No. A confidence interval describes uncertainty about an estimated parameter, such as a mean. A prediction interval describes where a future observation, such as tomorrow’s load, is expected to fall. Forecast bands for reserve and trading are prediction intervals, and their coverage is what gets backtested.
Why use a 95% interval instead of 99%?
Higher levels sit in the tails, where data are sparse and calibration is harder to check. Quantile regression in one net-load study was not calibrated below 1% or above 99% without special tail models. Reserve decisions need far more extreme quantiles than a 95% interval provides, so the tail has to be modelled and tested explicitly.
What are imbalance costs?
Imbalance costs are the charges a balance-responsible party pays when delivered energy deviates from its contracted position, settled at the balancing price. Under single-price balancing, deviations in the same direction as the system are penalised and deviations in the opposite direction are rewarded.
