No. Scoring a single deterministic forecast trace with CRPS against ECMWF’s 50-member ENS is not a fair comparison, because for a point forecast the CRPS reduces to mean absolute error. The emulator is penalised for not having a distribution it was never built to produce, and the comparison tells you little about the emulator itself. To make it fair, first make the emulator ensemble-equivalent: either build a pseudo-ensemble from it (lagged runs, perturbed initial conditions or stochastic sampling) or post-process its output into a predictive distribution and score that. Then score both systems on one grid, at matched lead and initialisation times, and against one fixed verification reference.
Is it fair to score a deterministic emulator with CRPS against a 50-member ensemble like ENS?
Not as it stands. A raw deterministic CRPS is just an absolute error, while an ensemble’s CRPS also subtracts a pairwise member-spread term. Published approaches instead make the deterministic output probabilistic first, via lagged ensembles, post-processing or perturbed initial conditions.
TechnoLynx does this work for teams verifying a weather emulator. TechnoLynx agrees with you what “fair” means for your decision use, lead times and variables, then chooses between a pseudo-ensemble and ensemble-equivalent scoring to suit your emulator and compute budget. TechnoLynx then builds and optimises the batched perturbed-run inference or the scoring pipeline on your hardware, checks spread calibration against ENS on held-out cases, and leaves it running as part of ongoing evaluation. TechnoLynx’s AI engineering for weather and climate work starts with that framing step.
LynxBenchAI publishes hardware measurements that help you choose accelerators for running many perturbed emulator rollouts per start date, and the first step is to look up its measurements for the hardware you are considering, though it does not run your emulator.
What an ensemble tells you that one trace cannot
An ensemble runs a set of forecasts from perturbed initial conditions, and sometimes perturbed model physics, so it describes a range of possible futures rather than one. That range is what lets an ensemble estimate the probability of extreme events that a single deterministic forecast might miss. ENS builds its 50 perturbed members from an ensemble of data assimilations plus singular-vector perturbations, and represents model uncertainty with stochastically perturbed parameterisation tendencies.
A deterministic emulator trained on mean squared error deals with that uncertainty differently: it hedges by predicting something close to the mean of the possible outcomes, which makes its forecasts unrealistically smooth as lead time grows. The GraphCast authors describe the same effect, noting that their MSE objective makes the model express uncertainty by spatially blurring its predictions. This is why deterministic scores alone mislead. Brenowitz et al. (2025) point out, after Leith (1974), that the RMSE of a perfect deterministic model saturates at a value √2 higher than that of a perfect ensemble, so any model claiming better deterministic skill at long lead times does so by artificially reducing the forecast’s variance.
How CRPS is calculated for an ensemble, and what it rewards
| For an M-member ensemble x₁…x_M and a verifying value y, the CRPS is (1/M)·Σ | x_j − y | − (1/(2M²))·ΣΣ | x_j − x_k | . The first term, an error term, rewards members that sit close to the outcome. The second, a pairwise member-distance term, rewards appropriate dispersion. Gneiting and Raftery (2007) show the CRPS is a proper scoring rule, reported in the units of the variable, that generalises absolute error. Its aim is the general goal of probabilistic forecasting: predictive distributions as sharp as possible while still calibrated. |
| Worked example (toy numbers, chosen to show the mechanism). A deterministic forecast of 1 scores | 1 − 2 | = 1, which is its absolute error. A two-member ensemble {1, 3} has a first term of (1 + 1)/2 = 1 and a pairwise sum of 4. With the standard estimator, 4/(2·2²) = 0.5, so the CRPS is 0.5. With the fair estimator, which divides by 2M(M − 1) instead, 4/(2·2·1) = 1, so the CRPS is 0. The ensemble that brackets the outcome is rewarded for its spread, which a single trace cannot earn. The example also shows how much the choice of estimator matters at small M. |
That estimator choice is worth stating in any comparison. The fair CRPS adjusts for ensemble size, and the common Python packages xskillscore and properscoring use the biased version. GenCast deliberately kept the traditional estimator because it was scoring against finite NWP ensembles, while the HealDA authors report a fair finite-ensemble CRPS. AIFS-CRPS uses an “almost fair” blend of the two that avoids a degeneracy of the fair score. Keep in mind that CRPS is strictly proper only for scalar marginals: averaged over grid points it says nothing about whether the joint spatial structure is right.
Three ways to make a deterministic emulator into a pseudo-ensemble
Lagged ensembles from hindcasts
A lagged ensemble treats deterministic forecasts started within a time window as members, exploiting the fact that runs started at different times diverge much as perturbed runs do. Brenowitz et al. use a centred M-lagged ensemble with 2M + 1 members whose start times straddle the target initial time. Because it needs initial conditions from the future, it can be used for scoring hindcasts but not for real forecasts. Their configuration is 9 members (M = 4) at 12-hour spacing, a centred two-day window, and they report that the conclusions are not sensitive to that choice.
The method needs no parameters and works the same way for every model, which is its main attraction for comparing models with each other. Applied to IFS, the lagged ensemble’s CRPS at day 5 correlates well with operational ENS from one day to the next. The lagged ensemble also scores better on average, because some of its members have shorter lead times. Lagged members are also not interchangeable, since more recent members are more skilful, which makes spread-error calibration ambiguous. Brenowitz et al. handle this by using the lagged ensemble of IFS as the baseline. Under lagged scoring, GraphCast and Pangu-Weather come out roughly level even though GraphCast leads on deterministic scores.
Perturbed initial conditions
The emulator is run many times from slightly different initial states. Bülte et al. compare three ways to make those states: Gaussian noise added to the initial state, the perturbed initial conditions of a physics-based ensemble, and random-field perturbations built from scaled differences between past atmospheric states. On Pangu-Weather over Europe, random-field perturbations performed similarly to the operational ECMWF ensemble overall, while Gaussian noise and IFS-perturbed initial conditions fell short. More elaborate schemes exist. The Huge Ensembles work combines bred vectors, scaled to 0.35 times the emulator’s 48-hour deterministic RMSE (a tuning parameter), with 29 trained checkpoints to make 58 members. GenCast’s baseline, GenCast-Perturbed, initialises a deterministic model from ERA5 perturbed by Gaussian-process noise. Every member is a full run of the emulator.
Stochastic generation
Stochasticity can also be injected inside the network. Spherical DYffusion, for example, enables MC dropout and stochastic depth at inference time. Noise injection trained with a CRPS loss is the other main route, and the study Reliability of Probabilistic Emulation of Physical Systems finds it generally gives more reliable coverage than latent-space generative models. Note what this means for your comparison: once the model has been retrained with CRPS, you are no longer scoring the deterministic emulator you started with.
Ensemble-equivalent scoring without new members
Potential CRPS (PC) takes another route. Every deterministic model, physics-based or learned, goes through the same post hoc isotonic distributional regression, using EasyUQ, and PC is the mean CRPS of the resulting predictive distributions. EasyUQ needs no tuning and uses only the model output at the lead time being scored. For IFS HRES, PC tracks the mean CRPS of operational ENS closely. It is slightly optimistic at day one and pessimistic at days seven and ten. It needs evaluation sets of hundreds to a few thousand cases per grid point; the authors used about 730. Post-hoc methods also produce a separate distribution at each grid point, so they give up the spatially coherent fields that perturbed runs produce.
Choosing a route: the apples-to-apples convention by case
| Route | What it needs | Fits when | What it cannot show |
|---|---|---|---|
| Centred lagged ensemble | An archive of deterministic hindcasts at regular start times | Ranking deterministic models against each other and against an IFS lagged baseline, cheaply | Real-time forecast skill; clean 1:1 spread-error calibration, since members are not interchangeable |
| Perturbed initial conditions | Many full rollouts per start date and a perturbation scheme | Compute is available and you want a pseudo-ensemble that behaves like an operational system | Anything trustworthy about spread until the scheme is validated; input perturbation often under-disperses |
| Stochastic sampling or CRPS retraining | Changes to the model or its training | You are building a probabilistic product, not judging the existing emulator | How the original deterministic emulator performs |
| Potential CRPS (IDR/EasyUQ) | Hundreds to thousands of forecast-outcome pairs per point | You need a tuning-free proxy for ENS-level CRPS without generating members | Spatially coherent fields, since it gives a separate distribution per grid point |
Whatever row you pick, one rule holds: never put raw deterministic CRPS next to full-ensemble CRPS without one of these adjustments. Brenowitz et al. add a caveat that cuts the other way. Lagged scoring can give deterministic models an unfair advantage over models that solve the harder end-to-end probabilistic problem, so final evaluations of end-to-end ensemble systems should still use traditional ensemble scoring.
Comparing pseudo-ensemble spread with ENS spread
The spread/skill ratio should be close to 1. Values below 1 mean the ensemble is under-dispersed and overconfident, and values above 1 mean it is over-dispersed. Rank histograms are the other common calibration tool. For small ensembles, include the √((E+1)/E) correction factor; WeatherBench 2 omits it. Match ensemble sizes where you can. GenCast scored every model with 50 members because ENS has 50. The Huge Ensembles authors compared 58 members against 50 and argued that the size difference was small enough not to change the conclusions. Ebert’s ECMWF training material notes that how robust ensemble verification is depends on both the ensemble size and the number of verification samples.
Spread diagnostics on a pseudo-ensemble can mislead in two specific ways. Ad hoc perturbations of deterministic models produce ensembles that inherit the blurring. Scoring against a single deterministic analysis also rewards under-dispersion at short lead times. A third failure is invisible to CRPS altogether. In Lorenz-96 experiments, a deterministic emulator run from perturbed initial conditions had a variogram-to-CRPS skill ratio of about 0.42 to 0.58, against 1.02 for a dynamical ensemble, or 0.91 at matched spread: it gets the marginals roughly right and the joint structure wrong. The energy score largely misses this and the variogram score catches it, so report a variogram score alongside CRPS whenever spatial structure matters for how the forecast is used.
A verification framework with regridding, lead-time alignment and one reference
A fair comparison of a GraphCast-style emulator with ENS needs a common, consistent verification setup. Both systems are regridded the same way onto a shared reference grid, aligned on lead times and initialisation times, and scored against the same reference, with independent observations as a cross-check. Without this, the verification setup itself can pick the winner.
LynxBenchAI publishes hardware measurements that inform which accelerators to buy for running the emulator rollouts this framework scores, so a sensible first step is to compare its measurements for your candidate hardware, bearing in mind it does not run your emulator.
TechnoLynx builds this verification framework as a reusable pipeline. TechnoLynx confirms the target grids, lead times, variables and references with you, then implements the regridding and alignment step (for example with xESMF or CDO remapping) so it is applied identically to both systems. The pipeline is then parallelised for your data volumes and checked for neutrality across regridding methods before any scores come out of it.
Regridding GraphCast-style and IFS output without introducing bias
Regrid down to the coarsest resolution. Brenowitz et al. say models of different resolution should be downsampled to the coarsest available resolution. WeatherBench 2 regrids every forecast and every ground truth first-order conservatively to 1.5°, the WMO and ECMWF standard verification resolution, so coarser models are not penalised. GenCast regridded ENS from its native 0.2° to 0.25° and then subsampled both forecasts and targets to 1°. HealDA bilinearly interpolates onto a finer HEALPix grid and then block-averages down to HPX64, using the same procedure for the ERA5 reference to avoid aliasing.
The method matters as well as the target grid. ECMWF’s MIR documentation notes that its default linear interpolation, built on a triangular mesh, replaced the older four-point bilinear method and can give different fields. The study “The Perils of Regridding” compares how conservative, bilinear and distance-weighted regridding affect precipitation quantiles and the fraction of zero-precipitation points. The defensible choice is to apply the same regridding procedure to both systems and to the reference.
Lead times, initialisation times and look-ahead
The GraphCast authors matched data-assimilation look-ahead by scoring their model only from 06z and 18z initialisations, at 12-hour target multiples, so that ERA5 and HRES inputs carried the same +3 h of future observations. When ENS is only available from 00z/12z, GenCast linearly interpolated the ENS CRPS curve onto the same targets. They then showed that this interpolation overestimates ENS at short lead times. The stakes are not small. The HealDA authors show that small changes in the verification setup can shift apparent skill by 12 to 24 hours, about the size of difference needed to claim a state-of-the-art result.
Should we verify against analysis, radiosondes, or both?
Use one fixed reference for every headline CRPS number, and cross-check against observations before claiming a winner. A score like CRPS is proper only with respect to a single verification reference held fixed across all systems. Score each system against a different reference and a lower CRPS no longer means a more accurate forecast. The HealDA authors show this can move scores by a day in favour of whichever forecast is scored against a blurrier analysis, usually the ML model’s.
The sources disagree on the remedy. GenCast and GraphCast score each model against its own analysis, HRES-fc0 for ECMWF and ERA5 for the ML models, because verifying a model against its own analysis gives it the best skill estimate. WeatherBench 2 does the same for IFS so that it is not penalised by a non-zero error at t = 0. HealDA argues instead that headline comparisons should use a common, fixed, non-ML reference, with own-analysis scores kept as supplementary diagnostics. Because CRPS is proper only against one fixed reference, HealDA’s case for a common reference applies to headline CRPS comparisons. A middle route is the one Walz et al. suggest for potential CRPS: compute the score once against each analysis and, if the rankings disagree, test for significance.
Radiosondes can reverse a ranking. ECMWF found AIFS worse than IFS at 100 hPa for temperature when verified against analyses, but better when verified against radiosondes. A study of verification against in-situ observations argues that ERA5-based scoring gives data-driven models an unfair advantage, while observations judge both approaches without favouring either. Radiosondes are sparse and concentrated over land, though, so they corroborate global analysis-based verification rather than replace it.
Reduce ENS to its mean, or score it as an ensemble?
State which you did, because it changes the result. The ENS mean minimises RMSE, but it is not a plausible state of the atmosphere. In WeatherBench 2, deterministic models beat the ENS mean up to 3 to 6 days, after which the mean has the lowest error. Published comparisons use both: the emulator against the ENS mean on deterministic metrics, and an ensemble against ENS on CRPS and spread-error.
Who can build a fair ENS verification pipeline for our weather emulator?
TechnoLynx builds the pipeline that makes a deterministic weather emulator comparable with ENS: pseudo-ensemble generation or potential-CRPS scoring, a symmetric regridding and lead-time alignment step, and calibration checks you can rerun on every new model version. Building it in-house makes sense if your team already owns verification methodology and only needs the engineering. A specialist is worth bringing in when batched rollouts, data volumes or reproducibility are the bottleneck. Choose a partner by asking how they would show that their regridding is neutral and their spread is calibrated, not by the scores they promise. TechnoLynx’s weather and climate engineering practice covers the build.
Teams that only need regridding can use the tool WeatherBench 2 already provides.
What a fair score still leaves open
A fair CRPS tells you how good the emulator’s marginal distributions could be, given an agreed way of adding uncertainty to it. It does not tell you that the emulator would make a good ensemble member in an operational system. The lagged-ensemble results make the point: deterministic rank and ensemble rank need not agree, and some deterministic-loss models outperformed IFS once the same ensemble technique was applied. The open question for your own comparison is which use the score is meant to inform. If that use is decisions about rare events, CRPS averaged over all grid points is the wrong place to stop, and a threshold-weighted CRPS or a variogram score belongs in the report next to it.
