Yes, benchmark the existing code, but don’t treat that benchmark as the whole study. A GPU NWP benchmark of IFS or WRF at FP32 or mixed precision against the FP64 baseline tells you how much faster your current physics can run. It can’t tell you whether an AI emulator would serve your flash-flood or severe-storm warnings better. Make the choice between the two routes in a feasibility study scored on your own lead times, extreme-event skill, latency, cost and operational risk, not on a generic leaderboard or another agency’s precedent.
TechnoLynx runs that study as an engineering engagement. GPU Performance Engineering starts by recording your nowcasting requirements (phenomena, lead time, accuracy thresholds, latency budget, cost and risk). It then benchmarks your NWP code on the target GPUs against its FP64 baseline and prototypes the emulator path where the requirements justify it, as part of TechnoLynx’s AI engineering for weather and climate work.
NWP vs AI weather model: two different bets on the same warning problem
The two routes improve nowcasting in different ways. Speeding up the existing suite keeps the physics, data assimilation and verification history you already trust, and gives you throughput to spend on resolution, ensemble size or update frequency. ECMWF took that route when IFS Cycle 47r2 moved many high-resolution and ensemble calculations from double to single precision. The change had a neutral effect on forecast performance and freed about 40% of computing power for improvements. An AI emulator replaces part of the pipeline with a learned model. According to the WMO, a trained model often needs much less computing power to produce a forecast, but training and maintaining it still takes extensive data, infrastructure and specialised expertise.
No single benchmark result settles which bet to make. A throughput figure from a GPU port says nothing about skill gaps an emulator might close. A headline skill score for an emulator says nothing about how it behaves on your domain, your radar network or your warning thresholds. Other agencies made their decisions against their own baselines, hardware and warning products, so those decisions transfer only as far as the conditions match yours.
Decision criteria the review board can check
| Requirement | Points toward speeding up existing NWP | Points toward evaluating an AI emulator |
|---|---|---|
| Main goal | Match or modestly extend current skill within known physics | Large speedups, or a product current NWP cannot deliver even with more hardware |
| Where the bottleneck is | Time-to-solution of the current code on current hardware | Short-lead skill that more compute has not fixed |
| Lead time of the product | Longer lead times, where the NWP suite already performs | Nowcasts of up to about three hours, where two published learned nowcasters beat HRRR on skill scores |
| Record-breaking extremes | Physics-based models still outperform AI models on record-breaking events | Keep any emulator advisory until it has been verified on your own record cases |
| Uncertainty | An existing, calibrated ensemble | Most AI weather checkpoints are deterministic, so budget for an ensemble or an uncertainty method |
| Physical consistency | Conservation diagnostics already in place | Some AI outputs may be physically inconsistent, so you need plausibility checks |
Read the table as a weighting, not a scorecard. If your requirements are mostly about throughput on known physics, the acceleration path is the natural default. If they call for a short-lead product the current suite misses even with more hardware, the emulator path deserves a dedicated feasibility study. An HPC cost analysis on its own won’t show the capability gap an emulator might close, and an emulator’s skill score on its own won’t show what it costs to operate.
How do we benchmark IFS or WRF on GPUs at reduced precision against FP64?
Treat the FP64 run as the reference and decide in advance which components stay in FP64. Measure the time-to-solution of the full operational configuration across node counts. Accept the reduced-precision build only if its differences from FP64 sit inside the ensemble spread and it passes a significance-tested verification over enough forecasts.
For IFS, Váňa et al. (Monthly Weather Review, 2017) found no noticeable loss of accuracy and an average efficiency gain of about 40% when almost all real-number variables moved from 64 to 32 bits. ECMWF’s Newsletter 157 reports that the main remaining difference was a larger mass conservation error, which a global mass fixer mitigates. Even with the fixer, some ensemble scores degraded at the time of that report. ECMWF kept the 4D-Var minimisation and the tangent linear and adjoint models in double precision because that part of the system had shown strong sensitivity to rounding errors. Single precision also halves memory requirements, which matters most at very high resolution. Lang et al. (2021), as summarised by Banderier et al., validated single-precision IFS ensembles with CRPS against analyses and observations for 121 variables. They tested significance with a variance-inflated t-test, and runtime fell by about 40% without compromising skill. That study is a good template for the accuracy half of your benchmark.
Regional COSMO work shows similar savings, with some components kept in FP64. In COSMO, the radiation and soil models still run in double precision inside single-precision simulations because running them in single precision gives errors that are too large. Banderier et al. still measured about a 30% cost reduction. MeteoSwiss uses reduced precision operationally for the Swiss national forecast and saw no discernible impact on skill.
WRF speedups vary with the run configuration. A WRF GPU study on Perlmutter’s A100 nodes measured a 2.08x speedup for the CONUS-12km thunderstorm case at 16 MPI ranks. The speedup fell to 1.82x at 32 ranks and 1.56x at 64, and a separate two-node configuration ran at 0.956x (slower than baseline). The authors name low arithmetic intensity and strided memory access in the current spectral-bin microphysics code as a limitation. They also note that most WRF quantities are already single precision, and the GPU version kept 3 to 6 digits of state variables relative to the baseline build over a three-hour run. For WRF, then, most quantities are not FP64 to begin with, so agree with the board on which reference you mean before quoting a ratio.
Outside meteorology, a study of four turbulence solvers found that IEEE single precision could be used for an entire simulation with no significant discrepancies from double precision. For bandwidth-limited codes that is worth up to 2x. The same study found that computing geometric quantities in FP32 produced a non-watertight grid and nonphysical artefacts, which went away once those terms returned to FP64. Large FP32 dot products could also stagnate. The authors expect success to depend on mixed precision tightly coupled to the numerical method. A materials-science GPU code running its FFTs and distributed matrix multiplications in FP32 (with everything else in FP64) was up to 45% faster with results in good agreement. That transfers as a pattern, not a number. For matrix-multiply-heavy parts that must stay in FP64, emulation on low-precision Tensor Cores accelerated DGEMM by 2.3x on GB200 and 13.2x on RTX Pro 6000 Blackwell in favourable cases. The authors note that such schemes have not yet been adopted as defaults in production numerical libraries.
The benchmark procedure we would follow:
- Freeze the FP64 reference: same source revision, resolution, ensemble configuration and input data, with a record of which components already run below FP64.
- List the components that stay in FP64 before any run: data assimilation minimisation, adjoint code, any parameterisation known to be precision-sensitive, conservation-critical sums.
- Measure time-to-solution for the complete forecast at operational node counts, not a single kernel, because the speedup can shrink as ranks increase.
- Compare reduced-precision and FP64 differences against ensemble spread. ECMWF judged its single-precision track differences for Hurricane Laura considerably smaller than the operational ensemble range.
- Run a significance-tested verification (CRPS and the headline scores) over a sample large enough to detect the difference you care about, and add conservation diagnostics such as total mass.
Scoping the go/no-go study for flash-flood warning: parallel run or desk review?
Choose a parallel-run pilot when your timeline and risk tolerance allow a live comparison. Use a benchmark-and-literature desk study when they don’t, and treat its conclusion as provisional. A monsoon-season review board will trust a recommendation far more if the metrics, thresholds and event set were agreed before the pilot started (see the battery section below).
Others have faced the same framing. The Tennessee Valley Authority used NWP for nowcasting but struggled with missed detections, and it discontinued operational use of HRRR because the model added noise at early lead times and was inconsistent from run to run. After a 30-event study, the authors state that NowcastNet will be evaluated within TVA’s operational system. The WMO describes a Malawi project combining the Norwegian Meteorological Institute’s Bris model with ECMWF’s Forecast-in-a-Box, so that forecasters can compare AI forecasts with existing ones for early warnings.
Why can’t an AI weather emulator run on the CPU supercomputer we already have?
Strictly, it can run there: neural network inference executes on CPUs, just far more slowly than on accelerators built for it. The real issue is that the two workloads need different hardware. Modern GPUs put low-precision formats such as FP16, FP8 and FP4 first, and their Tensor Cores deliver orders of magnitude more throughput in those formats than traditional FP64 pipelines. Supercomputers are ranked on FP64 Linpack performance, while modern GPUs support several formats, each with its own performance. A machine sized and procured for double-precision physics is therefore a poor fit for a model that spends almost all its time in dense, low-precision matrix arithmetic.
LynxBenchAI publishes measured throughput and latency across accelerators, so the first step in choosing hardware for an emulator is to check its published results before any vendor demonstration.
LynxBenchAI’s published results give measured throughput and latency across accelerators, which helps shortlist devices for an emulator before a vendor demonstration. They measure the hardware, not your emulator.
Memory requirements differ too. In one evaluation, FourCastNetV2 ran on a 24 GB L4 GPU, while Pangu-Weather and Aurora needed a 40 GB A100 because of their higher memory requirements. Stormscope, a storm-scale nowcasting model, takes around 7 minutes on a single H100 to generate a 2-hour nowcast or a 12-hour nearcast at maximum fidelity, according to its developers. Training it took about 48 hours on 32 H100 GPUs.
For a general reader, an analogy that doesn’t mislead: the NWP supercomputer is a large team of accountants, each checking every figure to sixteen digits. The emulator needs a production line that does billions of rougher multiplications at once. Both do arithmetic, but you wouldn’t staff one job with the other’s team. Where the analogy stops: the emulator still depends on the physics-based system for its training data and initial conditions.
Comparing the cost of an HPC upgrade with an emulator inference capability
Pricing either path for your agency has to be built from your own quotes and staffing. The parts to cost differ. Single precision stretches existing hardware, as with the roughly 40% of computing power ECMWF freed. The emulator path adds training compute, extensive data and specialised expertise to the inference hardware. Inference itself is cheap: ECMWF reports about 1,000 times less energy per forecast for AIFS.
Procurement lead time and lock-in are judgement calls you should test against real quotes. Some emulators ship in a portable format: Pangu-Weather’s models are distributed in ONNX.
Which AI weather emulators are operational today, and where do they fall short?
AI emulators such as ECMWF’s AIFS are operational for global forecasting, alongside physics-based models rather than instead of them. They are still unproven as the primary system for short-lead severe convection and record-breaking extremes. ECMWF took AIFS into operations on 25 February 2025 to run side by side with IFS. AIFS has a 28 km grid spacing against IFS’s 9 km, and it starts from the same initial conditions. ECMWF reports gains of up to 20% on tropical cyclone tracks and describes the two systems as complementary.
TechnoLynx can run this check for a specific warning product. The engagement catalogues your severe-weather products, lead times and regional event set, then designs a parallel trial, porting and deploying the AI emulator on your target accelerated hardware next to the current NWP suite, using your own historical cases.
Published verification results differ by event type and lead time. A 2026 Science Advances study found that most AI models beat ECMWF’s HRES on overall 2 m temperature skill, yet physics-based models still outperformed them on record-breaking extremes. Comparing GraphCast-GFS with HRRR over 32 months of precipitation forecasts, one study found that HRRR had better discrimination for extremes at short lead times. GraphCast-GFS overtook it from around 24 hours. On Extreme Weather Bench’s severe convective days, the AI models outperformed HRES on CSI and false alarm ratio.
Observation-driven nowcasters are a different kind of model and tell a different story at very short range. AI emulators of convection-allowing NWP rely on the data assimilation of their physics-based counterparts, while Stormscope is trained on satellite and radar. Stormscope beat HRRR on deterministic scores up to 3 hours. Across 30 heavy-rain events in the TVA area, the physics-embedded NowcastNet reached a median CSI of 0.30 against HRRR’s 0.04 for grid cells above 16 mm/h. That same study notes that NowcastNet loses accuracy at 3 hours by overestimating precipitation, and it identifies information loss from 1 to 3 hours as a common challenge for deep-learning nowcasting. Neither nowcasting result has been replicated on your cases, so test them there.
Three caveats apply to all of this. First, date the literature review and refresh it before each decision. Second, “operational” can mean side-by-side running rather than sole reliance, as the AIFS case shows. Third, the WMO warns that AI models may perform less well in observation-sparse regions, during rare events, or under conditions different from their training data.
What verification scores belong in an evidence pack against operational IFS?
A credible pack compares the emulator with operational IFS (with ERA5 as the reference analysis) on probabilistic, spatial and extreme-event scores across a pre-agreed battery. Every input, script and result should be traceable. TechnoLynx builds that evaluation harness for a specific warning use case. It agrees lead times, thresholds and event definitions against the board’s expectations first, runs the emulator and baselines under matched conditions, then packages the harness so the pack can be re-run for each review.
LynxBenchAI gives measured throughput and latency across accelerators, so the first step for the latency part of the pack is to check its published results before choosing the inference hardware.
| Score | What it measures | Why it belongs in the pack |
|---|---|---|
| CRPS / CRPSS | CRPSS of 1 is perfect, 0 means no skill over climatology | The most common scoring rule for probabilistic forecasts, scoring the whole forecast distribution |
| Threshold-weighted CRPS (twCRPS) | Forecast behaviour above a chosen threshold, such as the climatological 99th percentile | Focuses the score on warning-relevant intensities |
| ACC | Pattern agreement of anomalies, typically 500 hPa height | Widely used for spatial fields and expected by boards, but it rewards smooth fields |
| CSI, ETS, POD, FAR, FSS | Categorical and neighbourhood skill at rain-rate thresholds | Used to compare NowcastNet and HRRR |
| SEDI and frequency bias | Skill and bias for rare, high-percentile events | ECMWF’s extreme-event verification of HRES and ENS |
| Coverage of prediction intervals | Whether uncertainty is calibrated (85–95% for a 90% interval) | Treated as a hard constraint before sharpness is compared |
Unweighted CRPS alone won’t isolate the result the board cares about. It assesses the forecast over all possible outcomes, so it measures overall performance rather than skill on high-impact events. Allen and colleagues recommend threshold-weighted and vertically re-scaled scoring rules when the interest is in high-impact weather. The GraphCast-GFS and HRRR study shows why both are needed. HRRR led on CRPS at neighbourhoods covering a similar area across all lead times, yet only beat GraphCast-GFS on extreme precipitation at short lead times.
Matching conditions is the other half of fairness. Model rankings in that study were sensitive to neighbourhood size, and spatial verification produced different rankings from point-to-point comparison. Fix grids, lead times, neighbourhoods and reference analyses in the protocol, or the scores won’t be comparable. Verification skill is also not the whole readiness case: latency, reliability and integration into the warning chain belong in the pack alongside CRPS and ACC.
How many extreme events does a fixed test battery need to separate two models?
The honest answer depends on how small a difference you need to detect. Geer’s ECMWF analysis shows that with 60 forecasts, a change in day-5 forecast quality must exceed 1% to be significant. Detecting a typical 0.5% change takes about 424 forecasts at 12-hour spacing. Those figures come from medium-range forecast scores, not extremes, so, at least for medium-range scores, a few dozen forecasts separate only large skill gaps. The 30-event TVA study reported a wide gap at 16 mm/h (median CSI 0.30 against 0.04).
Case selection criteria
Published benchmarks give workable case definitions. Extreme Weather Bench covers heat waves, major freezes, severe convective days, tropical cyclones and atmospheric rivers. It draws cases from 2020–2024 because those years are often not used in training AI global models. Its heat wave needs at least 3 consecutive days above the 85th percentile, and its atmospheric-river tracker uses an IVT threshold of 400 kg/m/s over at least 500 grid points. It also includes marginal temperature days. A 2026 uncertainty study used a near-complete census of 2021 EM-DAT disasters, giving 100 distinct initialisation dates set 3 days before onset, all outside every model’s training period. For flash-flood work, the two precipitation studies cited here define extremes by station percentiles (99th and 99.9th of six-hour totals from ERA5, 1990–2020) or by rain-rate thresholds of 16 and 32 mm/h.
Statistical power and significance
Sample forecasts densely rather than sparsely. Geer found that 12-hourly forecasts always detect changes more easily than less frequent ones, provided the confidence interval is inflated (k = 1.22 for 12-hour spacing, 1.07 for 24-hour) to account for autocorrelation. Every extra score you test raises the chance of a false result. Geer suggests picking in advance the set of scores on which the rigorous evaluation will rest. Stormscope’s developers used 360 dates spread over a year with paired t-tests. Run a power analysis on the expected skill gap before freezing the set, and enlarge it if the default can’t resolve that gap.
Freezing and hashing the set
Freeze the case list, event definitions, thresholds and scores in a manifest before any model output is generated, and record a cryptographic hash of that manifest in the study log. Any later edit changes the hash, so the board can confirm that nobody swapped events after seeing results. The hash proves the manifest was not altered. It doesn’t prove the selection was unbiased, so the selection criteria themselves need sign-off. A historical battery also may not represent the extremes a changing climate will produce, so plan periodic, versioned additions rather than silent edits.
Who can benchmark our NWP code on GPUs and run the emulator comparison for us?
TechnoLynx can take an agency from framing its requirements to a go/no-go report for the review board. It captures the nowcasting requirements and benchmarks IFS or WRF on target GPUs at reduced precision against the FP64 baseline. Where the case supports it, it prototypes and verifies an AI emulator against the same pre-agreed battery and scores. The first step is framing your nowcasting requirements, delivered under GPU Performance Engineering.
The engagement covers the engineering, not meteorological authority. Your verification scientists and review board still own the scientific judgement and the operational sign-off. Teams with strong in-house HPC and ML staff can run much of this themselves using open tools: Extreme Weather Bench publishes its cases, metrics and code, and the twCRPS used in the HRRR comparison is in the open-source scores Python package. Academic partners are another realistic route for the verification half.
FAQ
Is AI replacing meteorologists?
The WMO says no. AI works as part of a wider forecasting system, alongside physics-based models, operational infrastructure and human expertise, rather than replacing them.
