Yes, operational numerical weather prediction now runs on GPU clusters. GPUs offer high throughput and improved energy efficiency through massive parallelism. MeteoSwiss became the first national weather service to run ICON operationally on GPUs, and after tuning and mixed-precision work its port reached a 5.5x speedup over the CPU baseline in a socket-to-socket comparison. For your own cluster the decision is a chain of separate calls: prove the gain on a representative test before spending the shared allocation, pick a porting strategy your team can maintain, check that the facility can power and cool the hardware under real load, confirm the forecast still meets its delivery window at full scale, and turn all of it into a case a budget board can approve.

Why are weather services moving their NWP suites onto GPU clusters?

In the ICON benchmark, the CPU reference on eight sockets took 536.5 s, and the optimised GPU code on eight NVIDIA A100 GPUs took 95.80 s, which gives the 5.5x figure. The first working port gave about 4.2x before any optimisation. The speedup on day one was not the speedup reached once the optimisation work was finished.

The mechanism is memory bandwidth, not peak flops. ICON is mostly memory-bandwidth limited, and the A100 has 7.8 times the theoretical main-memory bandwidth of the AMD EPYC CPU it was compared with (1560 against 204.8 GB/s). The ICON-A climate configuration shows a single-node CPU-to-GPU speedup of 6.4 on Piz Daint, matching the GPU-to-CPU bandwidth ratio. When a vendor quotes a far larger speedup for your bandwidth-bound dynamical core, ask which ratio it rests on.

The operational argument is that the speedup buys resolution inside a fixed window. Going from 50 km to 2 km horizontal resolution raises computational effort by at least a factor of 15,000. Each 33 h ICON-CH1-EPS ensemble forecast must finish within 3000 s; at first deployment the run on two nodes took about 3200 s, so each member needed three nodes, and after the full set of optimisations it ran in 2642 s on two nodes. That is the form a decision-grade speedup claim takes: the measured runtime, the hardware it ran on and the operational limit it has to beat. Our wider coverage of AI engineering for weather and climate places this inside the rest of the forecasting pipeline.

How does a GPU cluster for high-performance computing differ from your CPU cluster?

A GPU HPC cluster is built from nodes and an interconnect, with MPI handling communication across nodes. In practice it behaves differently. MeteoSwiss’s operational system has 42 nodes, each with one 64-core AMD EPYC CPU and four NVIDIA A100 96 GB GPUs, connected by Cray Slingshot-11. ICON runs one MPI task per GPU, and its communication library was ported so GPUs exchange data directly.

That architecture brings two design consequences for the porting plan. First, you cannot port only the hot kernels. In ICON, copying all variables from CPU to GPU memory takes about as long as one full GPU time step, so most model components must move to the GPU together and transfers are limited to infrequent operations such as output. COSMO’s GPU project reached the same full-port conclusion. Second, GPUs need a large amount of work per device to stay efficient: ICON-A needed a minimum of about 10,000 grid columns per GPU, against about 100 on CPUs. For ICON-A, that limits strong scaling on GPUs, while on CPUs it extends to larger processor counts.

Not every component has to follow. ICON keeps the data assimilation observation operators and the LETKF analysis on the CPU, and the GPU-to-CPU transfer this requires costs 0.09% of total runtime. The cluster you are buying is a hybrid machine, and the porting plan should say which side each component lives on.

Can we trial a new assimilation scheme on a smaller GPU test cluster first?

Yes, and protecting the shared allocation is the main reason to do it. Run a scaled-down but representative case (a shorter cycling window, a reduced domain or ensemble, or subsampled observations) on a test partition. Then project its node-hour cost to the full configuration, and commit production time only once that projection has held up against at least one larger run.

LynxBenchAI publishes results on how candidate devices run the public PyTorch stack, which helps size any PyTorch components such as an ML emulator in the trial; the first step is checking its results for the devices on your test partition.

TechnoLynx can build that trial for a candidate scheme. TechnoLynx identifies the scheme’s dominant cost drivers on the target cluster, designs the scaled-down case and the scaling method used to extrapolate node-hours, instruments throughput, latency and memory on available hardware, and checks predicted against actual cost so the go/no-go call rests on evidence before the allocation is touched. If the plan includes PyTorch components such as an ML emulator or post-processing model, LynxBenchAI’s published results show how candidate devices run the public PyTorch stack those models use. That makes them broadly indicative for custom PyTorch models, but they do not measure your Fortran assimilation code.

MeteoSwiss’s ICON team sized its benchmark case on purpose. ICON’s GPU benchmarking used a single ensemble member of the operational configuration for a one-hour forecast, chosen because diagnostics and output run hourly while most other components run more often. Those benchmarks ran on two GPU nodes (eight A100s). ECMWF added a GPU partition to its Bologna facility partly so it could continuously test the GPU code paths.

One gap needs stating plainly. Treat any rule for scaling an assimilation scheme’s node-hour cost from a reduced trial to full operational size as unvalidated until your own runs confirm it.

Sizing the minimum representative test case

Size the test to the drivers that dominate the scheme’s cost (grid resolution, channel count, cycling frequency, ensemble size), and do not simply shrink everything evenly. If cost scales linearly with those drivers, one scaled run plus extrapolation is defensible. If it does not, run at least two intermediate sizes and fit the curve before you trust the projection.

Ensemble assimilation is a case where cost does not scale linearly. The EnKF update phase has computational complexity of O(M²) in the number of ensemble members. Parallelising the update for 1,024 members gave walltime gains only up to 576 cores. Total compute hours followed a U-shaped curve against members per runner. Memory also sets a floor: the ICON-CH1 strong-scaling reference used 4 GPUs because that was the smallest count on which the problem fitted in memory.

Run length matters too. The CPMIP authors recommend timing a segment of a production run, because short runs give too much weight to startup and shutdown costs.

A scaled test can still mislead. A cluster can pass every health check and still underperform under a real distributed job, because of one slow GPU or one link that degrades under load. I/O is the other blind spot: over a decade, peak compute on the top machine grew 134x while I/O throughput grew 12x. Check whether queue contention and allocation policy on the test partition match the shared production cluster, and give any cost projection a margin.

Can our existing data centre power and cool an H100-class cluster?

The answer depends on one comparison. Set the facility’s usable power per rack and its cooling capacity against the sustained draw and heat output of the candidate configuration under representative NWP load. Nameplate figures do not settle it, and where the facility falls short, upgrades come before deployment.

Vendor specifications tell you what to provision, not what the system will draw. NVIDIA’s DGX H100/H200 system contains six power supplies with balanced load distribution, each rated 3300 W at 200-240 V. Six times 3.3 kW gives 19.8 kW of installed supply capacity per system. The same guide specifies an operating temperature of 5°C to 30°C. Models such as Wattlytics estimate average GPU power as the lower of TDP and a frequency-dependent curve, and bring cooling in through PUE. In that model, average GPU power never exceeds TDP.

Sustained H100 rack power under your NWP workloads has to be measured or modelled on your configuration. Newer parts can also raise draw even as they cut runtime: in one stencil study, CUDA time to solution fell from 340 s to 200 s on the newer GPU while power rose from 250 W to 510 W.

Facility feasibility checklist

• Usable power per rack after redundancy commitments (N+1 or 2N), not raw feed capacity.
• Measured or modelled sustained draw per node under the operational ensemble, at the GPU clocks you intend to run.
• Cooling capacity at the rack (air, rear-door heat exchangers, or a liquid loop) against that sustained heat load, with headroom for growth.
• Inlet temperatures held within the vendor’s stated operating range.
• Whether moderate GPU frequency reduction closes the gap: Wattlytics found moderate clock reductions cut energy substantially with minimal performance loss, while aggressive underclocking degrades performance nonlinearly.

If capacity falls short, there are three realistic routes: fewer GPUs per rack across more racks, power and cooling upgrades (possibly a move to liquid cooling), or colocation in a facility already built for this density. The choice comes down to comparing those three costs. The assessment holds only for the site and configuration you checked, and it needs redoing if either changes.

OpenACC or CUDA for the dynamical core: what do we give up, and can four people maintain it?

Choose OpenACC when the team is small and a single CPU/GPU source tree matters more than the last factor of speed. Choose a native CUDA rewrite only when maximum throughput is the overriding requirement and you have, or can justify, the people to maintain a second kernel chain.

TechnoLynx can turn that general rule into a decision about your code. TechnoLynx profiles the existing Fortran dynamical core to find the kernels worth porting and their speedup ceiling, prototypes representative kernels in both OpenACC and CUDA to measure the speed and effort difference on your stencils, and assesses what each path means for a team of your size. Once a path is chosen, GPU Performance Engineering implements and tunes it on the target hardware. The AI Porting & Deployment Pack covers the port itself, including validating numerical and performance equivalence against the CPU Fortran path.

The published gap is real but bounded. On 3D stencil kernels, the best-tuned CUDA implementation outperformed optimised OpenACC by 2.1x. On a CAM-SE tracer advection kernel, OpenACC was about 1.5x slower than optimised CUDA, and its development was substantially simpler. The same stencil study found the gap between models narrowing across GPU generations, and noted that finding the best CUDA variant required a costly search over many implementations. Calling a vendor API does not make code faster on its own; it exposes lower-level controls, and using them well takes sustained commitment that is not always worth paying for.

The effort side is just as concrete. COSMO’s OpenACC port needed more than 4,000 OpenACC statements and changes to 40 of its 160 source files, and the code ran 2.7x faster on GPU than on CPU. ICON carries about 21,000 OpenACC pragmas on 1.2 million lines of Fortran. ICON chose directives over CUDA or a DSL mainly for acceptance by the ICON community and the chance of merging the work into the main code, and chose OpenACC over OpenMP offload because the OpenMP compilers it evaluated were less mature. On SPEChpc 2021 accelerator systems, OpenMP target offload lagged OpenACC, a gap the authors attribute to compiler implementations.

OpenACC is not free of lock-in either. ICON’s authors note that the directives reduce readability, that the standard is fully supported only by the nvhpc compiler on NVIDIA hardware, and that there is no route for OpenACC on Intel GPUs. The two paths can also mix: COSMO rewrote its dynamical core in C++ with the STELLA DSL and a CUDA backend, and used OpenACC elsewhere, and ICON applied CUDA Graphs to selected routines for a 5% overall runtime reduction. Portability work earns its cost when it is tied to a named trigger such as a performance ceiling, a target constraint or a lock-in cost.

OpenACC or CUDA: decision table

Your situation	Lean towards	Why
Small team (around four), CPU path must stay first-class	OpenACC	One source tree; directives can be added step by step while keeping CPU performance
Community model you want to merge back upstream	OpenACC	ICON chose directives for community acceptance and re-integration
Hard throughput ceiling on a few dominant stencils	CUDA for those kernels	Tuned CUDA beat optimised OpenACC by 2.1x on stencils
Need to keep AMD or Intel GPUs open	Neither alone; plan for abstraction	OpenACC support is limited outside NVIDIA
Large code under active science development	Generated GPU paths from one codebase	ECMWF migrated its GPU recipes to a new IFS cycle with a few technical experts

Whether four people can carry a hand-written CUDA chain next to the Fortran path is a judgement, not a threshold. We would expect it to strain a team without existing CUDA depth, because every science change then has to be made twice and tested twice. The reported speedups also come from other centres’ kernel structures, so they calibrate expectations rather than predict your result.

How do we explain to budget approvers why the forecast needs a GPU cluster when the CPU model runs fine?

Give approvers a decision package, not a benchmark chart. That means measured cost and runtime for your model on both platforms, a lifetime cost and risk model, and a plain statement of what the money buys in forecast terms: higher resolution, more ensemble members or a safer margin under the delivery deadline.

LynxBenchAI publishes results on how candidate devices run the public PyTorch stack, giving approvers a reference for any PyTorch components such as an ML emulator in the plan; the first step is checking its results for the devices you are comparing.

TechnoLynx can assemble that package. TechnoLynx clarifies which gains the GPU platform delivers for your forecast, chooses the CPU and GPU comparisons and cost and risk metrics that matter, runs the benchmarks, checks that the numbers hold up to scrutiny, and writes the non-technical briefing. When the comparison has to span different hardware, it helps to turn benchmark results into blended cost metrics that finance can rank on one scale.

“The CPU model runs fine” answers today’s question, not the next upgrade. In 2015, the 1.1 km COSMO grid needed a computer 40 times more powerful than MeteoSwiss’s system at the time. GPU efficiency is what lets MeteoSwiss run ICON-CH1-EPS at up to 1 km resolution. The 2015 CSCS hybrid system for COSMO was reported as three times more energy-efficient and twice as fast as conventional CPUs.

Build the cost model on lifetime cost, not purchase price. Wattlytics defines TCO as capital plus cumulative operational expenditure, and warns that procurement built on isolated metrics such as vendor peak specifications tends to produce suboptimal decisions. In its sensitivity analysis, a 1% rise in H100 cost cut work-per-TCO by 0.76%, and the authors conclude that GPU cost sets the baseline while system usage drives financial risk. For a board, that turns into two plain questions: what the hardware costs, and how sure we are that the forecast suite will keep it busy. The CPMIP authors warn about headline ratings: one machine’s petaflop rating was considerably higher than two others combined, showing the pitfalls of inferring aggregate performance from it.

An analogy can help, as long as it does not replace the numbers. One workable framing is a wider road rather than a faster car: the forecast does not get cleverer, but more of it fits through before the deadline. Analogies oversimplify, so put the measured runtime-against-deadline figure right next to the one you use, and refresh the briefing as the model and hardware change.

Will the higher-resolution ensemble still meet the delivery window at full scale?

You can forecast this before you commit from a measured scaling curve. Measure per-node throughput for the upgraded ensemble at every node count you can reach, then extrapolate to the target count with an explicit confidence margin. If the projected completion time, margin included, falls inside the delivery window (six cycles a day, for example), proceed. If the margin is thin, fix interconnect or scheduling first and re-measure. If the shortfall cannot be recovered within budget, revisit the node count or the resolution target.

TechnoLynx can run that forecast. TechnoLynx pins down the delivery window, target node count and acceptable margin, benchmarks scaled configurations of the higher-resolution ensemble across available node counts, instruments compute, interconnect and scheduling separately, and checks the extrapolation against held-out intermediate runs before trusting the full-scale figure. Where the curve bends, GPU Performance Engineering covers the interconnect and scheduling changes, followed by a re-benchmark to confirm scaling has been restored.

On GPUs, strong scaling degrades once per-GPU work becomes too small. ICON-CH1 scaled well up to 12 GPUs, beyond which the physics component degraded while the dynamical core kept scaling. The cause was that per-GPU work became too small to keep the A100s busy, not communication overhead. For ICON-A, generally only two doublings of GPU node count kept cumulative strong-scaling efficiency above 0.5, while weak scaling stayed at about 0.98 over a 16-fold increase in grid and node count. COSMO on P100s began to saturate at about 200×200 grid points per node. On SPEChpc 2021 runs on JUWELS, the medium suite’s parallel efficiency dropped below 75% at 400 ranks for almost all benchmarks.

The practical consequence for an ensemble is that you buy throughput mostly by running more members side by side, each on a few nodes, rather than spreading one member thinly. MeteoSwiss runs each ICON-CH1-EPS member on three nodes. When COSMO-1 first came up it was slightly over its half-hour limit, and the fixes were faster inter-GPU and MPI communication, asynchronous communication and more parallelism in the code. Six months of such tweaking produced a 20% reduction in time-to-solution.

Check hardware before trusting any curve. A single degraded GPU slows a synchronous job to the speed of its worst rank, and on an MI300X one degraded intra-node link can cut AllReduce bandwidth by 15-30%.

How do we check that GPU numerical differences are acceptable for operational forecasts?

Test for statistical equivalence rather than bit-identical output, then verify against observations before going operational. GPU results are not bit-identical to CPU results, so the question is whether the differences change the forecast’s distribution. Ensemble methods answer that objectively. Comparing COSMO running on CPUs with COSMO running mainly on GPUs, an ensemble-based test produced no global rejections across 16 tested variables, so although the results were not bit-identical the authors judged the difference negligible. The same test detected differences between single- and double-precision versions for almost all variables, which shows it is sensitive enough to catch real changes.

The test is cheap enough to automate. A 10-day run of GPU COSMO took about four minutes per node, so a 50-member ensemble cost 3-4 node hours. The ultrafast ensemble consistency test has caught errors introduced while refactoring CESM code for GPUs. For setup, its authors suggest starting with an experimental ensemble ten times the number of output variables.

Statistical equivalence works as an automated pass/fail check for code changes. Forecast quality still needs verification against observations: every new model version for production is compared with observations over an extended period, typically a few weeks in each of the four seasons. ICON’s mixed-precision dynamical core was validated that way against a double-precision version with no impact on meteorological scores, for an 8% performance gain.

How have other operational centres validated GPU clusters before production?

MeteoSwiss ran new systems alongside the operational one before switching. When MeteoSwiss moved COSMO to its GPU-based system in 2015, both systems were planned to run in parallel through a transition period. For ICON, MeteoSwiss ran control forecasts from summer 2021, switched them to GPU from November 2022, ran full ensembles from summer 2023, and replaced COSMO operationally on 28 May 2024 once the quality criteria were met.

ECMWF built continuous testing into its GPU work. It added a GPU partition so GPU code paths are checked routinely through automated test runs, with moderate maintenance overhead. In that work, more than 85% of the atmospheric time step is offloaded to the GPU, and initial non-optimised GPU performance was already competitive with highly optimised CPU performance. At the hardware layer, vendor guidance converges on the same principle: a cluster can pass standard diagnostics and still fail under a real distributed job, so the best readiness test is running a workload, with validation gated phase by phase before nodes join the production pool.

Who can port and validate our NWP code on a GPU cluster?

TechnoLynx ports and tunes code for GPU clusters, and builds the benchmark evidence a weather service needs before committing allocation and capital. When choosing a partner, ask for prototypes measured on your kernels and validation against your CPU path, and avoid anyone selling a speedup quoted from someone else’s model. The AI Porting & Deployment Pack is the engagement for this work: it starts by profiling the current code and scoping which components move.

This engagement does not suit everyone. A centre with an established GPU group and an upstream community port may need little outside help, since ICON’s OpenACC changes are merged into the main code and the community has been given OpenACC training. Partnering with a national supercomputing centre, as MeteoSwiss did with CSCS, is another route. Meteorological verification and operational sign-off stay with the weather service.

The number that decides the purchase is rarely the headline speedup. ICON’s first GPU deployment, with optimisations still incomplete, missed its 3000 s window on two nodes, so it needed a third node per member until optimisation brought the run down to 2642 s. Before you take a business case to the board, make sure it shows that margin against the delivery window, measured on your configuration.