To make a numerical weather prediction (NWP) system faster without losing forecast skill, measure before you change anything, at every layer. In practice that means three things. Rank sounder channels by information content, or compress them with principal components, and then check the reduced set on your own forecasts instead of copying another centre’s list. Drop a compiler flag only after a bitwise or ensemble-statistical test shows the output has not changed. Profile the GPU dynamical core to find out whether uncoalesced memory access or halo exchange dominates before you choose between a data-layout rewrite and occupancy tuning.

The sections below follow that order: what you assimilate, how you build, and where the GPU time goes.

Choosing which hyperspectral sounder channels to assimilate

Rank hyperspectral infrared channels by degrees of freedom for signal (DFS), optionally add principal-component (PC) compression, and validate the reduced set against full-spectrum assimilation on your own analysis and forecast scores. Operational centres already thin heavily: EUMETSAT notes that IASI spectra arrive as thousands of channels, while only around 100 are typically assimilated in NWP models.

LynxBenchAI is the tool for the hardware side of this cost question: its published results compare throughput, latency, memory, energy and cost per unit of work across accelerators. It does not evaluate channel lists, and it does not run your assimilation system.

Information-content selection or PCA compression for IASI, AIRS and CrIS?

Information-content selection is the established baseline. Collard’s 2007 ECMWF technical memorandum used DFS as the figure of merit for choosing IASI channels, and found that switching to entropy reduction made little difference to the result. In that study, 252 channels captured 118.8 of the 168.0 available DFS, about 71%. IASI measures 8461 channels in total.

Coopmann et al. (AMT, 2020) made a new IASI selection accounting for observation-error covariances. At an equal channel count, the new selection cut analysis error by 3% for temperature, 1.8% for humidity and 0.9% for ozone. They also found that 172 channels gave 90% of the maximum total DFS reachable with 400. A layered, height-dependent variant of information-content selection for AIRS raised the atmospheric retrievable index to 0.54, against 0.38 for the unlayered method, and improved the accuracy of retrieved temperature by more than 11% between 100 and 0.01 hPa. That result comes from a retrieval study, not an assimilation trial, so read it as evidence about the selection metric, not about forecast impact.

PC compression works on a different principle. Instead of discarding channels, it replaces correlated radiances with a few uncorrelated scores that keep most of the information. EUMETSAT’s IASI PC products reach a compression factor of about 30 and have been operational since 2011. In ECMWF experiments, forecasts made from reconstructed radiances performed as well as forecasts from the original radiances. Lu and Zhang (MWR, 2019) applied PCA to a 199-channel AIRS subset for ensemble assimilation and found that the leading PCs explain orders of magnitude more variance than the lower-rank ones.

The choice follows from your constraint. If keeping forecast skill is the binding concern, start with DFS selection, using correlated errors where your system can estimate them, and validate it against full-spectrum skill. If compute, storage or transmission volume is the pressure, put PC compression on top. The cost of PCA should be known before you start. PCA is lossy, and reconstructed radiances carry only as many independent pieces of information as the scores behind them, with different error characteristics from the observations. NASA GMAO found that PC compression strips out noise, so cloud detection tuned to the original noise estimate cuts off the cold end of the distribution and pushes the bias positive.

How much forecast skill do you lose by thinning channels, and how many node-hours does it save?

One documented case shows no skill loss at a real cost saving. An ECMWF study replaced 165 IASI long-wave radiances with 20 PC scores in 4D-Var. That cut data volume eightfold and the overall cost of assimilation by 25%, with no detectable loss of skill in analyses or forecasts. Adding scores beyond about 20 produced no measurable improvement in fit to other observations.

That figure belongs to that system, and the authors say so: in a densely observed system the truncation threshold and observation-error covariance may need revisiting. The cost side of your trade-off has to be measured on your own solver. Profiling the cost per channel of 3D-Var, 4D-Var and hybrid EnVar schemes under hyperspectral loads is its own exercise.

Why validate a reduced channel set on your own forecasts instead of adopting another agency’s list?

Very different channel lists can perform almost the same, so a published list is one good answer among many, not the optimum for your system. A 2024 AMS comparison for CrIS built an optimal 455-channel selection that shared 224 channels with Carminati’s selection. The difference in analysis error variance was still small, and the authors attribute this to overlapping Jacobians in hyperspectral instruments. The new IASI selection of Coopmann et al. shared only 28% of its channels with Collard’s.

An information-content selection depends on the error statistics and assimilation system it was built for. When Collard reran his selection with a different background error matrix, 66% of the channels were common to both runs, and he advised testing any final selection against the optimal selection for a diverse range of atmospheric profiles. Transfer problems show up in compression too. GMAO’s account of ECMWF’s work reports degraded performance when the diagnosed observation error was replaced with one derived from reconstructed radiances.

Compiler flags in a production NWP build

Which flags are safe to drop depends on the compiling setup. You have to derive the drop list for your exact gcc and nvcc versions, GPU and solver, and back it with documented equivalence evidence from representative forecast runs. Liu et al. (GMD, 2015) explain why: compiler families, versions and flag sets each turn on different optimisation schemes, so different compiling setups can produce different results from the same program.

TechnoLynx can give you a documented list of flags safe to drop for your gcc/nvcc/GPU toolchain, backed by bitwise and statistical equivalence evidence. The work starts by inventorying the build flags and agreeing a numerical tolerance, then runs paired builds and deploys the passing flags with regression monitoring.

Which gcc/nvcc flags and debug-symbol settings can be dropped without changing numerical results?

Device debug symbols disable all device optimisations. NVIDIA’s CUDA-GDB documentation states that compiling with -g -G forces -O0 (apart from very limited dead-code elimination and register-spilling optimisations), increases binary size and reduces performance. For profiling, -lineinfo adds line-number information for device code without affecting execution performance, and for recent CUDA versions -lineinfo and -G are mutually exclusive. nvcc already uses -O3 for GPU code by default, and -DNDEBUG turns off runtime assertions that can also slow execution.

Check where your flags actually come from. nvcc honours NVCC_PREPEND_FLAGS and NVCC_APPEND_FLAGS from the environment, so a -G can be injected without appearing in any build script, although injected flags do show up in the --verbose log.

Optimisation and floating-point flags are a different matter. Removing them, or adding them, changes which optimisation schemes are enabled. Liu et al. traced one difference to the compiler calling pow at -O1 and cbrt at -O2 for the same cube root, and found a case where -O3 triggered a compiler bug. A 2024 study of GPU numerics names aggressive options such as -ffast-math among the root causes of numerical discrepancies between platforms. The GEOS-Chem documentation warns that chip-specific optimisation flags can change simulation output. Flags can still be worth a lot: on MeteoSwiss’s ICON GPU port, -O2 with -Mstack_arrays reduced total runtime by about 6.8%.

Has anyone benchmarked -g/-G overhead for your particular nvcc/gcc pairing? The answer has to come from measuring the build you ship. Keep the debug configuration as a build of its own, not a deleted one. The EC-Earth replicability study recommends compiling and running with debugging flags and without optimisations before launching production experiments.

How do you show forecast output is bitwise or statistically identical after a flag change?

Try bitwise identity first. If reordering legitimately breaks it, fall back to a statistical test against an ensemble whose tolerance was fixed before the experiment. Bitwise identity does not always survive a compiler change, and NVIDIA’s floating-point guide notes CPU and GPU results differ through FMA and math library implementations. In MPAS-A, Price-Broncucia et al. (GMD, 2025) found FMA as impactful as changing a scientific model parameter by 1% to 10%.

A procedure that holds up in review runs in this order:

1. Agree the tolerance and the variables it covers before any run, so that no one rationalises a result after the fact.
2. Change one flag at a time and record both the runtime and the output, keeping a change only if it buys measurable speed and passes the test.
3. Check bitwise identity on short integrations; Liu et al. used five model days, citing earlier work that a short run is enough.
4. If the bits differ, run an ensemble consistency test. Milroy et al. (GMD, 2018) recommend their nine-time-step UF-CAM-ECT as the first step for compiler flag changes, at roughly 70 times less CPU time than a 12-month CESM run.
5. Where a CPU reference exists, MeteoSwiss’s probtest approach builds a CPU ensemble with perturbations in the least significant digits, derives a tolerance for each variable, and passes the GPU run only if every diagnosed field falls inside the spread.
6. Rerun the whole check after any compiler or hardware upgrade, because the result belongs to the versions tested.

Calibrate the statistical test, because it can fail in both directions. Price-Broncucia et al.’s UF-ECT setup for MPAS-A aims for a false-positive rate at or below 0.5% while still failing changes known to produce statistically distinct output. Massonnet et al. (GMD, 2020) put the starting assumption plainly: treat model results as different across computing environments until you show they cannot be deemed incompatible.

Profiling a GPU weather solver: uncoalesced access or halo exchange?

Before choosing between a structure-of-arrays (SoA) rewrite and occupancy or register tuning, profile the actual kernel on the actual hardware and attribute its time to compute, memory access or halo exchange. Our position is that best-practice rules such as “use SoA” only make sense in the context profiling gives them, not as a default.

TechnoLynx runs this through GPU Performance Engineering. The first step is a kernel-level Nsight study of the advection kernel that separates compute, memory and halo-exchange phases. TechnoLynx then prototypes both an SoA variant and an occupancy/register-tuned variant, validates both against the existing dynamical-core output, and deploys the one that pays off before the forecast-cycle deadline. The general method is covered in our guide to profiling GPU kernels to find the real bottleneck; this section applies it to an NWP dynamical core.

How do you tell whether uncoalesced access or halo exchange is the real bottleneck?

Start with a timeline, then go down to the kernel. NVIDIA’s Nsight Compute triage guide advises using Nsight Systems first when it is unclear which kernel matters, then Nsight Compute. It also says to treat absolute kernel duration as the ground truth and to fix the single largest limiter first. In NWP, the timeline should also show host-device transfers. ICON’s developers measured that copying all model variables to the GPU takes about as long as a full GPU time step, because atmospheric codes have low arithmetic intensity.

Halo exchange has a measurable footprint. A 2014 study of elliptic pressure solvers on GPU clusters measured MPI overhead of about 10% for conjugate gradient and about 40% for multigrid at the largest problem size. Multigrid needs three halo exchanges per level, and the communication-to-computation ratio worsens on coarse levels. In MeteoSwiss’s ICON runs, communication took about 8% of runtime, against about 50% for dynamics and 30% for physics.

Inside a kernel, coalescing shows up in one ratio. NVIDIA’s guidance is that a fully coalesced warp load uses 4 sectors per request and a fully strided one 32, an efficiency difference of up to 8x. In Nsight Compute, a ratio above 4 means access is uncoalesced. In NVIDIA’s worked example, source-level analysis found an 8:1 transaction inefficiency on one line, and after fixing it the bottleneck moved to compute. NVIDIA’s guide likewise says to re-profile immediately after each change.

What the profile shows	What it usually means	Next move
Large MPI or copy gaps on the Nsight Systems timeline	Communication or host-device transfer dominates	Fix data residency and communication before touching kernel layout
Sectors per request well above 4 on hot loads	Uncoalesced global access	Consider a layout change such as SoA when warp threads each touch a single field
Non-zero stack frame, spill loads and stores	Register pressure; the kernel can be occupancy-limited and memory-bound at once	Tune registers and block size, then re-profile
High occupancy, low utilisation	Latency-bound; more occupancy will not help	Add instruction-level parallelism, prefetching or pipelining
Healthy kernels, idle gaps between launches	Dispatch or synchronisation pacing	Batch launches; ICON gained about 5% overall from CUDA Graphs

Is rewriting an advection kernel to structure-of-arrays worth the risk compared with occupancy and register tuning?

SoA is worth it only when the profile shows uncoalesced access dominating the kernel, and the kernel dominating the forecast run, by enough to pay for the rewrite before your deadline. Otherwise, occupancy and register tuning is the safer interim move. The size of the payoff varies widely. Sung et al. measured layout-transformation speedups from 1.07x to 6.6x across structured-grid benchmarks, tied to how far the new layout diverged from the baseline. Kernel gains also shrink at application level. In an unstructured-mesh study, the best SoA version sped one kernel up by 21%, which became 11% for the whole application. Michalakes saw the same thing in WRF, where a 5x to 20x module speedup gave 1.25x to 1.3x overall.

A rewrite has its own risks. A 2025 study of AoS-to-SoA conversion measured up to 75% degradation for linear-cost kernels on scattered memory once conversion overhead was counted, and its authors note that boundary data exchange in domain-decomposition codes can favour AoS. If the halo-exchange code reads the old layout, the rewrite reaches beyond the kernel. Tuning is cheaper and still matters: WRF’s CUDA tracer advection ran 6.7x faster than the Fortran original, and its performance tracked warp occupancy, which was sensitive to thread-block size and register count.

If halo exchange dominates, neither option is the priority. Fix communication first. ICON-A measured a speedup of about 10% in Piz Daint benchmarks from direct GPU-to-GPU communication, while the 2014 solver study got no speedup in preliminary attempts to overlap halo exchange with computation on GPUs.

Deadlines are real constraints, and the MeteoSwiss record shows it. Each 33-hour ICON-CH1-EPS forecast must finish within 3000 s. The first operational build took about 3200 s on two nodes and had to run on three. After the full set of optimisations it ran in 2642 s on two. When two ecRad strategies benchmarked equally, the team chose the one needing fewer code changes. Under deadline pressure, that is the right tie-breaker. The risk is that the same pressure picks the low-effort option even when the profile says the rewrite would pay for itself.

Mixed precision is a separate lever needing its own validation. ICON’s mixed-precision dynamical core gave 8% after validation against observations showed no impact on meteorological scores, but FP64 versus FP32 needs for the forecast models deserve their own analysis. Choosing GPUs for the next cluster is a separate question too. A benchmark has to reflect your own workload, and a write-up should keep measured sustained throughput separate from peak specifications.

Who can profile and tune our GPU weather solver before the next forecast cycle?

TechnoLynx profiles and tunes GPU weather solvers for teams that need a faster dynamical core without a regression in forecast quality. Whether to keep the work in-house or bring in a specialist depends on whether your team has time, before the next deadline, to run profiling and equivalence studies alongside operations. If you hire anyone, ask for profiling evidence before any rewrite proposal, and for an equivalence test agreed before any change ships. Through GPU Performance Engineering, the engagement starts with a kernel-level profiling study, then moves to prototypes validated against your existing output, and ends with the documented compiler-flag drop list and the chosen optimisation deployed.

The engagement covers GPU and build engineering. Signing off a channel selection, an observation-error model or a change to forecast verification stays with your data-assimilation scientists, and our wider work for forecasting teams is described under AI engineering for weather and climate. A buyer looking for the cheapest hours will be better served elsewhere.

Every validation here expires

Each piece of evidence above is tied to a configuration. NVIDIA’s triage guide says to re-evaluate the dominant limiter on each new GPU generation, because memory bandwidth tends to grow faster than SM count. Liu et al. point out that compiling setups have to be updated frequently, which threatens reproducibility, and the EC-Earth authors recommend running the replicability protocol every time the model is ported to a new machine. A channel selection should be tested against the optimal selection for a diverse range of atmospheric profiles. The question to settle early is not which optimisation to make but who reruns the evidence when the compiler, the GPU or the instrument changes, because that is when an imported assumption returns.

FAQ

What are the drawbacks of PCA for satellite radiances?

PCA is lossy: reconstructed radiances contain only as much independent information as the retained scores, and their error characteristics differ from the original observations. In practice, cloud detection may need modifying before PC reconstructed radiances assimilate cleanly.

What does the -O3 flag do?

-O3 is the most aggressive standard optimisation level in gcc, and nvcc applies -O3 to GPU code by default. Because optimisation levels switch different optimisation schemes on, a move to or from -O3 needs an equivalence check; one Earth-system study found a case where -O3 triggered a compiler bug.