For a WRF-style regional ensemble, a benchmark of your own configuration should decide which stages go to the GPU and where precision can drop. A blanket port or a headline speedup should not make that call. The ICON port moved nearly the whole workflow to the GPU, including the dynamics, physics and data assimilation, and kept only the low-frequency observation operators on the CPU. The published studies also point to reduced precision in advective terms and radiation, with pressure-gradient and gravity terms precision-sensitive and the 4D-Var minimisation kept in double. The benchmark’s timings then feed one cost per accepted unit of work. A TCO comparison across GPU SKUs and interconnects can be argued on that number, and so can a choice between people and automation.

Which parts of a WRF ensemble should move to the GPU first?

The GPU-accelerated stages that pay back are the ones that run every time step and take most of the runtime. In ICON those are the dynamics and the physics, and most model components had to be ported together.

TechnoLynx scopes this as a GPU Performance Engineering engagement. TechnoLynx identifies which WRF pipeline stages are GPU candidates and the skill-score tolerances that bound precision loss, then benchmarks A100s against your current CPU cluster.

Where the time goes is documented for ICON. In a typical NWP run, about 50% of the time is spent in the dynamics and about 30% in the physics. Communication takes 8%, initialisation (including the transfer to GPU memory) takes 4%, and diagnostics and output are each below 5%. WRF’s split will differ by configuration, so profile it rather than assume it.

The ICON team also showed why porting a few isolated kernels does not pay. Copying all the model variables from CPU to GPU memory took about as long as a full GPU time step. Porting only isolated kernels incurs a transfer penalty that their execution time cannot amortise. Their answer was a full port, with CPU-GPU transfers limited to infrequent events such as writing output. That is the strongest argument in the sources against a blanket “port the hot kernels” plan. It is also a reason not to commit to a full port before measuring, because a full port is a large job.

Data assimilation is not the easy first win

It is tempting to start with data assimilation and preprocessing, on the reasoning that they are self-contained and tolerate lower precision. The ICON and IFS evidence points the other way on both counts. In MeteoSwiss’s operational ICON setup, the data-assimilation observation operators run only about every 30 minutes and need only part of the model state. They were kept on the CPU, and the GPU-to-CPU transfer cost 0.09% of total runtime. The one exception was latent heat nudging of radar precipitation. It runs more often, so it was ported to the GPU. On precision, ECMWF keeps the 4D-Var minimisation and the tangent linear and adjoint models in double precision, because that part of the IFS has been strongly sensitive to rounding errors in the past.

The deciding factor is call frequency, not which box a stage sits in on the pipeline diagram. In ICON, anything called every time step went to the GPU with the dynamics; operators called only every 30 minutes stayed on the CPU.

What WRF physics ports have shown

The published WRF physics result covers one scheme, FSBM microphysics. On NERSC’s Perlmutter (A100 GPUs), porting parts of the Fast Spectral Bin Microphysics (FSBM) scheme with OpenMP offload gave a 2.08x overall speedup on the CONUS-12km thunderstorm case. With 16 MPI ranks and one GPU per rank, the FSBM routine itself ran about 3x faster and the whole program about 2.2x faster. FSBM is an attractive porting target because its cost grows with the square of the number of size bins per grid point, and profiling confirmed it as a hotspot. Memory is a limitation of the ported code, with low arithmetic intensity on the roofline. The authors explain the underuse by uneven cloud cover: cloud-free cells need fewer calculations, so GPUs with a single rank’s work sat partly idle.

Microphysics variables agreed to 1-5 digits. That is not bit-for-bit, so the real question is whether the differences sit inside the spread you already accept.

What porting effort and risk look like

The effort is measured in years and in lines of code. The CLUBB port to OpenACC (about 100,000 lines of Fortran) took roughly three years. Most of the work went into restructuring code to push the column loop down into low-level routines. ICON is about 1.2 million lines of Fortran with about 21,000 OpenACC pragmas added. The subset used for operational NWP is about 660,000 lines and 13,000 pragmas.

Validation has to be designed before the port, not after. TempoQuest’s method for AceCAST, its GPU WRF, is to run the same configuration on two or more different CPU-WRF setups. Their disagreement sets the acceptable error tolerance, and the GPU run has to fall inside it. The open-source wrf_gpu project fixes its limits in advance and never loosens them: root-mean-square differences of 1.5 K for 2 m temperature, 120 Pa for surface pressure and 1 mm for accumulated rain. Those checks found real bugs. A Tiedtke cumulus calling error made the rain difference peak at 1.1 mm, and after the fix the peak was 0.03 mm. MeteoSwiss verified ICON-CH1-EPS, run on GPU from November 2022, against the COSMO-1E system.

Stage-by-stage decision table

Pipeline stage	What the published ports show	Precision stance	First move
Dynamical core time stepping	About 50% of ICON NWP runtime	Mixed precision validated in ICON against observations; pressure-gradient and gravity terms precision-sensitive in GRIST	Port together with the physics
Physics parameterisations	About 30% of ICON runtime; FSBM gave 2.08x overall in WRF	wrf_gpu follows the standard WRF build in single precision	Port with the dycore; validate against CPU spread
Tracer transport and advection	GRIST tracer solver runtime cut by 44% with mixed precision	Mostly single precision	Strong mixed-precision candidate
Radiation (ecRad)	Already single precision in ECMWF operations	Single precision	Port; low precision risk
DA observation operators (low frequency)	Kept on CPU in ICON at 0.09% transfer cost	Not the bottleneck	Leave on CPU unless called often
4D-Var minimisation, tangent linear and adjoint	Strong past sensitivity to rounding	Double	Do not reduce precision
High-frequency diagnostics, output preprocessing	Ported in ICON to avoid per-step transfers; file I/O left on CPU	Follow the fields they read	Port with the time-stepping loop

Is mixed precision safe in the dynamical core, or only in post-processing?

Mixed precision is safe in parts of the dynamical core when you choose term by term. It does not extend to the precision-sensitive pressure-gradient and gravity terms.

The GRIST study is the clearest evidence at term level. Most of the precision-sensitive terms were the pressure-gradient and gravity terms, and most of the insensitive ones were advective. The advective terms are also the more computationally expensive ones, so the largest savings fall where precision risk is low. With that split, GRIST’s hydrostatic, non-hydrostatic and tracer transport solvers ran 24%, 27% and 44% faster. A modification was rejected if its error exceeded a fixed threshold of 0.05. In tropical cyclone tests the mixed-precision runs stayed within the spread of perturbed double-precision runs.

Operational centres have also validated reduced precision. MeteoSwiss runs a mixed-precision ICON dycore in which scientists chose which fields and operations drop to single precision. The model was validated against a double-precision version over a long period of observations, with no impact on meteorological scores. At ECMWF, single precision cut computing time for uncoupled IFS runs by about 40%. Váňa et al. (2017) found no noticeable loss of accuracy in annual integrations or medium-range ensembles.

The caveats are specific. In single-precision IFS runs the main difference was a larger error in total air mass conservation, and that error affected forecast scores. A global mass fixer made the differences in geopotential height error mostly insignificant. Precision sensitivity also changes with the job. In GRIST, weather forecasts differed mainly at small scales, while decade-long climate runs differed at larger scales.

FP16 and TF32 are a narrower case. Hatfield et al. (2019) studied the IFS through software emulation. With a targeted, minimal use of double precision, they could run almost all of the Legendre transform in half or mixed half/single precision without affecting forecast skill. That work targeted NVIDIA Tensor Cores in a spectral model. WRF is a grid-point model with no Legendre transform, so this result does not show half precision is safe in WRF’s dycore. For WRF we would treat FP16 as unproven inside the time-stepping loop. We would test TF32 or single precision term by term against skill-score tolerances you set for your own lead times, variables and region, because those thresholds do not transfer from another centre’s study.

Why a GPU performance comparison against a CPU cluster is easy to get wrong

A GPU performance comparison against CPUs measures workload choice and optimisation effort as much as it measures hardware. The fair version compares each device at its own best, on your workload. The CLUBB study in Geoscientific Model Development (2026) makes the point directly: plain speedup ratios can overstate the GPU’s advantage and hide the conditions under which CPUs compete, because they depend so much on the workload chosen. Its authors propose peak-based metrics instead. On NCAR’s Derecho, a node with two 64-core AMD EPYC 7763 CPUs was compared with a node of four NVIDIA A100s. The GPU node first beat the CPU node’s best throughput at 25,600 columns, and at its own peak it was 1.48x faster than the CPU at its peak.

The same paper shows how easily the result flips. In one matrix solver, the GPU-oriented loop order gave the GPU nearly a 5x peak-to-peak advantage. Reordering the loops to suit CPUs dropped that to 0.71x, so the GPU never beat the CPU’s peak. A large reported GPU speedup may simply indicate that the CPU version was poorly optimised. For normalising results across devices and vendors, see making GPU benchmarks comparable. For an NWP port, the checklist below covers what matters most.

Checklist for a fair CPU-versus-A100 ensemble benchmark

• Representative case: your domain, physics suite, vertical levels and forecast length. ICON benchmarked a single member of the operational ICON-CH1-EPS configuration for one forecast hour, because output and diagnostics run hourly.
• Peak against peak: tune batch size or ranks per device on both sides before comparing.
• Equal optimisation effort: state which side the code structure favours, since loop order alone can reverse the conclusion.
• Equal resources: compare like units (sockets, nodes, or a fixed power or cost envelope) and say which. ICON’s 5.5x figure is socket to socket: 536.5 s on the CPU against 95.80 s on the GPU.
• Repeat runs: report run-to-run variation. CLUBB’s GPU runtimes varied by less than 2% and its CPU runtimes by less than 5%.
• Accuracy gate: count a run only if it passes the CPU-spread or fixed-limit validation you defined in advance.

Has anyone published a workload-representative GPU-versus-CPU benchmark of batched ensemble execution? Two sources come close. TempoQuest’s AceCAST document gives a 10-member operational ensemble comparison, though it is a vendor’s own document. MeteoSwiss’s peer-reviewed ICON paper is a model other than WRF. Neither matches your ensemble size or hardware generation, so treat both as a reason to re-benchmark, not a substitute for it.

How do we model TCO across GPU SKUs and interconnect so benchmark results line up?

Express every option as cost per accepted ensemble, built from three things: the node count the benchmark says you need to meet the deadline, the fully loaded annual cost of those nodes and their interconnect, and the number of accepted ensembles a year. A per-node or per-GPU price comparison skips the step that matters, which is how many nodes the deadline forces you to buy.

Worked example: from a 10-member ensemble benchmark to cost per ensemble

The inputs come from TempoQuest’s AceCAST overview, a vendor document. The requirement was a 10-member ensemble on a 1250 x 1100 x 42 grid at 10 km resolution, forecasting 72 hours, finishing in 94-104 minutes. Per member, CPU WRF ran on 6 CPU nodes in 90 minutes. AceCAST ran on 3 GPUs in 91 minutes and on 4 GPUs in 75 minutes.

1. Size each option to the deadline. CPU: 10 members x 6 nodes = 60 nodes, matching the stated cluster of 60 nodes with two AMD CPUs each. GPU: 10 members x 3 GPUs = 30 GPUs. The stated cluster is 8 nodes of 4 A100s, which is 32 GPUs.
2. Annualise each fleet. Annual cost = capex ÷ depreciation years + power and cooling + interconnect + support + staff. AceCAST’s own TCO assumptions were 3-year depreciation, 220 days a year and 66% utilisation. Replace these with your own procurement terms.
3. Divide by accepted output. Cost per ensemble = annual fleet cost ÷ ensembles delivered on time and passing validation in the year.
4. Re-run the sizing after optimisation. Software tuning changes how much hardware you need. Before ICON’s full set of optimisations, a 33-hour forecast on two nodes (8 A100s) took about 3,200 s against a 3,000 s limit, so each member needed three nodes. After optimisation it took 2,642 s on two nodes. For ICON-CH1-EPS’s 11 members, that is 33 nodes falling to 22 (11 x 3 against 11 x 2).

We have left the prices as named terms on purpose. Your quotes will differ from any illustration, so named terms serve better than borrowed prices.

Utilisation decides owned versus rented hardware

Utilisation, the share of hours the hardware does productive work, is the input that most decides on-prem against cloud. The PromptQuorum 2026 guide suggests a rough break-even: divide the fully loaded 3-year on-prem cost (capex, power, cooling and staff time) by the fully loaded 3-year cloud cost at 100% utilisation. The result is roughly the utilisation at which the two cost the same. Its illustrative model uses a $250,000 on-prem server and a $3.50/GPU-hour blended reserved rate, and lands break-even at about 55-65% sustained utilisation. At 100% utilisation, 8 GPUs run about 70,080 GPU-hours a year, which costs about $245,000 at that rate. The guide says to rebuild the model with your own power cost, staff allocation and negotiated rate before using it. An ecn-apps 2026 analysis broadly agrees, putting break-even at 55-60% sustained utilisation, reached in 11-14 months.

High, predictable, near-constant utilisation favours on-prem, while bursty, seasonal or experimental workloads favour cloud. The same guide describes a hybrid model: size owned hardware to the measured baseline and send peaks to cloud.

The line items that change the answer

• Power and cooling: ecn-apps (2026) estimates an 8-GPU chassis drawing 10.2 kW at a PUE of 1.35 adds $13,200-$16,500 a year.
• Interconnect: the same source puts InfiniBand switching at $25,000-$45,000 of capex. MeteoSwiss’s operational A100 nodes are connected by Cray Slingshot-11. Whether a given interconnect actually helps your ensemble depends on how far each member has to scale out, which is a separate scaling question.
• Egress: AWS egress at $0.08/GB can add $10,000 or more a year, per the same analysis.
• Support, networking and cooling retrofit: these typically add 15-30% on top of an 8x H100/H200 hardware price, according to PromptQuorum.
• Staff: maintaining CUDA drivers, Kubernetes orchestration and firmware is a recurring on-prem cost.

One distortion is comparing an on-prem quote without power, cooling and staff against an on-demand cloud rate without the reserved discount. For the full model, including how to use it at a vendor renewal, see our guide to building a defensible TCO model for GPU infrastructure.

Who can benchmark and port our WRF ensemble to GPUs?

TechnoLynx delivers the AI Porting & Deployment Pack, which starts with feasibility, then ports one workload with a reproducible runbook. Outside help pays off most in the first benchmark, the stage-by-stage plan and the optimisation work that sets your node count. When choosing a partner, ask how they will make the CPU baseline as well tuned as the GPU version, and how they will validate against your ensemble spread.

Some of this is outside TechnoLynx’s remit. Accepting forecast skill is a meteorological sign-off that stays with your scientists. TechnoLynx provides the engineering and the evidence, not the verdict on forecast quality. On build versus buy, the work splits into profiling, porting, validation and operational rollout. A team with Fortran and OpenACC or CUDA experience can own the long-term code. A team that wants a ready-made GPU WRF can evaluate TempoQuest’s AceCAST or the open-source wrf_gpu, which runs more than 80 WRF physics options on the GPU. A team porting its own code can use tooling such as Codee, which inserts OpenMP and OpenACC directives into Fortran.

One blended cost per accepted unit of work, for compute and for people

A blended cost-per-unit metric divides all the spend on a workflow by the units of work it actually delivered. That puts a human path and an automated path on the same axis. TianPan’s 2026 analysis of agent budgets says the numerator should include every failed attempt, retry, abandoned session, escalation and shadow evaluation in the period, and the denominator should count accepted outcomes only. A run that consumed forty thousand tokens and ended in an escalation counts in the numerator but not in the denominator.

LynxBenchAI measures throughput, latency, energy and cost per unit of work on your own hardware, so the compute side can start from measured figures.

The examples in these sources are support tickets and LLM agents. We would apply the same method to cost per accepted forecast. For an ensemble, the numerator holds reruns, crashed jobs and validation runs, and the denominator holds forecasts delivered inside the deadline that passed verification. The unit the customer pays for is rarely the unit the provider bills for: customers pay for a resolved ticket, providers bill for tokens or calls.

How do I turn staff time per ticket into a dollar figure next to API cost?

Multiply each task’s human minutes by a fully loaded hourly rate divided by 60, then add that to the same task’s compute or token cost. The cost of a ticket is the sum of every token, tool call and human minute that ticket consumed, not only the call that produced the final answer. A fully loaded rate covers salary plus employer costs and overheads, not the salary alone. Using salary alone makes human work look cheaper than it is.

1. Fix the unit and the window. Use the same unit (a resolved ticket, an accepted keyframe, an accepted forecast), the same period and the same volume for both paths.
2. Set a quality threshold first. Callmissed’s guide says to compare options only after they meet the same quality threshold, defined as equivalent customer outcomes within a set tolerance. It also recommends fixing a non-inferiority margin in advance: the largest drop in quality the business would accept in exchange for savings.
3. Build the numerator. The guide’s formula is cost per resolved ticket = (model + retrieval + tools + infrastructure + human handling + review and rework costs) ÷ verified resolved tickets. Human escalations stay in the numerator.
4. Stress-test the assumptions. Vary the input/output mix, cache eligibility, retries and fallbacks, and the review workload.
5. Reconcile monthly. Divide total vendor-invoiced spend by accepted outcomes from the business system. TianPan notes that a 20% gap caught monthly leads to a tuning conversation, while caught quarterly it leads to a renegotiated contract.

A fair cost per ticket for triage funding

For the triage rollout, compare labour cost per resolved ticket with LLM pipeline cost per resolved ticket, where the LLM figure includes the human handling of everything it escalates. Report total cohort cost, verified resolution rate and cost per verified resolved ticket together. Then a cheap model cannot win by closing conversations early or passing unresolved work to agents. TianPan reports that the first honest calculation of cost per accepted outcome typically comes out three to eight times what the API maths suggested. A dashboard for this decision needs one axis, cost per resolved ticket, with both paths plotted on the same ticket cohort.

Is generated concept art cheaper than the vendor once art-director iteration is counted?

The same formula answers the build-versus-buy question per keyframe. Cost per accepted keyframe = (iteration rounds x minutes per round x loaded art-director rate + GPU compute for every generation, including rejects) ÷ keyframes accepted. Compare that with the vendor’s per-keyframe rate plus the director’s time reviewing and revising vendor work. Do not assume a number of iteration rounds per SDXL keyframe. Measure it on a pilot batch of your own shots and use the median and the bad tail, not a single-pass assumption. Licensing and legal risk belongs in the comparison as an explicit line item. Whatever value you give it has to come from your legal team, because pricing that risk is a legal judgement, not an engineering one.

Hidden costs that belong in a human-versus-automation comparison

• Failed attempts and retries: every one counts in the numerator.
• Escalations and human handling: included in the numerator.
• Review and rework: the intervention needed to keep quality equivalent.
• Iteration rounds: for creative work, the director’s prompt and pose iterations, measured rather than assumed.
• Shadow-mode evaluations: the runs used to check the pipeline.
• Licensing and legal risk: a line item priced by legal.
• QA and error correction: omitting them understates the true cost of either path.

Each omission pushes the same way. Underestimating retries and iteration makes automation look cheaper than it is.

Hold the port to cost per accepted forecast

The cost-per-unit discipline matters most for the WRF decision, because a speedup figure can improve while the cost of the forecast you deliver gets worse. TianPan’s point applies directly: unit economics depend on the metric the team is held to, and work that lowers cost per call while raising cost per resolution should not count as a win. For an ensemble, that means judging each optimisation and each precision change by whether it lowers the cost of an on-time, verified forecast at your real cycle count. If it only lowers runtime on a benchmark case, it has not yet done the job. Where your physics suite’s precision limit sits shows up only in a term-by-term test against your own skill scores.

FAQ

What does TCO mean?

Total cost of ownership is an estimate of all the direct and indirect capital and operating costs of hardware, electricity, networking, bandwidth and labour over a multi-year lifecycle.