GPU capacity planning for data-heavy scientific and grid workloads has to start from measurements of your own pipeline: the throughput it sustains, the memory and storage bandwidth it actually draws, its peak VRAM, and the metered power of the facility it will run in. Dataset size, datasheet bandwidth, nameplate watts and the GPU-utilisation percentage each mislead in a predictable direction, and a cluster sized from them ends up starved by I/O, short of memory, or bought against power headroom that was never there.
Why spec sheets and dataset size give the wrong capacity number
LynxBenchAI treats hardware specification metrics (peak compute, memory bandwidth, theoretical FLOPS) as structurally insufficient predictors of real AI performance, because the things that dominate performance (memory locality, on-chip data reuse, execution strategy and software behaviour) depend on the workload and cannot be read off a datasheet.
Theoretical peak GPU memory bandwidth is a product of memory clock, bus width and the data-rate multiplier, divided by eight to convert bits to bytes. Atlantic.Net’s bandwidth guide puts sustained bandwidth for well-optimised workloads at around 80–90% of that theoretical peak. Microbenchmarks show wider gaps on some parts: a 2018 microbenchmarking study of the Volta architecture measured 750 GiB/s of global memory bandwidth on a V100 rated at 900 GiB/s (83.3%) and 510 GiB/s on a P100 rated at 732 GiB/s (69.6%). A 2019 follow-up on the Turing T4 measured 220 GiB/s against a theoretical 320 GiB/s, a ratio of 68.8%.
The practical consequence is that the bandwidth you size against should be measured on the hardware you plan to use, with GPU-STREAM or STREAM-style copy, scale, add and triad kernels. The same guide treats a result 10–20% below theoretical as normal, and a much larger gap as a sign of outdated drivers, thermal throttling or misconfiguration to fix before the number goes into a capacity plan.
Once a workload is bandwidth-bound, faster clocks offer little to no gain. A GROMACS study across L4, L40, A40 and A100 found the largest biomolecular systems showed flat throughput curves at higher graphics frequencies, which the authors read as a sign they might be memory-bound. On the L40, raising the graphics clock from 1.5 GHz to 2.5 GHz on the large benchmarks gave no gain at all. Production telemetry points the same way: one month of data from 75,703 jobs on NERSC’s Perlmutter in 2025 classified 81% of jobs as memory-bound.
Dataset size is a storage-volume figure. It says how many bytes you will hold, not how fast they must move or how much of them must be resident on the GPU at once, which is why it is a poor starting point for sizing compute.
A capacity number is a requirement divided by a rate measured on your own pipeline; a datasheet figure is only an upper bound on that rate.
Precision changes the picture between devices that look alike
Double precision dominates Perlmutter’s workload: 44% of all jobs used FP64 only. In the 26Q3 results, Dense MatMul at FP64 scores 414.4 on the NVIDIA H100 80GB HBM3, 400.4 on the NVIDIA H200, 432.0 on the AMD Instinct MI300X, 132.4 on the NVIDIA A100-SXM4-80GB, 217.2 on the NVIDIA B200 and 7.0 on the NVIDIA B300 SXM6 AC. A generation label does not tell you where a device lands for the precision your solver runs; how much FP64 versus FP32 throughput an ensemble or forecasting model actually needs is a sizing question in its own right.
The categories diverge too. In the 26Q3 results the H200 scores higher than the H100 80GB HBM3 in training (812.0 against 709.3) and inference (1914.6 against 1678.1), while the H100 80GB HBM3 scores marginally higher in compute (487.1 against 480.8). Two devices of the same family trade places depending on what the test stresses, which is exactly why one headline number cannot size a pipeline.
Is high GPU utilisation proof the cluster is sized correctly?
No. The GPU-utilisation metric only records whether a kernel is executing at a given moment, and Trainy notes you can reach 100% utilisation by reading and writing memory while doing no computation at all. Trainy describes a training workload sitting at 100% GPU utilisation while reaching only 20% model FLOPS utilisation. Its SM-efficiency example makes the point mechanically: a kernel running for ten seconds on one of an H100’s 132 streaming multiprocessors registers 100% utilisation but 0.7% SM efficiency.
Production data shows the same gap from another angle. A study of Perlmutter jobs in July 2024 found mean GPU utilisation of 71.77% against mean memory utilisation of 28.64%, where memory utilisation is the share of GPU DRAM in use. In that study 37.12% of jobs never exceeded 15% memory utilisation. The utilisation percentage alone would not reveal that over a third of jobs never used more than 15% of GPU memory.
What to log instead is a set of counters read together against application throughput. NVIDIA’s DCGM documentation describes SM activity (the fraction of time at least one warp is active, averaged over SMs) and DRAM activity (the fraction of cycles with device-memory traffic), and warns that a high value in one field does not alone prove a workload is compute- or memory-bound. DRAM activity of about 0.8 is the practical maximum, not 1.0.
| Counter | What it tells you | What it does not |
|---|---|---|
| GPU utilisation | Whether any kernel is running | Whether the SMs or memory are doing useful work |
| SM activity | Fraction of time warps are active across SMs | Whether the workload is compute- or memory-bound |
| DRAM activity | Fraction of cycles with device-memory traffic; about 0.8 is the practical ceiling | Whether bandwidth alone sets the throughput |
| Frame-buffer used | How much VRAM is occupied | How fast that memory is being read |
| Application throughput (ns/day, states/s, images/s) | The rate the capacity plan divides by | Which subsystem limits it, without the counters above |
Should power be sized for the transient peak or the sustained load?
Power for a GPU expansion should be sized from metered draw: the electrical ceiling from the measured coincident peak of what already runs, and the energy and cooling budget from the sustained load the workload actually holds, not from nameplate ratings. Remaining headroom under a fixed facility cap is the cap minus that metered peak, which means the first deliverable from facilities is a time series of actual draw, not a list of installed equipment.
LynxBenchAI holds that only performance sustained under realistic, heavy load is operationally relevant: it discards a warm-up phase, then counts completed iterations inside a continuous timed window at a workload size already scaled until throughput stops improving. Power planning needs the same discipline, because a burst reading and a nameplate figure both describe a state the facility rarely sits in.
Nameplate overstated the energy of the measured H100 workloads. Kinetic Alpha’s analysis of a public National Laboratory of the Rockies dataset, which measured H100 training, fine-tuning and inference at 0.1-second resolution, found duty factors from 0.795 to 0.880, so real workload energy ran roughly 12% to 21% below nameplate. In its simulated 10 MW colocation facility the average IT load was 5.91 MW, a capacity factor of 59.1%, so a continuous-nameplate forecast of 87.6 GWh a year compares with 51.8 GWh from the measured-profile simulation. Its rule of thumb is worth keeping: nameplate for equipment ceilings, the measured peak ratio for a realistic facility peak, the capacity factor for annual energy and a measured duty factor for a GPU-hour.
Peaks can still reach or exceed the rating. Patel et al. (ASPLOS 2024) found peak GPU power in LLM training iterations often reaches or exceeds TDP, and in a production cluster peak GPU power exceeded the server’s GPU TDP by up to 500 W. Their DGX A100, rated at 6,500 W, never exceeded 5,700 W across their workloads. Workload type matters: training clusters swung by up to 37.5% of provisioned power within two seconds against 9% for inference, leaving about 3% headroom to oversubscribe for training against about 21% for inference.
Power capping is the usual lever when a cap binds, and its cost is workload-specific. In the GROMACS study, A100 performance held until the cap fell below roughly 150 W for small benchmarks and 250 W for larger ones. A 2026 comparison of the H100 and H200 found the H100 reaches peak effective memory bandwidth at around 350 W while the H200 needs at least 550 W. Factryze puts the loss from capping an H100 at 600 W at 5–15% of step throughput on transformer pretraining, and stresses that the figure is representative rather than universal and should be validated on your own workload.
Rack and PDU sub-metering versus the substation feed
Lawrence Berkeley National Laboratory’s 2017 metering guide states that metering facilitates capacity planning around space and power utilisation. At its highest monitoring level, values are metered continuously over time rather than estimated from product specifications. Raritan’s white paper separates the levels: inlet metering on a rack PDU gives the power usage and available capacity of the circuits feeding the rack, and outlet metering resolves consumption to the device or server. ServerTech describes PDU infeed monitoring as the basis for capacity planning and for finding stranded capacity, with the most value coming from readings over a long period and stored rack-level peak and minimum/maximum values.
The decision follows from what metering you have. With only a substation feed, compute headroom conservatively from aggregate peak draw with an added margin for the load diversity you cannot see inside the clusters. With rack or PDU sub-metering, sum the measured rack draw and use the observed coincidence of peaks, which gives a tighter and more defensible number. In either case, ask facilities for twelve months of draw including peak events, so seasonal patterns and unusual peaks are inside the data rather than outside it.
Three caveats keep that number honest. Sampling interval matters: the Perlmutter telemetry study notes its 10-second sampling may miss brief spikes. Meter readings need checking: LBNL describes comparing the sum of sub-meters against the main meter and an end-to-end check with a portable meter. Twelve months of history also cannot show the peak pattern of a pipeline you have not installed yet, so the new workload’s own metered draw on test hardware belongs in the sum.
How do we tell whether storage I/O, memory bandwidth or compute will limit the pipeline?
Measure three rates separately: how fast storage delivers data, how fast the CPU prepares it, and how fast the GPU consumes it. Mohan et al. (VLDB 2021) express the rule as: with prefetch rate F, pre-processing rate P and GPU rate G, data stalls appear whenever G is greater than the minimum of F and P. For ensemble data assimilation, that means benchmarking I/O and storage throughput against a representative ensemble workload at the target resolution, provisioning per-node storage and network bandwidth to meet or slightly exceed the GPU’s consumption rate, and sizing node memory to hold the working ensemble state without repeated re-reads.
The measurement itself is straightforward to set up. Mohan et al.’s DS-Analyzer takes G from runs on synthetic data already resident on the GPU, P from runs with GPU computation disabled and the data cached, and the storage rate from a cold-cache run with pre-processing and GPU compute both disabled. The stakes are real: the same study found data stalls account for up to 65% of training time and squander the gains of faster GPUs.
| Measured result | Limiting subsystem | Where capacity goes |
|---|---|---|
| G exceeds storage fetch rate | Storage I/O (fetch stalls) | Storage throughput or caching |
| G exceeds CPU prep rate | Host pre-processing (prep stalls) | CPU cores per GPU, or offloading prep |
| F and P exceed G, throughput flat as graphics clock rises | GPU memory bandwidth | Higher sustained bandwidth per GPU |
| F and P exceed G, throughput rises with clock | GPU compute | More or faster compute |
Scientific pipelines add another pattern: machine I/O throughput has grown far more slowly than compute. The Melissa-DA authors note that from Roadrunner in 2008 to Summit in 2018 peak compute grew by a factor of 134 while I/O throughput grew by a factor of 12. Their framework sizes the server by memory, roughly the sum of the member dynamic states, which came to 1.9 TiB for 16,384 ensemble members, with 2.9 TiB of state data crossing the network in each assimilation cycle.
Radio astronomy offers the same lesson. In LOFAR’s GPU-based Cobalt system, correlation is GPU compute-bound but beamforming is limited by output bandwidth to the storage cluster, and the team considerably over-dimensioned the transpose network because its achievable bandwidth was hard to estimate, targeting double the input bandwidth. They also verified on a single prototype node that the hardware met requirements before building out.
For GPU memory bandwidth itself, a STREAM-style Triad kernel (two reads and one write per iteration, as in BabelStream) mimics the memory behaviour of stencil and linear-algebra codes. For the filesystem, IOR and mdtest are the standard HPC tools for bandwidth and metadata load respectively. The benchmark has to reproduce the real assimilation I/O pattern (burst or steady, the read/write mix), or the sizing will be wrong. The results are also specific to the resolution and ensemble size tested, so rescaling either one means re-benchmarking, and they do not carry across to a different filesystem or interconnect.
Why GPU memory demand does not grow in step with dataset size
Peak VRAM in a reconstruction pipeline is set by what must be resident on the GPU during one step, and that depends on method and tuning, not simply on how many terabytes sit in the archive. A 2023 study of NeRF on large aerial datasets ties GPU memory during training to the training batch, meaning the sampled camera rays and the samples along each one. On an RTX 3090 with 24 GB, it measured training VRAM of 8.5 GB for Mip-NeRF, 9.9 GB for Mega-NeRF and 7.2 GB for the authors’ tiled method, and Mega-NeRF crashed at original image resolution, so the comparison had to downsample by a factor of two.
Ray batch size moves the number directly. The Multispectral-NeRF authors report 10 GB of video memory on an 11 GB GeForce RTX 2080 Ti at 16,384 training rays per batch, and 16 GB on an 80 GB A100 at 32,768. Gaussian splatting hits a harder wall: the TideGS authors estimate a 100-million-Gaussian scene needs nearly 90 GB, beyond a 24 GB GPU, with vanilla 3DGS typically reaching only about 11 million Gaussians. Their out-of-core design trains 1.1 billion Gaussians on a single 24 GB GPU, but performance then depends on fast NVMe storage, measured at 3.3 GB/s: the memory ceiling becomes an I/O requirement.
COLMAP’s documentation describes a similar trade for classical photogrammetry. Running out of GPU memory in patch-match stereo means reducing the maximum image size or the number of source images, and enabling geometric consistency raises the memory needed. Setting the CPU cache too low instead might lead to very slow processing and heavy disk load.
For a multi-terabyte dig site, the VRAM needed to reconstruct without downsampling cannot be derived from dataset size: profile COLMAP’s dense MVS stage and a representative NeRF run on a genuinely representative scene, then buy for the measured peak plus headroom. Whichever of the two pipelines shows the higher measured peak sets the GPU purchase, keeping in mind that NeRF memory can be traded against training time through the ray batch size.
How many GPU-hours will one field season’s capture take?
The throughput a reconstruction lab needs is the season’s expected capture volume divided by the compute hours the grant can fund; the throughput a candidate configuration achieves comes from running the lab’s actual COLMAP and NeRF pipeline on representative dig datasets before purchase. If achieved throughput meets the required figure with margin, that configuration clears the backlog within budget; if not, the options are faster hardware, pipeline optimisation, or lower per-site resolution.
The arithmetic has a clean precedent in batch transcription. Lyceum Technology’s sizing guide for Whisper argues that no published figure will match your pipeline, so you time one GPU against an hour of your own data end to end and use that measured multiplier for every calculation. Backlog divided by the multiplier gives GPU-hours, GPU-hours divided by the number of GPUs gives wall-clock time, and GPU-hours multiplied by the hourly rate gives the budget, plus a margin for the fraction of each hour the card sits idle. The domain is speech, but the method carries over unchanged to images or points per second.
Worked procedure: from a pilot run to a season’s GPU-hours
- Choose a pilot subset that spans the season’s heterogeneity (site types, image counts per site, lighting), because a convenient subset gives misleading throughput for the full season.
- Run the full pipeline end to end on that subset on each candidate configuration and record achieved images per second or points per second.
- Convert the grant’s compute allocation from money to hours using the cost per hour of each hardware or cloud option.
- Divide the season’s expected capture volume by those funded hours to get the required throughput.
- Compare achieved against required throughput and keep a margin for idle time and reruns.
- Add analyst time for quality checks to the timeline, because a backlog can form there even when compute keeps up.
Pipeline optimisation, when step 5 comes up short, is engineering work on the pipeline rather than a measurement question, and it should be re-benchmarked the same way before anyone relies on the improved figure.
How do we turn benchmark results into a capacity number procurement or grant reviewers will accept?
For a 40,000-plate multi-gigapixel digitisation, the storage and compute estimate comes from a measured pilot: capture and process a batch of plates, record raw bytes, processed bytes and processing hours per plate, multiply by 40,000, and if the total exceeds one year’s grant, phase the capture to fit each year’s budget with rented compute rather than one up-front purchase. The pilot is the evidence the plan rests on, and the decision on retaining raw captures belongs in the storage total.
A reviewer can accept a number they can trace: a measured rate on representative data, the requirement it was divided into, the margin added, and the assumptions that would change it. A number derived from a spec sheet or a dataset size has none of that trail, which is why it tends to attract questions. Renting before buying also helps here, since cloud platforms let you run your actual workload on different GPU types before a purchase decision.
| Situation | Basis for the capacity number | What to present |
|---|---|---|
| Full requirement fits this year’s budget | Measured pilot throughput, extrapolated with margin | One purchase or rental, with the measured rate and margin shown |
| Requirement exceeds one year’s budget | Per-unit measured cost and time, extrapolated to the whole archive | Phases sized to each year’s funding, rented compute in early phases |
| Multi-year funding secured, high sustained utilisation expected | Measured duty cycle of the pipeline | Ownership case, set against rental at the same measured throughput |
| Throughput close to the requirement | Pilot results with their spread | A hybrid or fallback plan, re-measured at the first phase boundary |
Each phase should state the rental rate it assumes and be re-priced at the phase boundary.
How do you measure real-time battery-dispatch inference cost per household?
The cost of real-time battery-optimiser inference per household and per dispatch cycle has to be measured on your own fleet. That measurement means instrumenting each dispatch cycle into four buckets (GPU compute, idle capacity held for bursts, data transfer and orchestration overhead) and repeating it at two or three fleet sizes, because extrapolating a per-household cost from one fleet size hides whether it falls, holds or rises.
Scaling behaviour depends on the workload, not only the hardware. Kinetic Alpha’s reading of measured H100 inference traces found that as request rate rises, power flattens while throughput keeps improving, so energy per token falls even when server power barely moves, and two operators on identical hardware can see different energy per unit of compute because of workload mix, batching and utilisation policy. Whether adding the next ten thousand homes lowers your per-household cost therefore depends on how well your batching and scheduling absorb the extra load, not on the hardware alone.
LynxBenchAI’s published results show how each device runs the public PyTorch stack a team’s own models run on, so they are broadly indicative for a custom PyTorch dispatch model when shortlisting hardware; they do not run or measure your optimiser or its cost. Burst demand when a grid event triggers simultaneous recommendations is a separate autoscaling problem, and capacity that scales ahead of the queue rather than after it needs its own plan.
Which parts of a grid pipeline need low-latency compute, and which can run as batch?
Classify each pipeline component by the decision window of the action it feeds: a component whose output drives an action within seconds or minutes (grid-monitoring alerts, for example) needs always-on low-latency compute, while one whose output is used hours later (day-ahead forecasting, for example) can run as a batch job, and each segment gets its own latency SLA rather than one blanket requirement.
LynxBenchAI benchmarks hardware you already have access to with stock PyTorch, measuring throughput, latency, memory and energy, so the first step is to run it on the candidate devices for the always-on segment before sizing that segment.
LynxBenchAI’s published results score inference throughput per model and precision on each device, which gives a measured starting point for how much hardware the always-on segment needs; LynxBenchAI’s benchmark results do not measure your pipeline’s end-to-end latency.
The method is short. For each component, name the downstream action and the time before that action is needed, then assign it to the low-latency or batch-tolerant class and write the SLA from that window. Separate SLAs let the infrastructure be sized for low latency only where it is needed, instead of provisioning always-on capacity across the whole system.
The cost of getting it wrong runs both ways: misclassifying a component either pays for always-on infrastructure it does not need or misses an alerting window. Boundary cases such as intra-day re-forecasts need an explicit review, and the classification is something to revisit. This decomposition tells you which latency each segment needs; it does not size the hardware to meet it, which still takes a measured throughput and latency test of each segment.
Will rented cloud GPU nodes match our on-prem turnaround for ensemble post-processing?
Cloud-rented nodes match on-prem turnaround for ensemble Kalman filter post-processing only if their measured throughput per node, net of data transfer and provisioning overhead, meets the capacity curve for the ensemble member count you expect by the end of the lease, with fallback provision for contention and outages. You find that out by running the same post-processing benchmark in both environments, with today’s on-prem utilisation and throughput as the baseline.
Who can run the cloud-versus-on-prem benchmark before we commit to a lease?
TechnoLynx runs this comparison for ensemble post-processing: it defines the turnaround SLA with you, together with current and projected member counts and fallback requirements, then benchmarks the workload on your on-prem cluster and on representative cloud GPU instances, including transfer overhead and failover scenarios. The route in is the Inference Cost-Cut Pack, and the first step is framing the SLA and member-count projections. The results are validated against the required capacity curve and the SLA with confidence margins, and TechnoLynx supports the renewal, migration or hybrid that follows, including ongoing monitoring of utilisation and throughput. LynxBenchAI’s published results, which score training, inference and compute tests across 48 devices with stock PyTorch, help narrow which cloud devices to test; they do not run your filter.
The comparison needs a representative slice of the workload moved to a test environment. Otherwise, extending the current on-prem arrangement and re-evaluating at the next renewal remains an option.
Headroom for a growing ensemble
Member-count growth does not translate into proportional extra update compute. In Melissa-DA, the EnKF update phase computes covariance matrices over M members, giving a computational complexity of O(M²), which the authors’ experiments confirmed. Under that scaling, doubling the member count multiplies the update cost by about four, so headroom for the lease term has to come from the projected member count run through the measured curve, not a percentage added to today’s figure.
Mixed environments add idle time. Member propagation times in Melissa-DA varied from about 1.5 s to 2.5 s around a 1.9 s average, and when runners sit on heterogeneous resources, such as nodes with and without GPUs, the server waits for the slowest member before it can update. A hybrid cloud and on-prem split should therefore be benchmarked as a hybrid, not as two separate halves.
The decision then follows the measurements. If cloud throughput, net of transfer and provisioning, meets the projected curve with adequate fallback headroom, migration is viable; if on-prem renewal already covers projected growth at lower cost and risk, renew; if cloud throughput is close but uncertain, adopt a hybrid with fallback and re-evaluate at renewal. Member-count projections are themselves uncertain and should be revisited, as should the cloud rates and transfer costs assumed.
The denominator is the part you have to measure
Every number on this page, from GPU-hours per season to headroom under a power cap to node count for a lease, is a requirement divided by a rate, and only the requirement can be written down in advance. The rate has to come from your own pipeline under sustained load: bandwidth measured rather than quoted, throughput timed end to end, power metered over a year rather than read off a label.
For the hardware part of that rate, the 26Q3 results on the LynxBenchAI benchmark and leaderboard score 48 devices across training, inference and compute tests at several precisions, which is a fair way to shortlist before testing your own pipeline. LynxBenchAI also benchmarks hardware you already have access to with stock PyTorch, measuring throughput, latency, memory and energy. It runs the public PyTorch test suite, not your solver, your filesystem or your facility’s power, so it narrows the hardware question while the pipeline benchmark and the meter readings answer the rest.
FAQ
What does GPU memory bandwidth mean?
GPU memory bandwidth is the rate at which data moves between the GPU’s memory chips and its processing cores, usually given in GB/s. It measures data transfer speed, not computational power.
Is HBM better than GDDR for bandwidth-heavy work?
For bandwidth-intensive workloads, GPUs with HBM generally outperform GDDR-based cards even when compute specifications look similar: Atlantic.Net cites up to 3.9 TB/s for the H100 with HBM3 against roughly 900 GB/s to just over 1 TB/s for a high-end GDDR6X consumer card. Compute-bound applications with high arithmetic intensity may not benefit much from HBM.
Is VRAM the same as memory bandwidth?
No: VRAM determines how much data the GPU can hold at once, while memory bandwidth determines how quickly that data moves to the compute cores, and a GPU can be large on one and modest on the other.
