Size an AI data center deployment from measured power draw, not from the nameplate, and price it per request from measured utilisation, not from list rates. In a Brookhaven National Laboratory study (Latif and colleagues, 2024), an 8-GPU NVIDIA H100 HGX node peaked at about 8.4 kW during training, 18% below its 10.2 kW rating, with the GPUs near full utilisation. The cost side has the same problem in a different form: a GPU priced as if it were always busy looks cheap, and the same GPU serving a trickle of requests is not.

The planning chain runs from measured node draw to rack density and cooling, through facility overhead to the meter, and then into a cost per GPU-hour and per request. Utilisation moves the final number most: one study found the assumed GPU utilisation is the dominant source of error in the public LLM cost calculators it surveyed.

Measured node draw, not nameplate, sets AI data center capacity

The Brookhaven measurements are worth reading closely. Training Llama2-13b, the node drew a median of 7.92 kW and a maximum of 8.42 kW. A combined GPU and CPU burn test, built to max out tensor cores and memory, produced a median of 8.43 kW. NVIDIA’s DGX H100 user guide gives the system’s maximum input as 10.2 kW. The lab’s racks had been designed against that manufacturer rating: 70 kW delivered to racks holding five HGX nodes, with roughly 27% redundant margin. The authors conclude that measured figures let operators deploy more IT hardware on the same power distribution.

A second team (Newkirk, Koomey, Shehabi and colleagues, 2025) found the highest average node draw across their workloads was 7.79 kW, never approaching the 10.2 kW rating. Their model put a 90-day training cluster at about 49.0 GWh, against 68 GWh if each node were assumed to run at nameplate.

Inference widens the gap. The ML.ENERGY Benchmark (2025) found that estimating energy from GPU TDP overstated consumption by up to 4.1 times, in the worst case CodeGemma 2B on H100. LLM decoding is memory-bound and draws well below the GPU’s maximum, while diffusion models run close to it. Two services on the same hardware can therefore have very different power profiles, which a spec sheet cannot show. Our piece on how hardware specs shape AI infrastructure performance covers what the spec numbers do and do not tell you.

Do not read this as permission to derate blindly. Microsoft’s ASPLOS 2024 characterisation found that peak GPU power during LLM training iterations often reaches or exceeds TDP, and in a production cluster peak GPU power exceeded the server’s GPU TDP by up to 500 W. We would size power delivery on a measured peak per node for your own workload, and use measured averages only for energy cost.

Getting a power reading you can trust

On A100 and H100, nvidia-smi reports the average of the past 25 ms once every 100 ms, so 75% of activity is never sampled. The same study found its error is roughly ±5% of the reading, not the ±5 W NVIDIA documents. Naive energy measurement varied by up to 70% across workloads, and a repeat-and-phase-shift method brought that down to about 5%. Comparisons between tasks on the same GPU stay reliable because the reading is close to linear. In the FinGraV work on MI300X, a GEMM kernel much shorter than the 1 ms power averaging window showed measurement error as high as 80%.

Two practical consequences follow. NVML’s energy counter, read once before and once after a run and subtracted, gives the energy used between the two calls without polling. And because CPU, memory and interconnect contribute substantially to node power, GPU-only readings undercount what the rack actually has to supply, so node sizing needs node-level measurement.

Rack density, cooling and the multiplier to the meter

The Uptime Institute’s 2024 global survey put the average typical rack at 8 kW, or 7.1 kW once outlier sites above 50 kW are excluded. Separately, a general reference puts AI-focused racks at around 60 kW against about 10 kW for general-purpose ones. Air cooling becomes impractical above roughly 50 kW per rack, according to Uptime Institute, while direct-to-chip cold plates handle 100 to 150 kW. NVIDIA’s DGX SuperPOD design guide places four DGX H100 systems per rack as typical and requires at least N+1 power, with each source sized for 50% of total peak load.

Four DGX H100 at the 10.2 kW rating is 40.8 kW of rack budget; at the 8.4 kW Brookhaven peak it would be about 33.6 kW, a difference of 7.2 kW per rack (4 × 10.2 and 4 × 8.4). That gap decides whether a rack fits under an air-cooled ceiling or a circuit limit, so it is worth measuring rather than assuming.

Then comes the overhead between the GPU and the meter. Using Epoch AI’s multipliers, derived from the GB200 NVL72 reference architecture, one analysis chains 1.53 times at the server, 1.14 times for shared IT equipment and 1.4 times for facility overhead, so each watt of GPU load becomes 2.44 watts at the meter. The 1.4× facility step is the PUE part of that chain. Separately, a PUE of 1.2 adds 20% to energy cost. A separate build-up in the same article, adding 10 to 15% for networking and 30 to 40% for facility overhead, sizes 12,500 DGX H100 servers at 10.2 kW each as 127.5 MW of server power and 190 to 200 MW from the grid. Note that this build-up starts from the nameplate figure, so every multiplier inflates the original overestimate.

Planning input	Spec-sheet habit	What to use instead
Node peak power	Nameplate (10.2 kW for DGX H100)	Measured node peak (8.42 kW training maximum, 8.48 kW under burn stress test at Brookhaven), with N+1 power
Energy per request	GPU TDP × time	Measured steady-state energy per completed request
Rack density	Systems per rack × nameplate	Measured node peak per node, checked against the per-rack cooling limit
Facility load	IT nameplate × PUE	Measured IT load through the server, IT and facility multipliers
GPU telemetry	Raw nvidia-smi samples	Energy counters or repeated, phase-shifted measurements

Inference and training leave different headroom

Microsoft’s study found LLM training clusters offer only about 3% headroom for power oversubscription, while inference clusters offer about 21% at cluster level. Training produced swings of up to 37.5% of provisioned power within 2 seconds, against 9% for inference. Prompt-phase spikes are not correlated across inference endpoints, so they average out across the cluster. POLCA’s simulation, replicating production power patterns, found 30% more servers could be deployed in the same inference cluster with minimal performance loss.

An inference fleet provisioned like a training fleet is leaving capacity stranded behind its power contract. For higher power density facilities electricity is over 10% of total cost of ownership. One 2026 study found the assumed GPU utilisation is the dominant error in public LLM cost calculators. For efficiency at the device level, see our guide to energy-efficient GPUs for machine learning.

How do I calculate cost per request from measured GPU throughput instead of list prices?

Take the fully loaded yearly cost of the capacity you hold and divide it by the requests that capacity actually served, adjusting for real uptime rather than assuming 100%.

TechnoLynx builds this baseline through the Inference Cost-Cut Pack, defining the unit of work, benchmarking your actual workload on the configurations worth comparing, and reconciling the result against your fleet’s billing and utilisation data, starting with agreeing what a representative request looks like.

NVIDIA’s TCO method makes the structure explicit: yearly server cost is purchase cost over the depreciation period plus yearly software and hosting, and cost per 1,000 prompts is that yearly cost divided by the requests a server can serve in a year. The weak input is the denominator. Utilisation is the result of arrival rate, model, quantisation and latency target, not a figure to type in.

Average utilisation hides the hours you pay for

Across four clusters and six months of the Helios trace, average utilisation ranged from 65% to 90%, the hourly standard deviation was 7% to 12%, and every cluster dipped 5% to 8% at night. Interactive inference shows a clear daily cycle, with low average power and bursty spikes tied to user activity. One average cannot show whether a fleet is evenly busy or saturated by day and idle by night. Those two shapes have very different costs per request.

Idle hours still cost money and power. An air-cooled H100 node at Brookhaven idled at a stable 1.8 kW. On rental, an H100 at $3.99 an hour running at 40% utilisation costs close to $10 per useful hour, according to CloudZero (2026), and CHG-MERIDIAN’s on-premise example rises from $2.89 per GPU-hour at full use to about $5.78 at 50%. So the utilisation question to ask is “by hour of day and day of week, over months”, not “on average”.

How batching and concurrency change cost per request on the same GPU

On identical H100 hardware, a Mixtral 8x7B FP16 deployment cost $15.25 per million output tokens at 1 request per second and $0.87 at 25 requests per second. Effective cost fell by roughly 20 times over the first 25 rps and then flattened as the engine saturated. A latency target stops you running at that floor: once a p99 time-to-first-token target applies, the SLA-feasible operating point costs more than the unconstrained saturation cost. Batching improves energy per generation too, with diminishing returns past a point.

Mixed request lengths widen the cost spread further. With variable-length arrivals the load-driven spread widened to 39.3 times on one configuration, against 31.9 times at fixed lengths.

Worked example: the same H100 node at two load levels

The figures below come from the same study’s vLLM sweep on an Azure node with two H100 NVL GPUs at $13.96 an hour on demand. The hardware cost is fixed; only the offered request rate changes.

Configuration	Offered load	Cost per million output tokens
Llama 3.1 8B, FP16, one GPU	1 rps	about $7.60
Llama 3.1 8B, one GPU	peak throughput	$0.311
Mixtral 8x7B, FP16, two GPUs	1 rps	$15.25
Mixtral 8x7B, FP16, two GPUs	25 rps	$0.87

For Mixtral, $15.25 ÷ $0.87 is about 17.5, so the identical node is 17.5 times more expensive per token at low load. At 1 rps, the dual-GPU Mixtral costs about twice the single-GPU models because of its two-GPU denominator, not its parameter count.

Benchmarking your own workload on candidate GPUs

NVIDIA’s sizing guidance starts by measuring the throughput each deployment unit produces under load and the latency it produces it at, then combines that with the latency limit and expected peak demand. FinOps guidance quoted by Koritsu says vendor throughput estimates are often unreliable and recommends load tests with representative input and output token mixes. Request shape changes the result: at mid load a RAG shape cost 2.4 to 2.7 times the chat baseline on the same hardware.

A benchmark you can put in a cost model follows a fixed order:

1. Define the unit of work (request, document, match) and capture a real traffic trace, including length distribution and arrival pattern.
2. Replay that trace on each candidate configuration across a sweep of arrival rates, not a single saturated run.
3. Record throughput, latency percentiles and energy at each rate.
4. Find the highest rate that still meets your latency target, and price the configuration at that point.
5. Cross-check the result against current fleet utilisation and billing before trusting it for a different SKU.

For the hardware shortlist, LynxBenchAI publishes results showing how each device runs the public PyTorch stack your own models run on, so they are broadly indicative for custom PyTorch models, though they do not run or measure your workload. How those results should be read and scored is LynxBenchAI’s methodology, not something to rebuild from this page.

Splitting prefill and decode only pays with enough interconnect

DistServe measured that serving prefill and decode on separate A100 GPUs could reach 3.3 requests per second per GPU against 1.6 colocated, a 2.1 times gain in goodput. Splitwise reported 1.4 times the throughput at 20% lower cost than existing clusters, a different baseline. The catch is the KV cache: a single 512-token request on OPT-66B carries about 1.13 GB, so 10 rps needs roughly 90 Gbps of transfer bandwidth for the overhead to stay hidden. Cloud InfiniBand offers 25 to 50 GBps per GPU pair. Whether disaggregation helps you depends on your context lengths, rates and fabric, and if the transfer cannot keep up it simply moves the bottleneck, so measure it under your own trace before redesigning the fleet.

Turning a shared GPU cluster bill into cost per inference

A defensible cost per inference splits the bill using explicit utilisation, batch size and idle-time assumptions, so that each input can be challenged and changed.

TechnoLynx builds this kind of model by instrumenting the serving pipeline to emit per-request GPU-seconds by model tier and batch composition, reconciling the output against the current bill and utilisation, and then running it as a live model that recomputes as sites are added. Until that telemetry exists, any per-inference figure is an estimate.

The live version is practical: one open tool reads vLLM’s Prometheus metrics on every tick and computes effective cost from observed token throughput and declared hardware cost. The denominator for shared clusters is cluster utilisation, active compute time over total provisioned GPU time.

What assumptions should a cost-per-request model state so finance can sanity-check it?

Each line below is something a finance reviewer can ask about, and a model that hides any of them invites the wrong argument:

• Unit of work. Request, completed response, inspected unit; per complete response captures model verbosity that per-token figures miss.
• Pricing snapshot date and tier. On-demand, reserved or spot, with the date the rate was taken.
• Time-resolved utilisation. By hour of day, not one average.
• Offered request rate and resulting batch size. In-flight concurrency follows from arrival rate and residence time by Little’s Law.
• Idle time and idle power. Hours held but unused, and the node’s idle floor.
• Latency target. The SLO caps achievable utilisation.
• Commitment and amortisation. Reservation cost at every utilisation level; depreciation period for owned hardware.
• Peak headroom. Capacity held for bursts, stated as a separate line.
• What is excluded. Staff, networking, power and cooling upgrades, and end value are the items most often left out.

Does cost per inference fall as a deployment scales, or stay flat?

It can fall, stay flat or rise, depending on whether pooling raises utilisation, whether peak headroom is added and whether pricing moves from on-demand to committed capacity. Pooling helps where loads are uncorrelated: prompt spikes across inference endpoints do not align, so cluster peaks sit below the sum of server peaks. The gain runs out, though; once offered load passes the point where the engine saturates, cost per token stops falling.

Scale can also make things worse. Consolidation raises utilisation but increases correlated failure risk, and correlated demand erodes the pooling benefit. Adding hardware per model is not free either: on A100, Mixtral at four-way tensor parallelism cost about 1.5 times more per token than at two-way, because throughput rose 1.33 times while GPU cost doubled.

How autoscaling headroom bends the cost curve

Inference demand passes into realised power much more directly than batch work, which can wait in a queue. GPU nodes also start slowly, so capacity has to be warmed ahead of known peaks. Headroom for simultaneous peaks at full scale is a cost that a straight-line extrapolation from today’s footprint will miss. One study reported 18% lower infrastructure cost and up to 31% shorter SLO-violation duration than reactive CPU-threshold scaling, from scaling earlier and wasting less. Packing workloads tightly is the other lever; our guide to bin-packing and multi-tenant GPU scheduling covers it.

At what scale does dedicated GPU hardware beat shared cluster time?

There is no universal breakeven count. Dedicated hardware wins at the scale where the projected shared-cluster cost per inference rises above the amortised cost of owning the capacity, and you find that point from your own scale projection.

LynxBenchAI’s published device results inform the throughput side of that comparison for candidate hardware, and the Inference Cost-Cut Pack is the TechnoLynx engagement that turns it into a decision against your measured shared-cluster costs.

CHG-MERIDIAN (2026) names three inputs that decide where the crossover falls: utilisation, the cloud rate you would actually pay and the hardware’s end value. The rate alone can move it a lot: in their B300 example, breakeven against AWS on demand came after about 8.5 months of round-the-clock use, and against a $6.95 market index after about 28 months. Cost is not the whole decision. Data residency and the predictability of a fixed bill also count. The broader own-versus-rent question is covered in our piece on data center GPUs for AI workloads.

The cost of finishing a training run, not the hourly rate

The true cost of a training run is the hours it takes at your measured throughput multiplied by the hourly rate under the pricing tier you choose; cost is duration times price, so throughput losses turn almost directly into cost.

LynxBenchAI is the measuring tool for the throughput half: its published results show how each device runs the public PyTorch stack, broadly indicative for custom PyTorch training too, though not a run of your own job.

Comparing hourly rates gets the answer wrong. CloudZero (2026) notes B200 delivers roughly 2.5 times H100 training performance, so the newer chip is often cheaper per result despite the higher rate. Wall-clock time matters as much: an 8-H100 fine-tuning run of 36 hours at $3.99 costs $1,149, and the same run stretched to 60 hours on an idle-prone cluster costs $1,915. More nodes is not automatically cheaper in energy either; at constant global batch size, adding nodes raised total IT energy by 60% and 148% in two measured jobs.

For a cross-cloud headline, use cost per epoch or per token from measured throughput and one dated pricing snapshot, with batch size, sequence length and precision held constant. Fold one-off fine-tuning cost into a serving price only if your engagement actually covers both.

Putting on-prem and cloud on the same per-unit footing

Build one normalised unit, cost per useful GPU-hour, for each option, including running costs and end value. On-prem wins when utilisation is high and sustained past the breakeven, and cloud wins when it is low or bursty.

TechnoLynx’s Inference Cost-Cut Pack fits here: TechnoLynx defines the unit and utilisation pattern, runs representative workloads on the candidate options, checks the model against measured data and contract terms, and hands over a cost model you can re-run. LynxBenchAI’s published results help narrow the hardware list before that.

CHG-MERIDIAN’s worked example shows the on-prem arithmetic: a $420,000 8-GPU B300 server worth about $114,000 after 36 months uses up $306,000 over 210,240 GPU-hours, about $1.46 per GPU-hour, plus about $1.43 in running costs, for $2.89 per GPU-hour if every hour is busy. Dividing by utilisation gives the real cost of each useful hour. The usual omissions on the on-prem side are staff, networking, facility power and cooling upgrades, and end value. On the cloud side, a full model should include reserved commitments, support tiers, monitoring and likely egress over five years.

Before you commit, re-run the model at lower utilisation and lower end value; if the answer flips, the decision is closer than it looks. The provider choice itself is covered in our cloud GPU versus on-premise total cost analysis.

Reserved capacity only pays above a measured utilisation

Reserved capacity beats on-demand only when your measured utilisation stays above a breakeven threshold, and that threshold has to be tested against seasonal swings rather than read off a single average.

LynxBenchAI publishes device-level throughput results on the public PyTorch stack, giving a reference for sizing the nodes you would reserve; the first step is to shortlist candidate GPUs from its published results.

AWS’s own comparison makes the logic plain. A three-year reservation for a c5d.4xlarge costs $9,224.28 regardless of use, against $25,228.80 on demand at full capacity. Dividing $9,224.28 by $25,228.80 gives about 36.6%, so below roughly that utilisation on-demand was cheaper in this example. GPU rates differ, but the structure carries over.

If demand is seasonal, reserve a baseline sized to the trough and burst to on-demand for peaks. Spare capacity is not guaranteed: SemiAnalysis reported on-demand H100 sold out and Blackwell lead times stretching months in 2026. Re-run the model at each rollout milestone, and check the rate itself with our guide to benchmarking reserved-instance and committed-use pricing against the market.

Who can build a cost-per-request model from our own GPU telemetry?

TechnoLynx builds cost-per-request models from your own fleet data, instrumenting serving to measure GPU-seconds per request, benchmarking your actual workload on the configurations you are considering, and reconciling the result with billing so finance can trace every input. Whether to do this in-house depends on whether your team can instrument the serving stack and replay real traffic across hardware it does not yet own. TechnoLynx’s GPU performance engineering measures your workload rather than quoting rate cards. The engagement is the Inference Cost-Cut Pack, and it starts by fixing the unit of work. For hardware shortlisting, LynxBenchAI is the measuring tool: published device results on the public PyTorch stack, indicative for custom models, not a run of your application.

This does not suit teams that only want a cheaper rate negotiated with a provider, or a generic cloud-spend audit with no GPU workload in it. If your traffic is small and steady, a spreadsheet built on the CHG-MERIDIAN steps is a reasonable first route. Provider-selection advice and FinOps tooling are the other realistic routes, and they work best once you have a measured per-request baseline to give them.

FAQ

How much power does a data centre use per day?

An extra 1 MW of AI facility load running at 90% utilisation uses about 21.6 MWh a day (1 MW × 24 h × 0.9). Facilities range from a few kilowatts for small server racks to tens of megawatts for large sites.

How much power does an H100 GPU draw?

The H100 has a 700 W TDP. LLM serving usually draws far less than that, because decoding is memory-bound and leaves the compute units underused.

Does inference cost more than training?

Over a model’s life it can in energy terms: inference can account for up to 90% of lifecycle energy when serving billions of requests. Microsoft’s characterisation also puts inference above 90% of overall LLM compute cycles.

What is a GPU hour?

A GPU-hour is one GPU running for one hour. Pricing in GPU-hours lets you compare owned hardware directly with a cloud rate.