A GPU cloud computing cost comparison is only worth taking into a budget meeting once every option (on-demand, reserved, spot, dedicated or owned) is expressed in one unit: cost per 1,000 requests, or cost per completed job, for your own workload. Per-GPU-hour list prices are the most quoted number and the least useful, because a well-utilised expensive GPU routinely beats a poorly utilised cheap one. Whether cloud, on-prem or a hybrid split wins then comes down to three things you can measure or model: sustained utilisation, the pricing tier you would actually commit to, and the constraints (capacity, residency, refresh timing) that remove options before price even comes into it.

This applies directly to teams running AI infrastructure and production SaaS. For them, the network side of a serving bill grows with every new user, and a GPU commitment is billed every month whether the hardware is busy or not.

Cloud GPU pricing comparison: normalise AWS, Azure and GCP to cost per 1,000 requests

You can get a comparable cost per inference across AWS, Azure and GCP. The way to do it is to deploy the same container on each provider and measure throughput under your real traffic shape. A spreadsheet of list prices will not get you there. What you end up with is one number per provider (cost per 1,000 inference requests) plus a capacity risk rating for each region.

TechnoLynx runs this normalisation as part of the Inference Cost-Cut Pack. It first defines the comparison workload and which instance tiers count as equivalent. It then runs the same benchmark workload on each candidate provider and reports any gap between its measured figures and the vendors’ published numbers before a figure goes into the board case.

The formula, and why the denominator is the hard part

Cost per 1,000 requests = ($ per GPU-hour ÷ requests served per hour) × 1,000. The price is the easy half. The denominator depends on the model, VRAM, batch size, quantisation and serving stack, so you have to measure it under real traffic rather than estimate it. You also need to define the unit before you measure anything: model and parameter count, requests per second, p95 latency target, daily volume and peak-to-average ratio. Training is compared the same way, per completed job. Run one representative job end to end on two providers and compare the cost of the completed job, not the hourly rate.

What the rate card leaves out

Prices for identical silicon vary widely between providers. Qovery checked in October 2026 and found a single on-demand H100 at about $6.88 per GPU-hour on an AWS p5 node and about $12.29 on an Azure ND H100 v5 node. CloudZero and Sesame Disk give the same AWS figure. CloudZero (2026) also counts spot and marketplace rates, which gives H100 rentals from $1.49 to $6.98 an hour, roughly a 5x spread.

Treat any single figure with care. Cast AI’s analysis puts the $12.29 rate on AWS p5 and $13 on Azure. That conflicts with the three sources above, which agree on about $6.88 for AWS, so we follow them. Comparisons priced before June 2025 also predate AWS’s June 2025 cut of 44% to P5 On-Demand prices. Prices move both ways: AWS raised H200 instance prices by 15% in January 2026. A normalised figure can go stale before the board signs off, so put a date on it.

Three things skew a comparison beyond the hourly rate:

• Packaging. AWS sells A100 GPUs only in fixed eight-GPU instances, while Azure and GCP let you provision a single GPU. On AWS, a job that needs fewer GPUs pays for idle ones. Cudo’s example is a team that needs six GPUs and has to rent eight.
• Egress. In one priced inference model, egress was 9 to 11% of the bill at about 10 TB a month and 32 to 40% at about 50 TB.
• Availability. In Qovery’s words, a cheaper region you cannot get capacity in is infinitely expensive. Check quota and stock before you build the price model.

Deciding which tiers are “equivalent” is a judgement call. If the benchmark workload does not reflect your production inference patterns, the normalised figure will mislead the TCO model.

Independent verification here means figures from running your own workload, not vendor-supplied claims. For the hardware layer, LynxBenchAI’s published device results show how each device runs the public PyTorch stack your own models run on. That makes them a broad guide for custom PyTorch models, but they do not measure your application, so you still need the provider-by-provider run.

Should we pilot reserved or spot GPU instances before signing a 12-month commitment?

Yes. Measure before you sign, and size any commitment to the floor of your demand. A commitment turns a volatile hourly rate into a fixed cost, which makes it a bet on your own utilisation forecast. The purchasing model moves the bill more than the choice of provider. One comparison using identical workload specifications across all three hyperscalers found on-demand rates remarkably similar. The big differences were in committed, spot and GPU pricing.

Treat the headline discounts as ceilings, not quotes. AWS Savings Plans go up to 66% or 72%, GCP committed use discounts up to 55% on general resources, and Azure Reservations up to 72%. None of the three publishes a fixed one-year or three-year percentage for each GPU type.

On GCP, a GPU commitment covers exactly one GPU type and is billed monthly whether you use it or not. It does not reserve capacity on its own, so a resource-based GPU commitment needs reservations attached to guarantee the hardware. Google gives its own break-even rule: with a 30% discount, an instance has to run more than 70% of the month to come out ahead.

Spot sits at the other end. AWS recommends Spot only for stateless, fault-tolerant work, and not for workloads that cannot cope with losing capacity. Qovery’s summary: spot is fine for offline inference and dangerous for anything with a latency SLA.

A pilot that settles the commitment question runs in this order:

1. Run the real container on two providers, with both allocation and compute utilisation instrumented.
2. Split the measured demand into a continuous baseline, predictable bursts and an interruptible tail.
3. Price the baseline on committed terms, bursts on shorter terms or on demand, and the tail on spot or preemptible capacity.
4. Before relying on spot, build asynchronous checkpointing and automatic requeue for the tail.
5. Model no commitment, one year and three years against the measured utilisation curve, with an availability note for each region.

Whether the committed rate you are quoted is competitive is a separate question, covered in benchmarking reserved-instance and committed-use pricing against the market. Exit terms are covered in managing renewal timelines, lock-in and cost accountability in cloud GPU contracts.

How many cloud GPU instances do we actually need?

Size the cloud deployment from measured workload behaviour: GPU and memory utilisation, memory bandwidth and interconnect traffic. Do not size it from the node count of the cluster you are leaving, because that count carries whatever idle capacity and mis-sizing the old cluster had. Cast AI reports average GPU utilisation of 5% across more than 23,000 Kubernetes clusters in its dataset, and a VentureBeat report cites audits that found the same figure.

TechnoLynx does this sizing as part of the Inference Cost-Cut Pack. It captures utilisation from your production runs, benchmarks representative kernel shapes on the candidate instance families, and recommends an instance family and node count with the measurements behind them.

At low utilisation, the hourly rate is not the real cost. Effective GPU cost is the hourly rate divided by actual utilisation. In Cast AI’s example, a $12.29 GPU-hour at 5% utilisation costs $245.80 per GPU-hour of compute actually delivered (12.29 ÷ 0.05).

Measure two utilisation numbers, not one

Allocation utilisation is the share of reserved GPU-hours assigned to a running job. Compute utilisation (SM occupancy, memory bandwidth and, ideally, model FLOPs utilisation) shows how hard the GPUs worked once assigned. A cluster at 85% allocation and 30% MFU has an efficiency problem, not a capacity problem, and buying more GPUs makes it worse. On AWS, Compute Optimizer reports GPUUtilization and GPUMemoryUtilization for each instance, which is a starting point. DCGM exporters feeding Prometheus will get you both numbers in an afternoon.

That gap is also why TechnoLynx audits the current serving path before recommending a migration. Batching, quantisation, runtime choice and serving configuration sit below the model. They are independently tunable without replacing the model.

Choose the instance family on memory and bandwidth before FLOPs

One cost guide’s advice is to check memory before compute. A job that is memory-bound at 40 GB runs better on a card with more memory and a lower clock than on a faster card it cannot fill. The A10G and the L4 show why a single datasheet figure misleads. Both have 24 GB, and the L4 delivers 73% more FP16/BF16 throughput, but the A10G has twice the memory bandwidth: 600 GB/s against 300 GB/s. A bandwidth-bound workload and a compute-bound one can rank those two cards in opposite order.

The longer-term trend points the same way. Across NVIDIA data-centre GPUs, compute has grown faster than off-chip memory bandwidth. LynxBenchAI’s benchmark results show how individual devices perform in measured stock-PyTorch runs, rather than on spec sheets.

Packaging also limits how finely you can size. A P5 instance carries up to eight H100 GPUs with up to 640 GB of HBM3 in total. On GCP, the A2 family (A100) is aimed at fine-tuning and cost-optimised inference, and G2 (L4) at cost-optimised inference.

Scaled-down or synthetic kernel benchmarks do not reproduce production data volumes or network contention. To turn the result into a requirement a vendor can quote against, see stating defensible GPU capacity requirements before you buy.

For a validated model, a move to cloud GPUs needs evidence that the system remains fit for its intended use. Use that evidence to decide which existing validation still applies and which infrastructure, interface and data controls need new testing. Validated state does not move with the servers, and the regulated organisation keeps the decision on whether the system is still fit for use.

When does on-prem GPU capacity become cheaper than renting?

Owning wins when utilisation is high and sustained, and the threshold moves with the cloud price you compare against. Sesame Disk (2026) puts on-prem break-even at roughly 50 to 83% sustained utilisation against hyperscaler on-demand pricing. Introl’s five-year TCO model finds cloud cheaper below 60 to 70% utilisation. CloudZero estimates that a used eight-GPU H100 system pays back against rental in about seven months at full utilisation and close to two years at 30%.

Break-even months for owned H100 capacity, by cloud pricing tier

Sesame Disk’s 2026 figures for an owned cluster against its own cloud tier rates:

Cloud tier	Rate per GPU-hour	Break-even at 85% use	Break-even at 60% use
On-demand	$3.25	3.6 months	5.1 months
1-year reserved	$2.35	5.5 months	7.8 months
3-year reserved	$1.85	8.7 months	not given
Spot	$1.65	past 10 months (use not stated)	not given

The table makes one judgement plain. In these figures, the cheaper the cloud tier compared against, the longer ownership takes to pay back. Comparing an on-prem quote with on-demand list prices therefore makes ownership look better than it is.

What does our current on-prem GPU server really cost?

A fully loaded figure covers hardware, facility, power, cooling, networking, storage, staffing, maintenance, spares, depreciation and redundancy overhead. Introl’s model shows how far the purchase price understates this. The $3 million for 100 H100 GPUs is about 35% of a five-year TCO of $8.6 million. Staffing exceeds the amortised hardware cost in many configurations: half an infrastructure engineer at fully loaded cost runs $75,000 to $100,000 a year. Utilisation has to be built in too. CloudRift’s run-cost estimates assume 100% utilisation, and at 80% the effective cost is about 25% higher.

The comparable number is a fully loaded cost per unit of work. Add up hardware, facility, power, cooling, staffing and maintenance over a period, then divide by the requests, claims or jobs served in that period. Shared facility costs and staff time have to be assigned through a consistent tagging and chargeback scheme, so the on-prem figure lines up unit for unit with cloud pricing. The utilisation you assume shifts the result materially, so state it, and your cost split, next to the figure. Cost allocation and TCO modelling for multi-tenant GPU systems covers the attribution in depth.

Six regional clusters or one cloud GPU region: costing centralisation and the hybrid split

What consolidating regional GPU clusters into one cloud region saves has to be modelled from your own numbers, not assumed. It turns on today’s patching and management labour, how well each cluster is utilised, and the egress and latency costs the central region adds. A defensible model sets infrastructure spend, patch and management labour, downtime and breach exposure, and compliance constraints side by side for both options, at current and projected volumes.

TechnoLynx costs the cloud side of this comparison through the Inference Cost-Cut Pack, measuring cost per 1,000 requests for your own workload in the candidate central region; the first step is fixing the comparison workload and the instance tiers that count as equivalent.

Residency costs money where it removes choice, and the bill shows it through regional prices, missing services and copies kept inside the permitted area. For example, keeping a second copy of Azure hot Blob storage in Germany doubles the storage price compared with locally redundant storage. Data transfer between two regions inside the same area is charged like any other inter-region transfer. Qovery notes that residency rules can remove the cheapest region from your table altogether.

If no jurisdiction imposes residency, compare full centralisation with the distributed estate directly. If some countries require local processing, cost the hybrid as the local infrastructure you keep plus routing and integration overhead. Centralise only where the savings exceed that added complexity. The model is only as good as your current data on patching labour and incidents, and legal requirements can change which countries must stay local.

How much headroom should we build in for model growth and retraining?

Treat headroom as a trade-off between idle owned capacity and paying for cloud burst. Settle it by modelling growth scenarios against the cost and service level of each provisioning option. Do not fix it upfront. Introl’s planning guidance targets 65 to 75% average utilisation with a 20 to 30% buffer for spikes and growth, and notes that inference typically runs at 40 to 50% because requests vary. It also recommends keeping 20%, 50% and 80% growth forecasts and adjusting them quarterly.

TechnoLynx sizes the cloud-burst side of this trade-off through the Inference Cost-Cut Pack, capturing utilisation from your production runs and recommending an instance family and node count with the measurements behind them; the first step is instrumenting allocation and compute utilisation on your current runs.

Getting this wrong is expensive in both directions. Introl reports that Meta underestimated its GPU needs by 400% in 2023, adding $800 million in emergency procurement. A Fortune 500 financial institution over-provisioned by 300% and left $120 million of GPUs idle for two years.

Over-provisioned capacity is fixed cost that sits idle, as the Fortune 500 example shows. Bursting to the cloud for retraining cuts fixed cost but relies on capacity being available when you burst. Sesame Disk describes a middle path: size on-prem capacity for median traffic, run it at 80% utilisation or more, and burst to the cloud for peaks and experiments. The burst part is often eligible for spot pricing.

Three-year projections of model complexity are uncertain, and burst prices change over the horizon.

GPU refresh cycles, stranded assets and the end-of-life server

Separate long-lived power and cooling from short-lived GPUs in capital planning, and finance them on different terms. Axiant describes three-to-five-year terms for compute and storage, matched to the refresh cycle, and longer terms for durable power, cooling and build-out. Financing both on the same term means either paying for infrastructure long after the servers are gone, or paying for servers over a term they will not survive.

TechnoLynx measures how your workloads actually run on the GPUs you own and on candidate replacements through GPU Performance Engineering, so a refresh decision rests on measured performance rather than hardware age; the first step is benchmarking representative kernel shapes on your current hardware.

How quickly GPUs age shows in their improvement rates. Epoch AI analysed 470 GPUs released between 2006 and 2021 and found that FLOP/s per dollar doubles roughly every 2.5 years. Del Sozzo et al. (2026) studied NVIDIA data-centre GPUs and found FP16 and FP32 performance doubling every 1.43 and 1.67 years. Over the same period, launch prices of the top GPUs rose 22.7% a year. Del Sozzo et al. also suggest that lifetime compute costs are increasingly dominated by the hardware purchase rather than by electricity. CloudRift’s run-cost model, for comparison, assumes a four-year useful life with 10 to 20% salvage value.

On a three-to-five-year refresh cycle, an on-prem TCO over that horizon has to carry a refresh. A comparison that prices only the first purchase favours ownership. A lease matched to the refresh cycle keeps you current without stranded capital. A stable, long-horizon workload at high utilisation can still justify buying outright. Power and cooling outlive several GPU generations, and per-GPU power keeps rising: Introl notes that per-GPU power has reached 1,200 to 1,400 W.

If the current server is near the end of its life, compare the cloud or GPU quote against forward-looking replacement cost, not historical depreciation. If it has real life left, use its remaining book value and measured power draw. Split shared-server costs by measured usage share, not equally. TechnoLynx can build that baseline as part of the Inference Cost-Cut Pack, checking it against utility bills and asset records before setting it against the quote’s projected cost per unit.

If privilege or residency rules rule out shared public cloud, is dedicated cloud GPU still cheaper?

Dedicated cloud GPU is still an option. A rule that forbids calls to shared, multi-tenant public cloud APIs does not automatically rule out single-tenant cloud GPU with contractual data isolation. Cudo notes that specialised providers offer dedicated clusters, residency assurances and clear SLAs for regulated workloads. Hyperscalers have sovereign offerings in some regions, but setting them up means working through complex service catalogues. Shared tiers are a different matter: CloudZero calls RunPod’s multi-tenant community tier fine for experiments and rough for compliance. Whether a dedicated tier qualifies is a legal and compliance decision, not a purely technical one. Contract terms such as subprocessor access can rule out a technically isolated option.

If a compliant dedicated tier exists, compute three-year TCO for it and for the on-prem refresh, under both expected and burst volumes. Choose on-prem if steady-state volume sits above the break-even point, cloud if volume sits below it or swings widely, and a hybrid (on-prem baseline plus cloud burst) if it hovers around it. The compliance, networking and support requirements that push you toward a hyperscaler also raise the cloud cost, which makes ownership look better. For hosted AI models, residency is a price tier in its own right. Microsoft prices its global deployment type lowest, and AWS says global inference profiles on Bedrock save about 10% compared with geographic ones.

An arXiv cost-benefit analysis of on-premise LLM deployment found ownership economically viable mainly for organisations processing at least 50 million tokens a month or bound by strict data-residency mandates. For 10 to 50 million tokens a month, the same study says hybrid setups that keep sensitive work local and send burst traffic to the cloud benefit most. Underestimating how often you burst favours on-prem, and overestimating it favours cloud.

Who can build a defensible cloud-vs-on-prem GPU cost case for us?

TechnoLynx builds the measured cost case behind a cloud, on-prem or hybrid GPU decision. That means cost per inference normalised across the providers you are considering and instance sizing from your real utilisation. The work is delivered through the Inference Cost-Cut Pack. It starts by fixing the workload unit and auditing the current serving path, so a migration is not priced against an untuned baseline. For the hardware choice itself, LynxBenchAI is the measuring tool: published device results on the public PyTorch stack, used as a reference rather than as a run of your application. Whoever you choose, ask to see measured figures, the method behind them and how discrepancies get reported.

The engagement covers inference cost and latency work, not cloud-bill negotiation. It does not decide legal questions such as privilege or residency, which need your counsel and compliance team. If your workload is stable and well understood, the provider’s own tools may be enough: GCP, for example, generates committed use discount recommendations by comparing your costs with and without a commitment.

FAQ

How much does an A100 cost per hour in the cloud?

CloudZero’s 2026 survey found A100 rates from $0.68 an hour on spot to $5.03 for GCP’s 80 GB on-demand, with CoreWeave’s dedicated tier at $2.21 to $2.70. Google lists a single-A100 a2-highgpu-1g at about $3.67 an hour on demand.

What percentage of cloud spend is wasted?

Flexera’s 2026 report, as cited by Qovery, put self-estimated cloud waste at 29%, driven by AI workloads, with idle non-production GPUs a large share of it.

Will GPU rental prices keep going up?

CloudZero cites GetDeploying’s index, which shows on-demand GPU rental prices up 1.7% over 12 months and down 0.5% over four weeks, with Blackwell demand absorbing supply that would otherwise have pushed rates down.