Why do most cloud-vs-on-premise analyses get it wrong?

Cloud GPU vs on-premise is not a technology debate — it is a financial modelling exercise. The answer depends on workload characteristics, utilisation patterns, and time horizon. Vendor comparisons that show cloud as universally cheaper or on-premise as universally cheaper are both wrong because they assume workload characteristics that may not match yours.

The relevant question: for your specific workload profile — utilisation rate, duration, growth trajectory, and data gravity — which option has a lower total cost of ownership over the planning horizon? The answer is calculable with concrete numbers, not debatable with abstract principles.

The cloud GPU cost model

Cloud GPU pricing follows a straightforward model with non-obvious implications:

On-demand pricing. As of early 2026, AWS, GCP, and Azure offer NVIDIA GPUs (A100, H100, L4, T4) at per-hour rates ranging from £1–£30 per GPU-hour depending on the GPU type, region, and provider. These figures shift with provider pricing changes and currency fluctuations — treat them as order-of-magnitude anchors, not quotable rates. The cost is proportional to the time the instance is running, regardless of utilisation — an A100 instance running at 10% GPU utilisation costs the same as one running at 90%.

Reserved instances. 1-year and 3-year commitments reduce the per-hour cost by 30–60% compared to on-demand. The trade-off: you pay for the reserved capacity whether you use it or not. We see this trade-off repeatedly in our infrastructure advisory work. As a representative example (early 2026 UK pricing): a 3-year reservation on 8× A100 instances at approximately £8 per GPU-hour on-demand reduces to approximately £3.50 per GPU-hour reserved — but the commitment is approximately £740,000 over three years regardless of utilisation.

Spot/preemptible instances. 60–90% discount from on-demand pricing, with the risk that the instance can be terminated with 30-second to 2-minute notice. In our experience across cloud GPU engagements, these are suitable for fault-tolerant training workloads with checkpointing; unsuitable for inference serving or latency-sensitive workloads.

Egress and storage. The GPU instance cost is the dominant factor, but data transfer and storage costs accumulate. Moving training data into the cloud, storing model checkpoints, and transferring results out incur charges that, in our experience across data-intensive engagements, can add 10–20% to the compute cost for data-intensive workloads (an observed range, not a benchmarked industry rate).

The effective annual cost for a sustained A100 workload on cloud (reserved pricing, early 2026 UK estimates): approximately £25,000–£30,000 per GPU per year. For an 8-GPU training node: £200,000–£240,000 annually. These figures are directional — actual costs depend on provider, region, contract terms, and commitment level.

The on-premise cost model

On-premise GPU infrastructure has a different cost structure: high upfront capital, low marginal operating cost, and a fixed capacity that does not scale elastically.

Hardware acquisition. An NVIDIA DGX A100 (8× A100, 80GB each) costs approximately £150,000–£200,000 through standard procurement channels (early 2026 UK pricing; availability and pricing vary by region and supplier relationship). Individual A100 PCIe cards cost approximately £8,000–£12,000 each, with the server chassis, networking, and storage adding £20,000–£40,000. An H100-based system costs approximately 1.5–2× the A100 equivalent. The capital outlay is front-loaded and significant.

Infrastructure. Power, cooling, rack space, and networking. A DGX A100 consumes approximately 6.5 kW at peak load. Annual power cost at £0.12/kWh (a representative UK commercial rate — actual rates vary by contract, location, and tariff): approximately £6,800. Cooling, rack space, and network connectivity add £3,000–£8,000 annually depending on facility type. Total infrastructure operating cost: approximately £10,000–£15,000 per 8-GPU node per year.

Maintenance and administration. Hardware failures, driver updates, security patching, and system administration require staff time. For small deployments (1–4 nodes), the administrative overhead is typically absorbed by existing IT staff. For larger deployments, dedicated GPU infrastructure operations staff are needed.

Depreciation. GPU hardware depreciates over 3–5 years. NVIDIA’s hardware release cadence means that a 3-year-old GPU delivers significantly lower performance-per-watt than the current generation — but it still delivers the same absolute performance it had when purchased. The depreciation model depends on whether the workload’s compute requirement grows over the planning horizon.

The effective annual cost for an on-premise A100 8-GPU node (amortised over 3 years, including infrastructure, based on early 2026 UK estimates): approximately £75,000–£90,000 per year. These figures assume standard procurement pricing, a 3-year depreciation horizon, and typical UK commercial power rates — organisations with volume purchasing agreements, different depreciation schedules, or co-location arrangements will see different numbers. Compared to the cloud equivalent of £200,000–£240,000 per year, on-premise is 2.5–3× cheaper on a per-year basis — if the utilisation is sustained.

The utilisation break-even

The critical variable is utilisation. On-premise costs are fixed: in our experience across infrastructure engagements, you pay the same whether the GPUs are running 100% of the time or 10%. Cloud costs (on-demand) are proportional to running time: you pay only when the GPUs are active.

The break-even utilisation — the point at which on-premise and cloud costs are equal — is typically across our engagements between 40–60% for on-demand cloud pricing and 60–80% for reserved cloud pricing (an observed range, not a benchmarked industry rate), though these ranges shift with regional pricing, procurement terms, and power costs. Below the break-even, cloud is cheaper because you are not paying for idle capacity. Above the break-even, on-premise is cheaper because the fixed cost is spread across more productive hours.

For sustained AI training workloads that run 24/7 — large-scale model training, continuous learning pipelines, pre-training runs — the utilisation is near 100%, and on-premise saves 2–3× over cloud (an observed pattern across our engagements, not a guaranteed outcome). For intermittent workloads — periodic model training runs, batch inference jobs, development and experimentation — the utilisation may be 20–40%, and cloud is more cost-effective.

The GPU underutilisation patterns affect this calculation directly: as an observed pattern across our engagements, if your workloads achieve only 30% of the GPU’s compute capability, the effective utilisation is 30% of the running time — and the break-even shifts toward cloud, because the on-premise hardware is idle (from a compute perspective) even when it is powered on.

Data gravity and latency constraints

Cost is not the only variable. Data location and latency requirements create constraints that the financial model alone does not capture.

Data gravity. If the training data lives on-premise (in existing storage infrastructure, behind a firewall, subject to data residency requirements), moving it to the cloud for GPU processing incurs transfer costs and transfer time. As an illustrative example from our infrastructure engagements: a 100 TB training dataset takes approximately 10 days to transfer over a 1 Gbps connection. If the data changes frequently (continuous learning, streaming data pipelines), the transfer cost and latency become recurring operational constraints. In these cases, deploying GPU infrastructure co-located with the data — on-premise — avoids the data movement problem entirely.

Inference latency. For real-time inference serving, the network round-trip between the application and the GPU affects the total response latency. Cloud GPUs add network latency (1–50 milliseconds depending on the region and the application’s location). On-premise GPUs co-located with the application minimise network latency. For applications with strict latency SLAs (sub-10ms response time), on-premise or edge deployment may be necessary regardless of cost.

The hybrid approach

In our experience, the cost-optimal infrastructure for most organisations is hybrid: on-premise capacity for the sustained baseline workload (sized at the average utilisation, not the peak), and cloud burst capacity for peak demand (training runs, experimentation, seasonal load increases).

The hybrid approach requires workload portability — the training and inference pipelines must run on both on-premise and cloud GPU infrastructure without modification. Containerisation (Docker, Kubernetes) and hardware-abstracted frameworks (PyTorch with CUDA backend, ONNX Runtime) enable this portability. The API choice between CUDA, OpenCL, and SYCL affects portability: a CUDA-only pipeline is portable across NVIDIA hardware in both environments; a workload that needs to run on non-NVIDIA cloud instances (AMD MI300X on certain cloud providers) requires a cross-platform API.

Modelling your specific scenario

The general principles above provide the framework. The specific answer for your organisation requires modelling with your numbers: your workload utilisation profile, your data volume and location, your latency requirements, your power costs, and your planning horizon.

GPU infrastructure cost calculation template

Use the variables and formulas below to model your own cloud-vs-on-premise scenario. All sample figures are illustrative, based on representative early-2026 UK pricing — substitute your own contract rates, power costs, and depreciation policies.

Variables — define these for your workload:

• N — number of GPUs required
• H — hardware acquisition cost per GPU node (£)
• Y — depreciation horizon (years, typically 3–5)
• P — annual power cost per node (£) — calculate as: peak kW × 8,760 hours × £/kWh
• M — annual maintenance, cooling, rack, and admin cost per node (£)
• C — cloud cost per GPU-hour (£, on-demand or reserved rate)
• U — average utilisation rate (0.0–1.0) — fraction of hours the GPUs are actively running workloads
• D — annual data egress and storage cost for cloud (£)

On-premise annual cost:

Cost_onprem = (H ÷ Y) + P + M

This is fixed regardless of utilisation. For an 8-GPU A100 node at representative early-2026 UK pricing: (£175,000 ÷ 3) + £6,800 + £5,000 ≈ £70,000/year.

Cloud annual cost:

Cost_cloud = (N × C × 8,760 × U) + D

This scales linearly with utilisation (the 8,760 factor is just hours per year, not an evidentiary claim). For 8× A100 GPUs at £8/GPU-hour on-demand, 60% utilisation — an illustrative example, not a benchmarked industry rate: (8 × £8 × 8,760 × 0.6) + £10,000 ≈ £347,000/year.

Break-even utilisation (on-demand cloud vs on-premise):

U_breakeven = (Cost_onprem − D) ÷ (N × C × 8,760)

Below this utilisation, cloud is cheaper. Above it, on-premise is cheaper. As an illustrative example (not a benchmarked industry rate): (£70,000 − £10,000) ÷ (8 × £8 × 8,760) ≈ 0.11 — meaning on-premise beats on-demand cloud at any utilisation above ~11%. Against reserved pricing (lower C), the break-even shifts higher.

Hybrid threshold — sizing on-premise baseline capacity:

N_onprem = N × U_sustained (round down to whole nodes)

N_cloud_burst = N_peak − N_onprem

Size on-premise hardware for the sustained average utilisation, not the peak. Burst above that baseline into cloud. As an illustrative example (planning heuristic, not a benchmarked industry rate): for a workload that averages 60% of peak, on-premise covers 60% of capacity, cloud covers the remaining 40% on-demand.

All figures are illustrative and based on representative early-2026 UK pricing. Actual costs depend on provider contracts, regional power rates, procurement terms, and currency fluctuations. Run these formulas with your own numbers before making a commitment.

The decision is financial, not philosophical. A GPU Performance Audit provides the infrastructure cost modelling and performance analysis your workload needs before committing to either path.